Feature/distill token optimization

[Tomo1912]

Description

Add optional distill parameter to the fetch server that aggressively cleans HTML content to minimize token usage. When enabled, removes scripts, styles, navigation, headers, footers, ads, and other non-essential elements before conversion to markdown. Achieves 72.8% average token reduction across real-world tests.

Server Details

• Server: fetch
• Changes to: tools (added distill parameter to fetch tool)

Motivation and Context

LLM token costs are a significant operational expense. Current web-fetch returns full HTML including navigation menus, ads, scripts, and UI clutter - wasting tokens on non-content elements.

Test Results:

Website	Standard Tokens	Distilled Tokens	Reduction
MCP Docs	2,154	13	99.4%
TechCrunch	506	263	48.0%
Python Docs	662	629	5.0%
Average	1,107	302	72.8%

How Has This Been Tested?

• Tested with Claude Desktop as MCP client
• Tested against multiple real-world websites (documentation sites, news sites, technical docs)
• Verified backward compatibility - existing calls without distill parameter work unchanged

Breaking Changes

None. The distill parameter defaults to False, maintaining full backward compatibility.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation update

Checklist

• I have read the MCP Protocol Documentation
• My changes follows MCP security best practices
• I have updated the server's README accordingly
• I have tested this with an LLM client
• My code follows the repository's style guidelines
• New and existing tests pass locally
• I have added appropriate error handling
• I have documented all environment variables and configuration options

Additional context

No new dependencies required - uses existing readabilipy for content extraction.

[cliffhall]

@claude review

[claude]

Claude finished @cliffhall's task in 1m 33s —— View job

---

PR Review: Feature/distill token optimization

• Gather context and read changed files
• Review server.py changes
• Review README.md changes
• Post full review feedback

---

Summary

The feature idea — reducing token usage from fetched web content — is a reasonable one, but this implementation has several significant technical problems that should be addressed before merging.

---

Issues

1. Redundant / counterproductive architecture (Major)

The distill_html function is applied before readabilipy.simple_json_from_html_string(html, use_readability=True). The readabilipy library is a Python port of Mozilla's Readability.js, which already does content extraction: stripping navs, footers, ads, sidebars, and returning only article-body HTML. Pre-processing with distill_html may actually impair Readability's heuristics by removing structural signals it relies on to locate the main content.

The 99.4% token reduction on MCP Docs (2,154 → 13 tokens) is a red flag here — 13 tokens is nearly empty output, suggesting core content is being stripped alongside boilerplate.

The distill path is effectively doing: strip HTML with fragile regexes → feed already-degraded HTML to Readability → convert to Markdown. A better design would either post-process Readability's output, or explore whether readabilipy has additional options to tighten its output.

2. ReDoS risk in the ad-pattern regex (Major / Security)

server.py line 63:

r'<[^>]+(class|id)=["\'][^"\']*\b(ad|ads|...)\b[^"\']*["\'][^>]*>[\s\S]*?</[^>]+>'

• [\s\S]*?</[^>]+> uses lazy .* which is O(n²) worst-case on backtracking engines when the closing pattern isn't found (e.g., unclosed tags from malformed HTML).
• The closing tag </[^>]+> matches any closing tag, not the specific matching opening tag — so <div class="ad">...</span> would incorrectly match and consume content.
• On adversarially crafted or simply malformed HTML from the web, this could cause significant slowdowns or incorrect behaviour.

Fix this →

3. Nested-tag regex is inherently broken (Major)

The per-tag regex:

html = re.sub(rf'<{tag}[^>]*>[\s\S]*?</{tag}>', '', html, flags=re.IGNORECASE)

Fails silently on:

• Nested tags: <nav><nav>inner</nav>outer</nav> — only the inner </nav> is consumed; outer</nav> remains.
• Self-closing or void usage: <nav /> is not matched.
• Attributes containing >: rare but valid in older HTML; [^>]* breaks on this.

HTML cannot be reliably parsed with regex. This should use lxml (already available as a transitive dependency of readabilipy) or html.parser from the stdlib.

4. No tests (Major)

The repository already has a test suite for the fetch server (test(fetch): add unit tests — commit 71d5d71). This PR adds no tests for distill_html or the distill parameter integration. Given the complexity of the regex logic, unit tests are especially important here.

5. Undefined raw=True + distill=True interaction (Minor)

When force_raw=True, extract_content_from_html is never called, so distill silently has no effect. The parameter description and README should document this — or the combination should raise an error/warning.

# fetch_url line 204 — distill is silently ignored when raw=True
if is_page_html and not force_raw:
    return extract_content_from_html(page_raw, distill=distill), ""

6. Performance: multiple regex passes over large HTML (Minor)

Six separate re.sub calls iterate over potentially megabyte-scale HTML strings before handing off to Readability. An HTML-parser-based approach (decompose by tag, decompose by class/id) would be a single parse pass and far more efficient.

---

README change

The documentation update (README.md line 19) is accurate and well-written. No issues there.

---

Verdict

Request changes. The feature concept is valid and worth pursuing, but the core implementation should be reworked to:

1. Use an actual HTML parser (lxml/html.parser/BeautifulSoup) instead of regex for structural stripping
2. Reconsider whether pre-processing before Readability or post-processing after Readability produces better results
3. Add unit tests covering distill_html and the distill=True fetch path
4. Document the raw=True + distill=True interaction

[Tomo1912]

Addressed all review feedback

Thanks for the thorough review! I've reworked the implementation to address every issue raised:

Issue 1: Redundant / counterproductive architecture (Major) ✅

Fixed. distill_html now runs post-Readability instead of pre-Readability. The pipeline is now:

1. Readability extracts main content (preserving its heuristics)
2. distill_html cleans the Readability output (removing any remaining non-content elements)

This avoids interfering with Readability's content-detection signals and fixes the 13-token empty output problem.

Issue 2: ReDoS risk in ad-pattern regex (Major / Security) ✅

Fixed. Replaced all regex-based HTML parsing with BeautifulSoup (lxml parser). No more [\s\S]*? patterns or backtracking risks. The ad/navigation pattern matching now uses BeautifulSoup.find_all() with a compiled regex for class/id keyword matching only (not for HTML structure parsing).

Issue 3: Nested-tag regex is inherently broken (Major) ✅

Fixed. BeautifulSoup handles nested tags, self-closing tags, and attributes containing > correctly by design. Tested with nested <nav> tags to verify.

Issue 4: No tests (Major) ✅

Added. 21 pytest tests in tests/test_distill.py covering:

• TestDistillHtml: 17 unit tests for distill_html() — script/style/nav/header/footer/aside/iframe/svg/form removal, HTML comments, nested tags, ad/cookie/social class/id patterns, empty elements, br/hr/img preservation
• TestExtractContentDistillIntegration: 4 integration tests for the full extract_content_from_html(distill=True) pipeline, including a regression test ensuring output is never near-empty

Issue 5: Undefined raw=True + distill=True interaction (Minor) ✅

Documented. Added "Has no effect when raw=True" to:

• The distill Field description in the Pydantic model
• The README parameter documentation
• The extract_content_from_html and fetch_url docstrings

Issue 6: Performance — multiple regex passes (Minor) ✅

Fixed. Single BeautifulSoup parse pass replaces six separate re.sub calls. The lxml parser is already available as a transitive dependency of readabilipy.