WebFetch → Hybrid fetch_document Conversion (v1.2.0)

[Number531]

Summary

Replace the Agent SDK's built-in WebFetch tool with a custom MCP tool (fetch_document) that tries direct HTTP fetch first, then falls back to Exa /contents on 403/timeout/empty responses. Eliminates the entire class of SEC.gov 403 failures observed in trial runs.

Plan: docs/pending-updates/websearch-conversion.md (v1.2.0)

Problem

• SEC.gov and other government sites block direct fetch() from cloud IPs → 403
• SDK WebFetch retries the same URL, gets 403 again → content lost
• Exa /contents already crawls these URLs successfully (proven by 27 hybrid clients)

Architecture

fetch_document(url, prompt)
  → Phase 1: Direct HTTP fetch (free, fast)
  → Phase 2: Exa /contents fallback (on 403/timeout/empty)
  → _hybrid_metadata: { source, fallback_reason, confidence }

Key distinction: fetch_document uses Exa /contents (extract text from known URL), NOT Exa /search (find URLs by query). A "direct hit" means raw fetch() with zero Exa involvement.

Scope

• 10 files modified, 1 new file, 2 new test files
• Feature-flagged: HYBRID_WEBFETCH=true (default off, zero behavior change when off)
• Rollback: single env var flip, zero code revert

Implementation Roadmap

Phase A: Wire the New Tool (blocking prerequisite)

• A1. Create DirectFetchHybridClient.js (direct fetch → Exa /contents fallback)
• A2. Add HYBRID_WEBFETCH feature flag (featureFlags.js)
• A3. Add fetch_document tool definition (toolDefinitions.js)
• A4. Register handler (toolImplementations.js)
• A5. Add direct-fetch domain mapping (domainMcpServers.js — 20 research subagents)
• A6. Instantiate client in server (claude-sdk-server.js)
• A7. Smoke test — verify tool accessible (147 tools)

Phase B: Replace WebFetch (requires Phase A complete)

• B1. Update legacy STANDARD_TOOLS (legalSubagents.js)
• B2. Update modular STANDARD_TOOLS (_standardTools.js)
• B3. Update ORCHESTRATOR_ALLOWED_TOOLS
• B4. Update MCP fallback instructions

Phase C: Observability

• C1. PostToolUse _hybrid_metadata extraction (sdkHooks.js)
• C2. PostToolUseFailure dual-failure extraction
• C3. Verify hookSSEBridge auto-forwards new fields

Phase D: Testing & Frontend

• D1. Unit tests (~29 assertions)
• D1b. Hook observability tests
• D2. Domain MCP server tests (28 → 29)
• D3. Regression (existing suites unchanged)
• D4. Live integration test (SEC.gov, Wikipedia)
• D5. Frontend timeline rendering (app.js — green/blue/red tags)
• D6. Enable HYBRID_WEBFETCH=true and full validation

Phase E: Rollback Verification

• E1. Confirm HYBRID_WEBFETCH=false restores all original behavior

Related

• Exa API modernization (v3.1.0)
• 27 existing hybrid clients (proven pattern)
• Deferred tool loading for Agent SDK subagents — blocked on upstream support #14 (deferred tool loading — complementary)
• Observability: code execution timeout awareness + subagent lifecycle ledger #24 (observability enhancements — complementary)

🤖 Generated with Claude Code

[Number531]

Closing as superseded.

This was opened 2026-03-10 to eliminate SEC.gov 403 failures via a hybrid fetch_document (direct HTTP → Exa /contents fallback). The simpler all-Exa path was chosen instead: EXA_WEB_TOOLS=true routes all WebFetch through Exa /contents directly.

Production state achieves the same goal:

• EXA_WEB_TOOLS=true in production flags.env since 2026-04-18 (commit 0f37daea, PR v6.0.0 — Wave 1 Observability Release (Institutional Audit Traceability) #76)
• Empirically validated 2026-05-12 (PRs test: citation-verifier A/B harness — tooling-only (verdict marked INVALID per postmortem) #118 + test: production-fidelity citation-verifier A/B — answers the EXA_WEB_TOOLS question (96.8% vs 96.1%) #119): Exa arm 358/370 = 96.8% vs Anthropic arm 340/354 = 96.1% on 467-footnote citation-verifier fixture, both PASS production gate
• G5 observability now in place (T1+T2 via PRs feat(g5-observability): T1 — citation_verdicts table + audit endpoint + WORM export (v6.8.6) #122 / feat(g5-observability): T2 — Prometheus metrics + alerts + structured log (v6.8.7) #124 / docs(skills): align operator skills with T1+T2 telemetry, schema, alerts #125 / docs: T1+T2 residual gaps — api-reference + root CHANGELOG + README + backup verification #126 / fix(g5-observability): T2 metric prefix + otel_trace_id correlation (v6.8.7.1) #127 / docs(changelog): reflect v6.8.7.1 telemetry alignment fix (PR #127) #128): citation_verdicts table + 4 Prometheus series + 3 alert rules + event=citation_verifier_completed Cloud Logging with otel_trace_id Cloud Trace pivot

The hybrid approach proposed here would be an optimization (faster + cheaper for sites that allow direct fetch) over an already-validated production path. Not worth re-litigating after the EXA_WEB_TOOLS path has shipped, validated, and is now under full observability.

If we ever observe Exa cost or latency pressure on the verifier path, we can re-open with the optimization framing.