CI: end-to-end execution of documentation code blocks

[planetf1]

Add a CI job that extracts Python code blocks from docs and executes them in a sandboxed environment to catch runtime errors (wrong attributes, type mismatches, NotImplementedError, etc.).

Context: PR #601 review found 5 runtime errors (E2, E4, E6, E7, E8) that static checks couldn't detect — wrong method calls, missing attributes, incorrect module paths that only fail at execution time.

Scope:

• Extract fenced python blocks from docs/docs/**/*.md
• Run each in a subprocess with timeout (skip blocks marked as fragments)
• Requires mellea installed + Ollama backend available
• Non-deterministic LLM output is expected — validate no crashes, not output content
• Runs on PR and nightly (backend availability may limit PR runs)

Related: #617 (build process), #636 (Document.parts bug found by this kind of testing)

[jakelorocco]

Is it also feasible to test the code examples in docstrings?

[planetf1]

Investigation summary

Spent time on this properly. Short version: the original scope is harder than it looks, and the right tradeoff is incremental opt-in rather than blanket execution.

What we found

376 Python code blocks across docs/docs/**/*.md. Categorised by runnability:

Category	Count	Notes
Needs a backend (Ollama/HF/API)	191	Most guide examples — call start_session, m.query, etc.
Import-only (no backend call)	~80	Class definitions, @generative stubs, config snippets
Pure fragments	17	Standalone expressions assuming prose context (form.parse(thunk), ctx = ChatContext(window_size=10))
Other	32	Config, output, mixed

What we tried

Opt-in annotation (# pytest: <markers>) — already used on 4 pages in docs/docs/examples/. Built tooling/validate_doc_snippets.py to run annotated blocks as isolated subprocesses. Works, but coverage is 5 blocks total. Doesn't address the original motivation.

Per-file concatenation — take all Python blocks from a page in document order, join and run. Doesn't help: 191 blocks still need Ollama regardless of grouping, and glossary/reference pages have fragments that reference variables defined in prose (NameError noise with no signal value).

Import-only pass — run the ~80 blocks that have no backend calls. Results: 65 pass cleanly, 4 skip (optional deps like docling), 9 noise (NameErrors from fragments, bedrock module reading credentials on import), 2 genuine wrong-mellea-import failures. This is the most signal per effort — no Ollama needed, fast, catches bad module paths.

Conclusion

The original goal (runtime execution of all doc blocks) is only achievable with Ollama available, and even then ~17 blocks are genuine unannotatable fragments. A blanket approach would have a meaningful false-positive rate that erodes trust in the check.

What's in place: tooling/validate_doc_snippets.py provides the execution infrastructure. The # pytest: <markers> convention (already used in docs/docs/examples/) is the practical path — annotate a block when you've verified it's self-contained. Coverage grows as pages are written or reviewed, rather than all at once.

The import-only pass (running ~80 non-backend blocks automatically) is the one addition that would add clear value without Ollama. That's a narrower follow-up if wanted.

Closing as the infrastructure is in place and the scope is better addressed incrementally.