Non-deterministic graphs from identical inputs: extraction lacks seed/prompt-override controls and extract_corpus_parallel ignores extraction-spec

[reeve-c]

Summary

Running graphify on an identical set of input files produces a materially different graph on every run — different node counts, different node sets, different communities.

Important constraint: the semantic cache is not an acceptable workaround for us. Our runs are independent (fresh/cold cache each time — no shared cache state), so "identical cache → identical output" doesn't apply. We need extraction itself to be reproducible from a cold start, without relying on cache reuse. This report is about the controls needed to make generation deterministic.

Environment

• graphify 0.9.5 (graphifyy==0.9.5)
• Driven headlessly via the Claude Agent SDK running the /graphify <dir> --mode deep skill
• Corpus: 12 Markdown documents (~104k words total; ~8.7k words each), docs-only (no code)
• Backends tested: host Claude subagents; and extract_corpus_parallel with OpenAI gpt-4.1 / gpt-4.1-mini and Gemini gemini-2.5-pro / gemini-2.5-flash, via an OpenAI-compatible endpoint

What I observed

Repeated runs over the exact same files (cold cache each time):

Backend / mode	Nodes across runs	Node-ID overlap (Jaccard), run-vs-run
Host Claude subagents (default temp)	22 / 72 / 100	very low
OpenAI gpt-4.1 (temperature 0)	97 vs 80	~43%
OpenAI gpt-4.1-mini (temperature 0)	84 vs 83	~30%

Everything downstream of extraction looks deterministic (sorted file collection, label-derived IDs, dedup, Leiden with random_seed=42). The variance is entirely in semantic extraction (Pass 3).

Experiments and findings

1. Default path uses non-deterministic subagents. With no GEMINI_API_KEY/GOOGLE_API_KEY set, the skill routes doc extraction to the host agent's subagents, which run at the model's default sampling temperature with no seed — runs diverge wildly (22 → 100 nodes).

2. extract_corpus_parallel (temperature 0) helps but is far from reproducible — ~43% node overlap for gpt-4.1. Two structural causes:

   • Open-ended entity selection — the built-in prompt asks to "extract named concepts and entities"; which spans become nodes varies per run.
   • Free-form labels → derived IDs — "FSD" vs "FSD (Supervised)" vs "Full Self-Driving" yield different IDs for one entity; dedup catches some, not all.

3. extraction-spec.md does not affect the extract_corpus_parallel path. Editing it to make extraction stricter had no effect — that file is only the subagent prompt. extract_corpus_parallel uses the hardcoded _EXTRACTION_SYSTEM in graphify/llm.py, with no env var / param to override it.

4. No seed is sent on the OpenAI-compatible call. _call_openai_compat sets temperature but never seed, so seeded/reproducible sampling can't be pinned through graphify.

5. Reasoning ("thinking") models can't be made deterministic here — gemini-2.5-pro run-to-run (fresh reasoning trace each call).

6. gemini-2.5-flash frequently produced invalid JSON. With no JSON/structured-outpue and were dropped, changing the graph.

7. Subagent chunking is by file count, not tokens — and overflows. skill.md Step B1 grouped 4 large docs into one subagent, which blew the model's 200k-token session limitmid-write; the chunk was lost and the run failed with no graph.json. The extract_corpus_parallel path handles this (_pack_chunks_by_tokens, expand_oversized_files, adaptive bisect_slice), but
   the subagent skill path has none of it.

Requests (to make cold-start extraction reproducible)

1. Overridable extraction system prompt for extract_corpus_parallel — an env var/param to supply a strict, closed-schema prompt (mechanical entity categories, canonical verbatim labels, closed relation
   vocabulary). Currently impossible without patching the package; this is the single bigges
2. Forward a configurable seed to OpenAI-compatible / Anthropic calls, so backends ing can be pinned.
3. Optional JSON / structured-output mode (response_format) for backends that suppolure-driven variance.
4. Token-aware chunking in the subagent skill path (skill.md Step B1) — chunk by wo_corpus_parallel` already does, so large-document corpora don't overflow and silently
   fail.
5. Docs: note that (a) reasoning/thinking models are unsuitable for reproducible extrpec.mdgoverns only the subagent path, notextract_corpus_parallel`.

Minimal repro

1. `graphify extract <dir-of-large-.md-docs> --backend openai --model gpt-4.1 --mode deep
2. Run twice into separate output dirs (cold cache)
3. Compare graph.json node ID sets → ~40–45% Jaccard, different node counts