Hardcoded max_tokens=8192 causes truncation cascade on dense docs (3× cost overhead)

[himanshup-locus]

Hardcoded max_tokens=8192 causes truncation cascade on dense docs (3× cost overhead)

TL;DR

graphify extract's Claude backend uses max_tokens=8192 in _call_claude (graphify/llm.py). On dense Markdown corpora, output JSON regularly exceeds this cap. The adaptive-retry layer correctly catches finish_reason="length" and recursively splits the chunk — but the truncated API call is fully billed (input tokens + 8192 output tokens) and produces zero usable nodes/edges. The same content then gets re-extracted from scratch in smaller chunks.

For our docs-only repo (1,094 Markdown files, mostly entity specs and PRDs averaging 2–8 KB body content), running graphify extract on the ~74 uncached files cost $4.31 when an honest cost estimate is closer to $1.50 — roughly 3× overhead from truncated-then-discarded calls.

Reproduction

Corpus profile:

• 1,094 Markdown docs, 1 PDF, 12 images
• Cache hit rate: 93% (preflight verified)
• Genuinely uncached: ~74 files
• Files are dense PRD/entity-spec content (typical 2–8 KB body each)

Command: graphify extract docs/ --backend claude --out .

Observed: every initial chunk of 14–22 files triggers finish_reason="length" at depth 0 because the JSON output for that many dense docs blows past 8192 tokens. Adaptive retry splits them; depth-1 chunks of 7–11 files frequently truncate again. By depth 2 (chunks of 3–5 files), most succeed. A handful hit depth 3 (single-file chunks) and graphify gives up with a partial result.

Sample log lines:

[graphify] LLM returned invalid JSON, skipping chunk: Expecting value: line 1 column 28711 (char 28710)
[graphify] chunk of 19 truncated at depth 0, splitting into halves of 9 and 10
[graphify] chunk of 22 truncated at depth 0, splitting into halves of 11 and 11
[graphify] chunk of 14 truncated at depth 0, splitting into halves of 7 and 7
[graphify] chunk of 9 truncated at depth 1, splitting into halves of 4 and 5
[graphify] LLM returned invalid JSON, skipping chunk: ... (depth 2)
[graphify] LLM returned invalid JSON, skipping chunk: Expecting property name enclosed in double quotes: line 886 column 37 (char 25096)
[graphify] single-file chunk docs/README.md truncated at max_completion_tokens — partial result kept

Final result for the run:

[graphify extract] tokens: 267,749 in / 234,087 out, est. cost (~claude): $4.3146

234K output tokens is suspicious — it's nearly 1:1 with input, when extraction output should typically be 30–40% the size of input for a clean run. The ratio implies the LLM repeatedly produced 8192-token responses that were thrown away.

Why it matters

We're using graphify extract as the canonical CI publisher of the corpus graph (the workflow regenerates and commits graph.json on every merge to master, per your "graph is a build artifact" guidance). In that model:

• First run: bears the full cost of populating cache. For our corpus this is once — but the truncation overhead made it 3× more expensive than necessary.
• Subsequent runs: only re-extract changed files. If a single dense entity-spec changes, the truncation cascade fires again on that file's chunk, paying overhead per merge.

Over a year of normal docs PRs (50–100 merges that touch denser specs), the cascade overhead compounds. Each merge that hits truncation pays an extra $0.30–$1.50 above the "should-cost" of extracting the changed files cleanly.

Root cause

In graphify/llm.py, _call_claude:

def _call_claude(api_key: str, model: str, user_message: str) -> dict:
    ...
    resp = client.messages.create(
        model=model,
        max_tokens=8192,   # ← hardcoded
        ...
    )

The chunk packer (_pack_chunks_by_tokens) defaults to token_budget=60_000 for input sizing. This input budget assumes a ~10:1 input/output ratio, but for dense extraction the ratio is closer to 4:1. Result: input fits the budget cleanly, but output regularly exceeds 8192.

Sonnet 4.6 supports up to 64K output tokens; Sonnet 3.7 up to 128K. The 8192 cap is far below what the model supports.

Proposed fixes (based on my limited understanding of Graphify intents)

1. Bump default max_tokens for the claude backend (lowest-effort, biggest win)

# graphify/llm.py
BACKENDS = {
    "claude": {
        ...
        "max_tokens": 16384,   # was implicit 8192
    },
    ...
}

def _call_claude(api_key: str, model: str, user_message: str, max_tokens: int = 16384) -> dict:
    ...
    resp = client.messages.create(model=model, max_tokens=max_tokens, ...)

16K is conservative (well under model limits) and would fit the vast majority of our chunks in a single call. 32K would be even better and still safe.

2. Make max_tokens configurable

Either as a CLI flag (--max-output-tokens N) or environment variable (GRAPHIFY_MAX_OUTPUT_TOKENS). Lets users tune per corpus density.

3. Proactive chunk shrinking based on observed output size

When a chunk truncates, halve the input budget for subsequent chunks in the same run instead of just splitting that one chunk. This is a heuristic — if one dense chunk truncates, others probably will too. Avoids paying the truncation cost N times.

4. Stream + parse incrementally

Stream the LLM response, parse JSON tokens as they arrive. If the stream ends mid-string, salvage everything parsed so far instead of throwing away the entire response. This is more invasive but recovers the wasted output tokens.

Workaround we're using

Pinning graphifyy>=0.7.4, accepting the one-time cost, watching the trend on subsequent merges. We've added a CI preflight gate (check_semantic_cache against expected hit rate) that catches cache-key drift early but doesn't help with truncation.

Other observations from the run

• The [graphify extract] semantic cache: X hit / Y miss log line never appeared in our run despite 93% cache hits being confirmed externally via preflight. Possibly a stdout buffering issue under GitHub Actions runners — would help debugging if it always flushed before the first chunk fires.
• pip install graphifyy does not pull in anthropic as a hard dep. The error message is clear (Run: pip install anthropic) but listing it as an optional extra (graphifyy[claude]) would be more discoverable.

Happy to provide more data or test patches.

[safishamsi]

Fixed in v0.7.6 — default max_tokens raised from 8192 to 16384 for both claude and kimi backends. Override with GRAPHIFY_MAX_OUTPUT_TOKENS env var.

[himanshup-locus]

Thanks for the quick fix — the env var override is great. Upgraded and will rerun.

I want to flag a deeper observation before we close this out, though, because I think max_tokens isn't the root cause — it's a knob that papers over a chunker mismatch.

What we actually saw in the run. Multiple chunks hit finish_reason=length even at recursion depth 3 — including a single-file chunk on docs/README.md. The log line was:

[graphify] single-file chunk docs/README.md truncated at max_completion_tokens — partial result kept

A single-file truncation means the file's own extracted JSON exceeds 8192 tokens. Bumping to 16K probably saves this case. But it implies that for at least some files in dense corpora, no fixed max_tokens is safe — there will always be a file dense enough to push beyond whatever ceiling is set. We can't keep raising the cap forever; output tokens cost real money and add latency, and at some point you bump into the model's hard ceiling (64K for Sonnet 4.6, 128K for 3.7).

The deeper issue, I think, is in how _pack_chunks_by_tokens sizes work:

1. It packs by input tokens (default 60K). That implicitly assumes a roughly 10:1 input:output ratio — fine for sparse content, broken for dense entity-spec / PRD content where the LLM emits dozens of nodes + edges per file.
2. The adaptive retry treats truncation as a per-call accident and reacts after the fact, paying for the truncated call (full input + 8K wasted output) before splitting. For a corpus where most chunks truncate at depth 0, this overhead compounds.

A few directions that would address the root rather than the cap:

• Pack by an estimated output budget, not input budget. E.g., heuristic based on file character count + structural density (count of headers, tables, code blocks). Calibrate from observed output:input ratios in cache.
• Pre-split files whose estimated output exceeds the output budget. For Markdown, splitting by ## sections is usually safe. The cache key would need to include the section identifier, but that's mostly clean.
• Adapt proactively in-run: if the first chunk in a run truncates, halve the chunk size for all remaining chunks instead of just splitting that one. One bad chunk usually predicts more.
• Stream + parse incrementally: when a stream ends mid-string, salvage everything up to the last complete {...} instead of throwing the whole response away. This recovers the wasted output tokens.

For our corpus (1,000+ dense Markdown files), the v0.7.6 16K cap will absorb most cases but I expect we'll still see depth 3 failures on the 5–10% densest files. We'll keep an eye on it after upgrading and report back. Just wanted to flag this so the long-term direction is clear — happy to provide more data or test patches.

Either way, thanks again for the responsive fix.