OpenAI backend caps output at 8192 tokens (half the 16384 other backends use) → truncated semantic chunks on dense repos

[django-hewson]

Summary

The openai backend has no max_tokens key in its BACKENDS config (llm.py), so it falls through _resolve_max_tokens(cfg.get("max_tokens", 8192)) to 8192 — while every other backend (claude, gemini, kimi, deepseek, bedrock) is set to 16384. On dense repos/chunks the model's JSON output hits this ~8192-token (~32k-char) ceiling mid-generation (finish_reason == "length"), producing LLM returned invalid JSON … Unterminated string starting at … char ~32000, forcing the chunk-split-retry path. Inferred edges in the truncated tail are lost.

Repro

graphify extract <large-code-repo> --backend openai --token-budget 20000

→ LLM returned invalid JSON, skipping chunk: Unterminated string starting at: line 1 column ~32000
→ chunk of N truncated at depth 0, splitting into halves …

Truncation char positions cluster at 29k–36k chars (≈ 7–9k output tokens — just over 8192).

Impact

• Suppressed INFERRED/semantic edges on big repos (measured: an 8.9k-node repo gained +67 INFERRED edges once the cap was raised; 4 truncation events → 0).
• Wasted tokens on truncated-then-retried chunks.
• Silent inconsistency vs other backends — openai users get half the output headroom, no signal.

Workaround (works today)

export GRAPHIFY_MAX_OUTPUT_TOKENS=16384 restores parity. Bonus: truncated chunks aren't cached (they errored), so a --force re-run only re-does the failed chunks — recovery is cheap.

Suggested fix (RFC — pick the altitude)

1. Parity: add "max_tokens": 16384 to the openai backend config (one line; removes the surprise).
2. Output-aware: chunking is sized by input (--token-budget) but the cap is about output. On finish_reason == "length" (already detected), auto-retry the chunk once at a higher cap before falling back to split — or size chunks by estimated output (≈ f(#entities)).
3. Per-model introspection: derive the cap from the model's actual max_completion_tokens capability rather than a hardcoded default.

Happy to PR option 1. The env override is a fine interim; the ask is (a) fix the default for parity, (b) consider output-aware chunking so dense repos don't silently lose edges.

[safishamsi]

Fixed in 0.8.41 (released today) via #1365 - thanks for the precise root-cause analysis.

Your "option 1" is exactly what shipped: the openai backend config gained "max_tokens": 16384. The fix also went one layer deeper than this report assumed - the OpenAI-compatible dispatch was reading a max_completion_tokens config key that only gemini defines, so ollama, deepseek, and kimi were also silently capped at 8192 despite their configs listing 16384. The dispatch now reads either key, so all of them resolve to their intended 16384 (verified). GRAPHIFY_MAX_OUTPUT_TOKENS still overrides.

graphifyy 0.8.41: openai effective output cap -> 16384

pip install -U graphifyy (or uv tool upgrade graphifyy) to pick it up.

Your option 2 (output-aware chunking / auto-retry at a higher cap before splitting, since chunking is sized by input but the ceiling is about output) is a good separate enhancement - if you'd like to pursue that, a fresh issue scoped to the chunker would be welcome. Closing this one as the cap bug itself is resolved.