[Feature Request] Enhancing Local LLM Performance (Ollama) & High-Core CPU Scaling via graphify.yaml

[RANGERVIII]

I’ve been stress-testing graphify on a high-end local workstation (Ryzen 9 9950X3D, RTX 5090). While the tool is conceptually brilliant, the current hardcoded limits and environment-variable-only configuration create significant bottlenecks for local-first workflows.

Current Bottlenecks Identified:

1. CPU Underutilization: In extract.py, max_workers is capped at 8. On a 32-thread CPU, this limits AST extraction performance by 400%.
2. Local API Timeouts: The default token_budget (60k) and max_concurrency (4) are ideal for Cloud APIs but cause Request timed out or VRAM OOMs on local Ollama/llama.cpp instances during semantic extraction.
3. Authentication Friction: The CLI blocks execution if OLLAMA_API_KEY is missing, even when targeting a local instance where no auth is required.

Proposed Solution: The graphify.yaml Configuration Layer
To support the growing segment of local power users, I suggest introducing an optional configuration file (graphify.yaml) to override defaults persistently.

Example graphify.yaml Structure:

# Performance Tuning
extraction:
  max_workers: 32         # Fully utilize high-end CPUs (9950X3D)
  token_budget: 16384     # Optimized for 24GB+ VRAM (Gemma 4 31b)
  max_concurrency: 1      # Avoid local API queue congestion

# Backend Overrides
backend:
  name: "ollama"
  model: "gemma4:31b"
  base_url: "http://127.0.0.1:11434/v1"
  # Suggested: bypass API_KEY check if backend is Ollama and base_url is loopback

# Model Parameters
parameters:
  temperature: 0.3        # Deterministic graph generation
  seed: 42
  num_ctx: 65536

Why this is a "Win-Win":

• User Experience: Removes the need to re-export set/export commands in every terminal session.
• Scalability: graphify becomes the gold standard for high-performance local knowledge graph extraction.
• Maintainability: Decouples hardware-specific tuning from the core logic.

Verified Performance:
By manually patching these values in my local instance, I achieved 95% GPU saturation on an RTX 5090 and reduced AST extraction time significantly. I’m happy to provide more telemetry or contribute code snippets if this aligns with the project roadmap!

[RANGERVIII]

Addendum: UX, Observability & Hard Timeouts during Local Inference

Further testing on a large corpus (383 docs) has revealed a critical issue regarding connection stability with local backends:

1. The "Hard" HTTP Timeout: Even with a high-end GPU (RTX 5090), processing a 16k token chunk with a 31B model can take longer than the default 60-second timeout of the openai Python client. Since llm.py does not specify a custom timeout during client initialization, the process aborts prematurely: [graphify] chunk 3/26 failed: Request timed out.
2. Lack of Progress Telemetry: There is currently no live progress bar (e.g., via tqdm) for semantic extraction. Users cannot see if a chunk is being processed or if the system is truly hung.
3. Silent Logic/Logs: A missing debug.log or --verbose mode makes it impossible to audit why specific chunks are slow or failing.

Proposed Technical Fixes:

• Configurable Timeout: Expose an api_timeout parameter (defaulting to 600s or None for local backends) in the configuration.
• Progress Bar: Implement a progress bar in extract_corpus_parallel to track chunk completion.
• Persistent Logs: Write an execution log to /graphify-out/ to track which files succeeded and which were lost due to timeouts.

[RANGERVIII]

def _call_openai_compat(
    base_url: str,
    api_key: str,
    model: str,
    user_message: str,
    temperature: float | None = 0,
    reasoning_effort: str | None = None,
    max_completion_tokens: int = 8192,
    *,
    backend: str = "",
) -> dict:
    """Call any OpenAI-compatible API (Kimi, OpenAI, etc.) and return parsed JSON."""
    try:
        from openai import OpenAI
    except ImportError as exc:
        pkg_hint = "graphify[kimi]" if backend == "kimi" else "openai"
        raise ImportError(
            "Gemini/Kimi/Ollama/OpenAI-compatible extraction requires the openai package. "
            f"Run: pip install {pkg_hint}"
        ) from exc

    # EXPLOIT: 10 Minuten Timeout für die RTX 5090 Inferenz
    client = OpenAI(api_key=api_key, base_url=base_url, timeout=600.0)
    
    kwargs: dict = {
        "model": model,
        "messages": [
            {"role": "system", "content": _EXTRACTION_SYSTEM},
            {"role": "user", "content": user_message},
        ],
        "max_completion_tokens": max_completion_tokens,
        "top_p": 0.15,
        "seed": 42,
        "extra_body": {
            "options": {
                "top_k": 10,
                "num_ctx": 65536
            }
        }
    }
    
    if temperature is not None:
        kwargs["temperature"] = temperature
    if reasoning_effort is not None:
        kwargs["reasoning_effort"] = reasoning_effort
        
    # Sauberer Code: extra_body updaten (falls Moonshot), nicht überschreiben
    if "moonshot" in base_url:
        kwargs["extra_body"].update({"thinking": {"type": "disabled"}})
        
    resp = client.chat.completions.create(**kwargs)
    result = _parse_llm_json(resp.choices[0].message.content or "{}")
    result["input_tokens"] = resp.usage.prompt_tokens if resp.usage else 0
    result["output_tokens"] = resp.usage.completion_tokens if resp.usage else 0
    result["model"] = model
    
    # `finish_reason == "length"` means the model hit max_completion_tokens
    # mid-generation. The JSON we got back is truncated.
    result["finish_reason"] = resp.choices[0].finish_reason
    output_tokens = result["output_tokens"]
    
    if output_tokens < 50 and backend == "ollama":
        print(
            "[graphify] warning: ollama returned very few tokens — the model may be "
            "too small or not following the JSON instruction format. "
            "Try a larger model with --model (e.g. --model qwen2.5-coder:14b).",
            file=sys.stderr,
        )
    return result

[safishamsi]

Shipped the underlying fixes to v7 — declining the graphify.yaml format itself since everything you need is now available as CLI flags and env vars, consistent with how the rest of graphify is configured (OLLAMA_BASE_URL, GRAPHIFY_GEMINI_MODEL, etc.). Put them in a .envrc or shell alias for persistence.

What shipped:

• Removed the hard 8-worker AST cap — now min(cpu_count, work_items), override with GRAPHIFY_MAX_WORKERS
• graphify extract --max-workers N — AST parallelism
• graphify extract --token-budget N — semantic chunk size (default 60k tokens)
• graphify extract --max-concurrency N — parallel LLM calls (default 4)
• graphify extract --api-timeout SECONDS — explicit HTTP timeout (default 600s), also GRAPHIFY_API_TIMEOUT
• Fixed the OLLAMA_API_KEY gate for loopback URLs (127.x, localhost, ::1) — no dummy key needed for local Ollama
• Per-chunk progress prints during graphify extract so you can see it's not hung

Declined:

• graphify.yaml — three config surfaces (env / yaml / CLI) with precedence rules adds complexity for zero capability gain
• seed, num_ctx, top_k, top_p knobs — set these server-side via Ollama Modelfile or OLLAMA_NUM_CTX; per-backend hyperparameter exposure is out of scope
• tqdm progress bar — chunk-completion prints cover the 'is it hung?' question without a new dep
• Persistent graphify-out/extract.log — graphify extract . 2>&1 | tee graphify-out/run.log works today

[RANGERVIII]

(.venv) PS I:\Obsidian>                                 
(.venv) PS I:\Obsidian> . 'I:\Obsidian\Graphify_run.ps1'
[Gemma4-Wiki] Starte High-End Inferenz auf RTX 5090...
graphify : error: backend 'ollama' requires OLLAMA_API_KEY to be set.
In I:\Obsidian\Graphify_run.ps1:17 Zeichen:1
+ graphify @Params 2>&1 | Tee-Object -FilePath "graphify-out\lucia_audi ...
+ ~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (error: backend ..._KEY to be set.:String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError
 
[Gemma4-Wiki] Inferenz-Zyklus beendet. Log geschrieben.

[RANGERVIII]

[Bug] Adaptive Retry fails on local LLM overload (0 tokens) / JSONDecodeError --> v0.7.11

Hey @safishamsi,

First of all: Thank you so much for the incredibly fast v0.7.11 update! The new CLI flags (--max-workers, --api-timeout, etc.) are fantastic and make local execution so much smoother. Really appreciate you shipping this so quickly! 🚀

I wanted to report a specific edge case with local Ollama backends that currently breaks the adaptive_retry logic and leads to skipped chunks/data loss.

The Issue:
When a local model is overwhelmed by a complex chunk (e.g., hitting internal VRAM limits or context constraints), it doesn't always gracefully return a finish_reason="length". Very often, it simply chokes and returns an empty string ("" / 0 tokens).

Right now, _parse_llm_json catches this empty string, throws a JSONDecodeError, and logs this constantly:
[graphify] LLM returned invalid JSON, skipping chunk: Expecting value: line 1 column 1 (char 0)

Because the chunk is immediately dropped as "invalid JSON", the _extract_with_adaptive_retry mechanism never triggers. The chunk isn't split into smaller halves; it just gets completely skipped.

How to reproduce (without needing a high-end GPU):
You can easily simulate this hardware-choke behavior on any standard machine. Just run graphify extract using a very small model (like qwen2.5:0.5b or llama3.2:1b) and force a massive chunk size, e.g., --token-budget 60000.
The tiny model will instantly choke on the huge context, return an empty string, and you'll see the JSON errors flood the terminal instead of the expected chunk-splitting behavior.

Suggested Fix:
If the generated response contains 0 tokens or an empty string, it would be awesome if the script could catch this and treat it as a trigger for the adaptive retry (effectively forcing the chunk to split), rather than just skipping the files.

Thanks again for building such a great tool and being so responsive!

[safishamsi]

Yeah I haven’t rotated the PyPI version yet, that’s the reason you’re hitting the same error let me do it now, was working on more additions and let me know if you hit the same issue

…

On Sat, 9 May 2026 at 20:44, RANGERVIII ***@***.***> wrote: *RANGERVIII* left a comment (Graphify-Labs/graphify#792) <#792 (comment)> (.venv) PS I:\Obsidian> (.venv) PS I:\Obsidian> . 'I:\Obsidian\Graphify_run.ps1' [Gemma4-Wiki] Starte High-End Inferenz auf RTX 5090... graphify : error: backend 'ollama' requires OLLAMA_API_KEY to be set. In I:\Obsidian\Graphify_run.ps1:17 Zeichen:1 - graphify @params <https://github.com/params> 2>&1 | Tee-Object -FilePath "graphify-out\lucia_audi ... - + CategoryInfo : NotSpecified: (error: backend ..._KEY to be set.:String) [], RemoteException + FullyQualifiedErrorId : NativeCommandError [Gemma4-Wiki] Inferenz-Zyklus beendet. Log geschrieben. — Reply to this email directly, view it on GitHub <#792 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BTSTP6Z2SX5DXB5KR2PK6NL4Z6DADAVCNFSM6AAAAACYXF4TQGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DIMJTGUYTQMJXGQ> . You are receiving this because you commented.Message ID: ***@***.***>

[safishamsi]

Fixed in v0.7.12 — added _response_is_hollow() predicate that detects empty/null/zero-token Ollama responses and routes them through the existing adaptive bisection retry path. Also added --max-workers, --max-concurrency, --token-budget, and --api-timeout flags to graphify extract for CPU/concurrency tuning.

[RANGERVIII]

[Critical] Context Window Saturation / Missing Session Reset between Chunks (Ollama Backend)

Hey @safishamsi,

Quick follow-up after testing v0.7.12 on a high-end local setup (RTX 5090, Gemma 4:31b).

We’ve identified a major bottleneck in how the Ollama backend handles multiple chunks. After about 3-4 chunks, the model starts returning the "very few tokens" warning and essentially stops producing valid JSON.

The Diagnosis: It appears that graphify is not explicitly resetting the Ollama session/context between chunks.

What's happening:
In local inference (especially via Ollama’s /api/generate or /api/chat), if the context parameter (the KV cache) isn't cleared or handled statelessly for each new chunk, the model "saturates." It carries the noise and tokens from Chunk 1 and 2 into the context window for Chunk 3. Even with a large 5090 VRAM, the attention mechanism gets overwhelmed by the uncleared history, leading to the "0-token" or "closing-bracket-only" responses we are seeing.

The Evidence:

• Chunk 1-2: Fast, perfect inference.
• Chunk 3-4: Sudden drop in token count, "few tokens" warning, but the GPU is still under load (processing the saturated context).
• Restarting the script: Fixes it immediately for the first few chunks again.

The Fix:
Each new chunk or file extraction call to the Ollama API should be stateless. Please ensure that for every new extraction call, the context array is either empty or the session is explicitly reset. A simple flag or a fresh request object per chunk would solve this and prevent the "context-leak" that currently ruins long extraction runs.

Looking forward to v0.7.13! 🚀