Skip to content

fix(distributed): stage backend companion assets to remote nodes - #10330

Merged
mudler merged 1 commit into
masterfrom
fix/distributed-stage-companion-dirs
Jun 14, 2026
Merged

fix(distributed): stage backend companion assets to remote nodes#10330
mudler merged 1 commit into
masterfrom
fix/distributed-stage-companion-dirs

Conversation

@localai-bot

Copy link
Copy Markdown
Collaborator

Problem

In distributed inference, a model whose ModelFile is a single file (e.g. sherpa-onnx VITS/piper, where parameters.model is the .onnx) fails to load on a remote worker. The backend resolves sibling assets from the model directory — tokens.txt, lexicon.txt, the espeak-ng-data / dict directories, Kokoro's voices.bin — but staging only ships the declared ModelFile. The worker then hits:

failed to create sherpa-onnx TTS engine from /models/.../it_IT-paola-medium.onnx

and TTS produces no audio. (countStageableFiles returns 1 for a regular file; whole-directory expansion only happens when ModelFile itself is a directory.)

Fix

Lean on the existing option-path staging (vae_path:-style) rather than hardcoding filenames in the router:

  • stageGenericOptions resolves an option value relative to the model's own directory as well as the frontend models dir, so a shared config can declare companions with bare names regardless of whether Model includes a subdirectory (new resolveOptionPath helper).
  • It now expands directory-valued options (e.g. espeak-ng-data) file-by-file via stageOptionDir, instead of handing a directory fd to the stager.
  • gallery/sherpa-onnx-tts.yaml declares the companion assets as option paths once for all sherpa TTS voices (tokens, lexicon, espeak-ng-data, voices.bin, dict, per-language lexicons). The backend ignores these keys and keeps resolving siblings from the model dir; they exist only so distributed staging ships them. Files a given voice doesn't ship are skipped.

Tests

  • New core/services/nodes/router_optionstage_test.go covers staging an option-declared sibling file and expanding a directory option via the model-dir fallback.
  • Full core/services/nodes suite passes; go vet clean.

🤖 Generated with Claude Code

A model whose ModelFile is a single file (e.g. sherpa-onnx VITS/piper: the
.onnx) failed to load on remote worker nodes because the sibling assets the
backend resolves from the model dir — tokens.txt, lexicon.txt, the
espeak-ng-data / dict directories, Kokoro's voices.bin — were never staged.
Only the declared ModelFile was shipped, so the worker hit "failed to create
sherpa-onnx TTS engine" and TTS produced no audio.

Lean on the existing option-path staging instead of hardcoding filenames:

- stageGenericOptions now also resolves an option value relative to the model's
  own directory (not just the frontend models dir), so a shared config can
  declare companions with bare names regardless of whether Model includes a
  subdirectory; and it expands directory-valued options (e.g. espeak-ng-data)
  file-by-file rather than handing a directory fd to the stager.
- gallery/sherpa-onnx-tts.yaml declares the companion assets as option paths
  (tokens, lexicon, espeak-ng-data, voices.bin, dict, per-lang lexicons). The
  backend ignores these keys and keeps resolving siblings from the model dir;
  they exist only so distributed staging ships them. Absent files are skipped.

Adds router_optionstage_test.go covering file + directory companion staging via
the model-dir fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@mudler
mudler merged commit e5c95e0 into master Jun 14, 2026
58 of 59 checks passed
@mudler
mudler deleted the fix/distributed-stage-companion-dirs branch June 14, 2026 14:43
@localai-bot localai-bot added the bug Something isn't working label Jun 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants