Skip to content

Let Lemonade auto-select its llama.cpp backend - #3

Closed
rominf wants to merge 1 commit into
powderluv:jam/updatesfrom
rominf:rominf/lemonade-auto-backend
Closed

Let Lemonade auto-select its llama.cpp backend#3
rominf wants to merge 1 commit into
powderluv:jam/updatesfrom
rominf:rominf/lemonade-auto-backend

Conversation

@rominf

@rominf rominf commented Jun 10, 2026

Copy link
Copy Markdown

Summary

  • The Lemonade adapter hard-forced lemonade load --llamacpp rocm with no fallback. On GPUs the bundled ROCm build does not support (e.g. the Radeon 780M / gfx1103 iGPU), Lemonade reports llamacpp:rocm as "Unsupported GPU", so every managed load returned a body-less 500 and spun in a load/evict/unload loop. Standalone Lemonade serves the same model because it falls back to Vulkan.
  • Query lemonade backends, pick the best supported llama.cpp backend (rocm > vulkan > cpu), install it on demand, and load with it instead of hardcoding rocm.
  • Backend is resolved at install and re-resolved at serve time, so existing installs self-heal without a reinstall. Health verification, persisted state, and adapter messaging now follow the selected backend rather than assuming rocm.

Changes (engines/lemonade/src/lib.rs)

  • run_lemonade_backends_list + parse_llamacpp_backend_statuses parse the lemonade backends table for the llamacpp recipe.
  • select_best_llamacpp_backend chooses the highest-priority supported backend (LLAMACPP_BACKEND_PRIORITY).
  • ensure_best_llamacpp_backend installs the chosen backend if needed; used by both install (install_best_llamacpp_backend) and serve_http.
  • lemonade_model_load_args / run_lemonade_model_load pass the selected backend; --force dropped from backend install so only supported backends install.
  • Health check (health_has_loaded_model / lemonade_backend_matches), state fields, and detect/install/resolve copy reflect the selected backend.

Test plan

  • cargo test -p rocm-engine-lemonade (15 passing, incl. new parser/selection tests)
  • cargo build --release -p rocm
  • Manual: restarted a previously-500-looping Qwen3-4B service; log shows Installing backend: llamacpp:vulkan, Using LlamaCpp Backend: vulkan, model loaded on AMD Radeon 780M Graphics, chat smoke 200 OK (~10.5 tok/s); services list reports ready.

🤖 Generated with Claude Code

The Lemonade adapter forced `lemonade load --llamacpp rocm` with no fallback.
On GPUs the bundled ROCm build does not support (e.g. the Radeon 780M /
gfx1103 iGPU), Lemonade reports `llamacpp:rocm` as "Unsupported GPU", so every
managed load returned a body-less 500 and spun in a load/evict/unload loop.
Standalone Lemonade serves the same model fine because it falls back to Vulkan.

Now query `lemonade backends`, choose the best supported llama.cpp backend
(rocm > vulkan > cpu), install it on demand, and load with it instead of
hardcoding rocm. The backend is resolved at install and re-resolved at serve
time, so existing installs self-heal without a reinstall. Verification, state,
and adapter messaging follow the selected backend rather than assuming rocm.

Co-Authored-By: Claude Opus 4 <noreply@anthropic.com>
@rominf
rominf marked this pull request as ready for review June 10, 2026 10:56
@rominf rominf closed this Jun 12, 2026
@rominf
rominf deleted the rominf/lemonade-auto-backend branch June 12, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant