Skip to content

Limit serving engines to Lemonade and vLLM - #79

Merged
rominf merged 2 commits into
mainfrom
remove-nonessential-engines
Jul 7, 2026
Merged

Limit serving engines to Lemonade and vLLM#79
rominf merged 2 commits into
mainfrom
remove-nonessential-engines

Conversation

@rominf

@rominf rominf commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

What

Narrows the supported model-serving backends to Lemonade and vLLM, and removes the standalone atom, llama-cpp, pytorch, and sglang engine integrations (crates, workspace members, path deps, and all serve/registry/inventory/dispatch wiring).

Why

Supporting Lemonade (with the engines it bundles) plus vLLM is a reasonable backend set for now, and it shrinks the maintenance, testing, and packaging surface considerably. vLLM stays a separate first-party backend rather than relying on Lemonade's bundled vLLM.

User-visible changes

  • rocm serve --engine, rocm engines install|shell, and rocm config set-engine|set-default-engine now accept only lemonade and vllm. These values are advertised in --help and shell completion (via a clap value_parser), and an unknown engine is rejected with a clear error:
    error: invalid value 'pytorch' for '--engine <ENGINE>'
      [possible values: lemonade, vllm]
    
  • Built-in model recipes are retargeted off the removed engines: GGUF recipes serve via Lemonade, HF safetensors recipes serve via vLLM, and existing vLLM preferences are unchanged.

Intentionally retained

  • Framework diagnostics for PyTorch and llama.cpp — rocm diagnose / examine still help debug a user's own PyTorch/llama.cpp installs; that is a separate feature from serving backends.
  • Lemonade's own bundled llama.cpp backend, which Lemonade uses internally.

Scope of the change

Beyond the engine crates, this prunes the EngineKind registry, both engine inventories, the TUI engine lists, the local-assistant system prompt, docs (removes the dedicated ATOM/SGLang adapter docs), CI, and the acceptance/smoke scripts on both Linux and Windows.

Verification

cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace (all suites), and scripts/smoke_local.py pass.

Comment thread crates/rocm-engine-protocol/src/lib.rs Dismissed
Comment thread crates/rocm-engine-protocol/src/lib.rs Dismissed
Comment thread crates/rocm-engine-protocol/src/lib.rs Fixed
@rominf
rominf force-pushed the remove-nonessential-engines branch 2 times, most recently from 4147004 to a84f44f Compare July 6, 2026 09:19
Comment thread crates/rocm-engine-protocol/src/lib.rs Dismissed
@rominf

rominf commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Note on the CodeQL check: it flagged 3 rust/path-injection alerts in crates/rocm-engine-protocol/src/lib.rs, all inside the discover_engine_plugins_returns_sorted_unique_plugin_binaries unit test. These are false positives — the flagged path derives from the test helper unique_temp_dir(), which composes a path under std::env::temp_dir() with a pid/nanosecond suffix; all engine/file names are compile-time string literals with no attacker-controlled input. They surfaced only because this PR necessarily edits those test lines (the engines they referenced were removed); the identical env::temp_dir() + fs::write pattern is unflagged elsewhere in the same file. I've dismissed the three alerts as false positives with that rationale.

Narrow the supported model backends to Lemonade (with its bundled engines)
and vLLM, and remove the standalone atom, llama-cpp, pytorch, and sglang
engine integrations. This shrinks the maintenance, testing, and packaging
surface. vLLM is kept as a separate backend since Lemonade's bundled vLLM is
not yet tuned for Instinct.

- Delete the four engine crates and their workspace members, path deps, and
  serve/registry/inventory/dispatch wiring.
- Constrain the user-facing `--engine` / `engines install|shell` /
  `config set[-default]-engine` arguments to `lemonade`/`vllm` via a clap
  value_parser, so `--help` and shell completion advertise exactly those two.
- Retarget built-in model recipes off the removed engines: GGUF recipes serve
  via Lemonade; HF safetensors recipes serve via vLLM; existing vLLM
  preferences are unchanged.
- Prune the EngineKind registry, TUI engine lists, the local-assistant system
  prompt, docs, CI, and acceptance/smoke scripts (Linux and Windows).

Framework diagnostics for PyTorch and llama.cpp (as user frameworks that
`rocm diagnose`/`examine` help debug) and Lemonade's own llama.cpp backend are
intentionally retained.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf
rominf force-pushed the remove-nonessential-engines branch from a84f44f to f74180e Compare July 6, 2026 12:31

@volen-silo volen-silo left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at f74180e (fan-out review over the 54 files + synthesis; findings verified against source). Verification gates green locally: cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace --all-targets (all suites, 0 failed), and scripts/smoke_local.py all pass — confirming the PR's claim. Approving — the removal is complete and clean, no blocking issues.

Removal completeness is compiler-enforced: EngineKind is now exhaustively {Vllm, Lemonade} with no wildcard arm, so nothing could be left silently half-wired, and a clean -D warnings clippy run means no orphaned dead helper survived. Cargo.toml members / apps/rocm/Cargo.toml path-deps / Cargo.lock are all pruned; dispatch bails or safe-defaults on unknown engine strings (no panic/unwrap). Every intended survivor was verified as intentional, not stale removal: PyTorch/llama.cpp diagnostics (examine/diagnose/fix), Lemonade's bundled llama.cpp, PyTorch runtime-dep handling (libatomic/libnuma/OpenMPI) that vLLM + the SDK wheel need, and the "One Dark — Atom" editor theme. The --engine value_parser restriction is correctly wired to all five subcommands (serve, engines install|shell, config set-engine|set-default-engine) with help + completion.

Non-blocking

  1. Recipe metadata mismatch — crates/rocm-core/src/lib.rs:5319. The Qwen/Qwen2.5-1.5B-Instruct recipe (aliases qwen2.5, qwen-small) is retargeted to preferred_engines: ["lemonade"] but keeps loader: "transformers", dtype: "float16", and a safetensors artifact_hint — the only lemonade recipe not marked loader: llamacpp / dtype: gguf. At serve time resolve_lemonade_model_ref silently remaps it to Qwen3-4B-Instruct-2507-GGUF (a ~4 GiB Q4 GGUF), so rocm models shows float16/transformers/1.5B while a 4B GGUF actually loads. Serving works (lemonade remaps internally); the metadata just misreports. Suggest aligning it with its qwen/qwen-gguf siblings: loader: "llamacpp", dtype: "gguf", and an artifact_hint noting the GGUF it resolves to.

  2. Back-compat UX for a stale saved default_engine. default_engine deserializes as a plain string and isn't validated at config load (the value_parser only guards CLI args). An upgrader with default_engine = "pytorch" (or sglang/atom/llama-cpp) hits a clean-but-developer-flavored failure at serve time — engine \pytorch` is not built into this rocm binary/ \"build the workspace or install the engine package\". Not a blocking regression (no panic/corruption;rocm config clear-default-enginerecovers it, and the platform default is valid), but the message offers no migration guidance for an ordinary upgrader. Suggest: afterselect_serve_engine, if the configured default isn't in the inventory, bail with something like \"the saved default_engine '{engine}' is no longer supported; run \rocm config set-default-engine lemonade` to update your config."

  3. Stale engine-list strings.

    • crates/rocm-core/src/lib.rs:5473 — the live qwen3_5 "not a verified PyTorch smoke path" warning (shown via rocm models) now describes a vLLM recipe; reword or drop.
    • Pre-existing, outside this diff, but this PR makes them wrong: apps/rocm/src/serve_summary.rs:15 module doc (lemonade, vLLM, SGLang, PyTorch, llama.cpp, atom), and skills/rocm-cli-assistant/SKILL.md still lists SGLang/PyTorch/llama.cpp as serving engines (would steer the assistant to manage engines that no longer exist). Worth a follow-up trim to lemonade/vllm.

Tradeoffs (deliberate, no change requested)

  • Lemonade's internal safetensors-alias → GGUF remap is what lets the mismatched qwen2.5 recipe "just work"; it hides the metadata drift in #1 until someone inspects rocm models.
  • therock_sdk_install_test.py now asserts on the external_vllm runtime_kind JSON field instead of the old llama-cpp log fragment — correct, slightly more format-coupled.

Positives

Compiler-enforced completeness via the exhaustive EngineKind match; clean single commit whose message explicitly documents the intentional survivors (no AI footer, DCO signed-off); no broken doc links (deleted docs/{atom,sglang}.md have zero inbound refs); README/MANIFEST/AGENTS/engine-plugins engine lists all updated; deleted *_therock_gpu_test.py scripts fully de-referenced from CI; no internal URLs / Jira refs introduced.

Deployment note

Breaking change: removed --engine values and stale saved default_engine values are rejected. Worth a release-note line. No DB/migration steps.

Align the Qwen2.5-1.5B recipe's loader/dtype/artifact_hint with the
GGUF model Lemonade actually resolves it to, and drop stale mentions
of removed serving engines (SGLang, PyTorch, atom, llama.cpp) from a
recipe warning string, the serve-summary module doc, and the CLI
assistant skill doc.

Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
@rominf

rominf commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator Author

Addressed the non-blocking findings in #79 (review) (a6da53d):

  1. Recipe metadata mismatch — fixed. Qwen/Qwen2.5-1.5B-Instruct now uses loader: "llamacpp" / dtype: "gguf" and an artifact_hint noting the internal GGUF resolution, matching the pattern used by its qwen/qwen-gguf siblings.
  2. Stale saved default_engine UX — leaving as-is. The current failure path isn't a panic/data-loss risk and is recoverable via rocm config clear-default-engine; not planning further work here for now.
  3. Stale engine-list strings — fixed all three: the qwen3.5 warning now says "vLLM" instead of "PyTorch", serve_summary.rs's module doc and skills/rocm-cli-assistant/SKILL.md no longer list SGLang/PyTorch/llama.cpp/atom as serving engines.

No CHANGELOG file exists in this repo, and the PR description already documents the breaking --engine value change, so no separate release-note artifact was added.

@rominf
rominf added this pull request to the merge queue Jul 7, 2026
Merged via the queue into main with commit fc2902d Jul 7, 2026
14 checks passed
@rominf
rominf deleted the remove-nonessential-engines branch July 7, 2026 07:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants