Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions tests/e2e-cucumber/expectations.toml
Original file line number Diff line number Diff line change
Expand Up @@ -126,9 +126,11 @@ serve_timeout_secs = 90
# On a lemonade LINUX host this needs a live lemonade serve, which reaches ready
# then shuts down immediately (EAI-7423). os=linux: passes on native Windows
# lemonade (run 29354305288 XPASS'd it), consistent with the other lemonade
# serve scenarios.
# serve scenarios. therock_family=gfx* scopes this to a REAL GPU host: the
# scenario now also runs on the no-GPU mock lane (backed by MockServer, not a
# real lemonade serve), where EAI-7423 does not apply and it must expect-pass.
[["chat-tool-definitions-accepted"]]
when = { effective_engine = "lemonade", os = "linux" }
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the chat request never reaches a live model."
serve_timeout_secs = 90
Expand All @@ -140,9 +142,11 @@ serve_timeout_secs = 90
# then 3/3 green re-runs), so there is NO vLLM xfail for this scenario — the CLI chat
# path works. Only the lemonade LINUX path stays xfail: it needs a live lemonade
# serve, which reaches ready then shuts down immediately (EAI-7423); native Windows
# lemonade serve works (run 29338475133 XPASS).
# lemonade serve works (run 29338475133 XPASS). therock_family=gfx* scopes this to
# a REAL GPU host: the scenario also runs on the no-GPU mock lane (MockServer, not
# a real lemonade serve), where EAI-7423 does not apply and it must expect-pass.
[["chat-end-to-end-local-model"]]
when = { effective_engine = "lemonade", os = "linux" }
when = { effective_engine = "lemonade", os = "linux", therock_family = "gfx*" }
bug = "EAI-7423"
reason = "Lemonade managed serve reaches ready then shuts down immediately, so the chat request never reaches a live model."
serve_timeout_secs = 90
Expand Down
12 changes: 10 additions & 2 deletions tests/e2e-cucumber/features/chat.feature
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,22 @@ Feature: Chat and endpoint detection
When the user lists running services
Then the served model endpoint is listed

@id:chat-tool-definitions-accepted @requires-gpu
# Runs on every lane: on a GPU host `a model is served in the background` does a
# real `rocm serve`, on the no-GPU mock lane it's backed by MockServer. The
# assertion (a tools-bearing request is accepted) is engine-agnostic, so no GPU
# is required — dropping @requires-gpu gives this per-PR mock-lane coverage.
@id:chat-tool-definitions-accepted
Scenario: 5 - Chat requests that include tool definitions are accepted
Given a managed runtime is active
And a model is served in the background
When a chat request with tool definitions is sent
Then the chat response is successful

@id:chat-end-to-end-local-model @requires-gpu
# Runs on every lane (see scenario 5): real serve on a GPU host, MockServer on
# the no-GPU mock lane. Asserts only that a served model returns a non-empty
# reply, which is engine-agnostic — real generation is covered by the
# @requires-gpu serve-*-inference scenarios.
@id:chat-end-to-end-local-model
Scenario: 6 - End-to-end chat through a locally served model
Given a managed runtime is active
And a model is served in the background
Expand Down
4 changes: 3 additions & 1 deletion tests/e2e-cucumber/features/model_serving.feature
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,9 @@ Feature: Model serving

# Honest device selection: a `--gpu` index that does not exist on the host is
# rejected outright, never silently remapped to another device (no device-0
# fallback). Runs on GPU hardware.
# fallback). Runs on GPU hardware: on a no-GPU host the GPU-required pre-flight
# refuses ("no usable AMD GPU") before the index is ever validated, so the
# index-specific rejection can only be observed where a real device is present.
@id:serve-absent-gpu-index-rejected @requires-gpu @requires-os:linux
Scenario: 13 - Serving pinned to a GPU that does not exist is refused
When the user serves a model pinned to a GPU index that does not exist
Expand Down
9 changes: 9 additions & 0 deletions tests/e2e-cucumber/tests/e2e/runtime_steps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,15 @@ async fn setup_standard_rocm(_world: &mut E2eWorld) {}

#[given("a managed runtime is active")]
async fn setup_active_runtime(world: &mut E2eWorld) {
// On a no-GPU host this is a no-op: a managed TheRock SDK runtime can only be
// installed where there's a GPU family to select wheels for (see
// `runtime-install-active`, @requires-gpu). The only scenarios that reach this
// step without `@requires-gpu` are the mock-lane chat scenarios, whose serve
// is backed by MockServer and needs no runtime at all — so skip the install
// rather than attempt a multi-GiB SDK pull that can't succeed here.
if !e2e_cucumber::capability::host_capability().has_amd_gpu {
return;
}
// This precondition only needs *a* runtime present — it does not assert a
// clean slate — so opt into the shared runtimes tree: the first scenario to
// hit an empty shared tree installs once, and every later scenario finds the
Expand Down
20 changes: 19 additions & 1 deletion tests/e2e-cucumber/tests/e2e/serving_steps.rs
Original file line number Diff line number Diff line change
Expand Up @@ -412,7 +412,25 @@ async fn setup_large_gpu_model(world: &mut E2eWorld) {

#[given("a model is served in the background")]
async fn setup_background_model(world: &mut E2eWorld) {
setup_gpu_model(world).await;
// This precondition backs behavioural chat scenarios (tool-defs accepted,
// end-to-end reply) whose When/Then talk to the served endpoint over HTTP and
// never assert on real generation — only that a served model answers. So it
// doesn't need a real GPU serve: on a host WITH an AMD GPU we still exercise
// the real `rocm serve --managed` path (extra real coverage where hardware
// exists), but on a no-GPU host we back it with the in-process MockServer +
// a planted managed-service record. This lets the scenarios drop
// `@requires-gpu` and run on the GitHub-hosted mock lane every PR, with no
// coverage loss. Real inference stays covered by the `@requires-gpu`
// serve-*-inference scenarios.
if e2e_cucumber::capability::host_capability().has_amd_gpu {
setup_gpu_model(world).await;
} else {
let mock = MockServer::start("TestModel/E2E-1B").await;
world.endpoint = Some(mock.base_url());
world.model_name = Some("TestModel/E2E-1B".to_string());
world.mock = Some(mock);
world.register_mock_service();
}
}

#[given("the served model has been detected")]
Expand Down
Loading