test(e2e): harden chat checks and CI gating - #119
Conversation
a11f4cb to
dec0f4b
Compare
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
dec0f4b to
6b269f0
Compare
volen-silo
left a comment
There was a problem hiding this comment.
Reviewed against the surrounding harness and ran the new unit test (cargo test -p e2e-cucumber --lib → 24 passed). All three changes are correct, focused, and low-risk. LGTM with one minor, non-blocking nit.
✅ Looks good
ci.yml— addingtests/e2e-cucumber/expectations.tomlto theheavyfilter is right: the e2e lanes gate onneeds.changes.outputs.heavy == 'true', and an xfail-matrix edit should re-run them to validate the new expectations. Placed alongside**/*.feature, its natural sibling. (The other two files are.rs/already covered byheavy, so this closes the expectations-only-change gap.)chat_response_is_successful+ unit test — clean helper; the test covers the meaningful cases (missing key,null, non-array, empty array, non-empty array). Tighteningassert_chat_successfulfromchoices.is_some()to "non-emptychoicesarray" is strictly better and won't false-negative a genuine completion (a valid OpenAI-style response always has a non-emptychoices).- timeout on the tool-calling POST — mirrors
send_chat'sinference_timeout_for(world), so a hung tool request fails fast instead of blocking to the job cap.crate::inference_timeout_forresolves fine from the submodule.
🟡 Minor (non-blocking): the models-discovery GET in the tool path is still un-timed
In send_chat_with_tools, only the chat POST got the timeout client — the preceding reqwest::get(&models_url) (chat_steps.rs:45) is still the bare, timeout-less call. Its sibling send_chat (e2e.rs:543–556) runs both the models GET and the chat POST through the timeout client. So a stalled /v1/models could still hang the harness here, and the PR's "apply the timeout to tool-calling requests" goal is only partially met.
Low impact in practice (serve readiness is awaited separately, and /v1/models returns immediately once ready), but for consistency with send_chat consider routing the models GET through the same client:
let client = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(crate::inference_timeout_for(world)))
.build()
.expect("failed to build HTTP client");
let resp: serde_json::Value = client.get(&models_url).send().await? ... ;
// reuse `client` for the POST belowBoth hardened scenarios (4, 5) are @requires-gpu, so this only affects the GPU lanes, not the mock gate.
Signed-off-by: Roman Inflianskas <Roman.Inflianskas@amd.com>
|
Addressed the review nit in 9702e95: the tool-calling path now constructs the timeout-configured client before model discovery and reuses it for both the |
Summary
choicesarrayexpectations.tomlchangesFollow-up to #69, now rebased onto
main.Risk: low — this only tightens E2E assertions and CI selection.
Test plan
cargo fmt --all -- --checkcargo clippy -p e2e-cucumber --all-targets -- -D warningscargo test -p e2e-cucumber --lib