test_qwen36_spec_decode fails inside the full serial CUDA suite and passes
standalone, on the same binary. It is currently recorded as
"suite-context-dependent" in .agents/specs/triton-aot-default-on.md, and that
label is not yet a finding — the decisive experiment was never run.
What is actually established
Failure, in-suite: CHECK(got_prefix == want_prod) at
tests/.../test_qwen36_spec_decode.cpp:142, 9 assertions / 8 passed. Full serial
run: 97% tests passed, 11 tests failed out of 395, full-serial-exit=8.
Standalone on the same default-ON build it passes 11/11 — as shipped, with all
four GDN Triton realizations disabled together, with each of
chunko/deltah/wu/packeddecode disabled alone, plus repeats.
Why that does not close it
Every arm run was standalone, and none of them reproduces the failure. An arm
that cannot reproduce a failure cannot exonerate anything. The reasoning as
recorded is "the bisect found nothing to bisect, therefore no Triton GDN kernel is
implicated" — which does not follow.
The one decisive comparison was never taken: the full serial suite on
unmodified origin/main. The record correctly rejects an earlier 5-test
origin/main arm as not like-for-like, then treats the absence of a valid
comparison as closure. (That earlier mistake is itself instructive and is
preserved in the spec: an A/B is only an A/B when both arms run in the same
context.)
This matters more than the average flake because Qwen3.6 is a GDN model, so
the vendored Triton kernels genuinely execute in it — 48/48 GDN layers on the 27B
and 30/30 on the 35B fire the Triton chunk_o path at prefill with zero hand-path
fallbacks. It is the one failure in the suite that a kernel-selection change could
plausibly reach, and it is a token-drift assertion, which is exactly the failure
class such a change produces.
Two cross-test-state mechanisms exist only in a default-ON build
Both are recorded in the same spec, and neither was applied to this test:
tests/CMakeLists.txt:974-980 registers test_ops_gdn_aot_concurrent_first_load
only if(VLLM_CPP_TRITON). A default-ON suite therefore runs one extra
Triton test that a default-OFF main suite does not — the sole such guard in
that file.src/vt/cuda/cuda_backend.cu:102-104 runs ReleaseGdnTritonScratch() inside
the generic CUDA DestroyQueue(), which every CUDA test executes. The spec
uses exactly this to withdraw the "GDN-only cubin set" argument for four other
failures, then does not apply it here.
Cheap decisive experiments, none attempted
- Full serial suite on unmodified
origin/main, same box and toolchain, -j 1.
- The default-ON suite with
-E test_ops_gdn_aot_concurrent_first_load.
- That extra test run immediately before the spec-decode gate.
Method note for whoever picks this up
Run these in-suite, not standalone. A standalone re-run of a model gate here
was ~50x slower and reached 59 GB RSS on a 119 GB box (52 s in-suite versus >1 h
alone, killed to avoid an OOM reboot), because the suite leaves the checkpoint
warm in page cache. Use ctest -j 1 — -j 4 OOM-rebooted this box.
Found during #219 / #232. The spec promised this test its own issue; this is it.
test_qwen36_spec_decodefails inside the full serial CUDA suite and passesstandalone, on the same binary. It is currently recorded as
"suite-context-dependent" in
.agents/specs/triton-aot-default-on.md, and thatlabel is not yet a finding — the decisive experiment was never run.
What is actually established
Failure, in-suite:
CHECK(got_prefix == want_prod)attests/.../test_qwen36_spec_decode.cpp:142, 9 assertions / 8 passed. Full serialrun:
97% tests passed, 11 tests failed out of 395,full-serial-exit=8.Standalone on the same default-ON build it passes 11/11 — as shipped, with all
four GDN Triton realizations disabled together, with each of
chunko/deltah/wu/packeddecodedisabled alone, plus repeats.Why that does not close it
Every arm run was standalone, and none of them reproduces the failure. An arm
that cannot reproduce a failure cannot exonerate anything. The reasoning as
recorded is "the bisect found nothing to bisect, therefore no Triton GDN kernel is
implicated" — which does not follow.
The one decisive comparison was never taken: the full serial suite on
unmodified
origin/main. The record correctly rejects an earlier 5-testorigin/mainarm as not like-for-like, then treats the absence of a validcomparison as closure. (That earlier mistake is itself instructive and is
preserved in the spec: an A/B is only an A/B when both arms run in the same
context.)
This matters more than the average flake because Qwen3.6 is a GDN model, so
the vendored Triton kernels genuinely execute in it — 48/48 GDN layers on the 27B
and 30/30 on the 35B fire the Triton
chunk_opath at prefill with zero hand-pathfallbacks. It is the one failure in the suite that a kernel-selection change could
plausibly reach, and it is a token-drift assertion, which is exactly the failure
class such a change produces.
Two cross-test-state mechanisms exist only in a default-ON build
Both are recorded in the same spec, and neither was applied to this test:
tests/CMakeLists.txt:974-980registerstest_ops_gdn_aot_concurrent_first_loadonly
if(VLLM_CPP_TRITON). A default-ON suite therefore runs one extraTriton test that a default-OFF
mainsuite does not — the sole such guard inthat file.
src/vt/cuda/cuda_backend.cu:102-104runsReleaseGdnTritonScratch()insidethe generic CUDA
DestroyQueue(), which every CUDA test executes. The specuses exactly this to withdraw the "GDN-only cubin set" argument for four other
failures, then does not apply it here.
Cheap decisive experiments, none attempted
origin/main, same box and toolchain,-j 1.-E test_ops_gdn_aot_concurrent_first_load.Method note for whoever picks this up
Run these in-suite, not standalone. A standalone re-run of a model gate here
was ~50x slower and reached 59 GB RSS on a 119 GB box (52 s in-suite versus >1 h
alone, killed to avoid an OOM reboot), because the suite leaves the checkpoint
warm in page cache. Use
ctest -j 1—-j 4OOM-rebooted this box.Found during #219 / #232. The spec promised this test its own issue; this is it.