make test-all: vLLM suites fail with FlashAttention CUDA kernel-image mismatch (main + feature branch)

[AlonMalach]

Summary

make test-all fails the entire vLLM GPU suite on both main and feature/reduce-control-token-kv-overhead with the same root cause. The CPU suites (unit / composer / hf) pass on both branches.

Branch	Result	Duration
feature/reduce-control-token-kv-overhead	374 failed, 47 passed, 1 error	1:20:01
main	377 failed, 50 passed	1:26:17

Why the pass/fail counts differ between branches (it's not a regression)

The 50-vs-47 pass gap and the 377-vs-374 failure gap come entirely from test-collection differences, not from any behavioural divergence. The feature branch deliberately removes the legacy KV-hiding code path (60920bb "Remove the legacy KV-hiding code path", bd0f893 "Replace control-token KV hiding with token-exchange by default"), and the tests that exercised that removed code path were deleted along with it.

git diff main..feature/reduce-control-token-kv-overhead --stat -- tests/vllm/ shows ~734 lines of test code deleted, including:

• tests/vllm/test_position_zero_nan.py and its helper _position_zero_nan_tests.py (~474 lines) — entire file removed
• tests/vllm/test_kv_hiding_gap_equivalence.py and its helper _kv_hiding_gap_tests.py — removed
• TestControlTokenKVInvisibility class in tests/vllm/test_model_forward.py — removed

Mapping that to the failure-list deltas:

Test failing only on main	Status on feature branch
test_position_zero_nan.py::TestModelFiniteness::test_suite	File deleted (legacy code path)
test_position_zero_nan.py::TestFixSensitivity::test_suite	File deleted (legacy code path)
test_model_forward.py::TestControlTokenKVInvisibility::test_suite	Class deleted (legacy code path)
test_single_switch.py::TestGeometry::test_geometry_matches_config	Exists on both branches — see note below
Test failing only on feature branch	Status on main
test_single_switch.py::TestFallbackGeometry::test_config_overrides_fallback	Exists on both branches — see note below

Note on TestGeometry / TestFallbackGeometry: both classes/tests exist on both branches. They appear in only one branch's failure list because the worker restarts between groups of tests, and the first test in a worker surfaces the CUDA crash as RuntimeError: Worker died unexpectedly, while later tests in the same worker surface it as cascading BrokenPipeErrors. Which test happens to be "first" varies run-to-run — this is not a behavioural difference.

The 3-test gap in pass count is similarly explained by CPU-suite test-collection cleanup on the feature branch (dead-class purge, removed legacy fixtures), not by tests that pass on main and fail on the feature branch. No real regression is visible in either run — and importantly, none can be inferred either way, because:

No vLLM forward pass actually executes successfully on either branch in this environment. Every vLLM test crashes before reaching its assertions. We cannot conclude anything about behavioural pass/fail divergence between the branches until the environment is fixed.

Root cause (identical on both branches)

INFO [cuda.py:334] Using FLASH_ATTN attention backend out of potential backends:
                  ['FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION'].
INFO [flash_attn.py:596] Using FlashAttention version 2
CUDA error (/workspace/.deps/vllm-flash-attn-src/hopper/flash_api.cpp:697):
  no kernel image is available for execution on the device

The vllm-flash-attn build in this environment ships only Hopper (SM 9.0) kernels, but the GPU at runtime is a different architecture, so the kernel launch fails immediately. vLLM auto-selects FLASH_ATTN even though three other backends (FLASHINFER, TRITON_ATTN, FLEX_ATTENTION) are listed as available.

This is an environment / build problem — not a regression introduced by the feature branch.

Failure shapes (same on both branches)

Three traceback patterns appear, all rooted in the same crash:

1. First test in a worker — RuntimeError surfacing the CUDA error directly:

   tests/vllm/test_single_switch.py:101: in _send_request
       raise RuntimeError(f"Worker died unexpectedly:\n{stderr}")
   RuntimeError: Worker died unexpectedly:
   ...
   CUDA error (.../hopper/flash_api.cpp:697): no kernel image is available for execution on the device
   

2. Subsequent tests in the same worker — BrokenPipeError when writing to the dead stdin:

   tests/vllm/test_single_switch.py:95: in _send_request
       _worker_proc.stdin.write(json.dumps(req) + "\n")
   BrokenPipeError: [Errno 32] Broken pipe
   

3. Geometry helpers — same BrokenPipeError, via _query_geometry() at line 114.

Affected test files

main branch — 3 files

• tests/vllm/test_model_forward.py
  • TestForwardOutputShape::test_suite
  • TestAdapterIndicesWiring::test_suite
  • TestControlTokenKVInvisibility::test_suite
  • TestKVVisibility::test_suite
• tests/vllm/test_position_zero_nan.py
  • TestModelFiniteness::test_suite
  • TestFixSensitivity::test_suite
• tests/vllm/test_single_switch.py — full parametrized suite across:
  • TestTokenMatching, TestAdapterRetrieval, TestEdgeCases,
    TestShapeCorrectness, TestContextLengthSweep, TestGainSensitivity,
    TestGeometry, TestFA3Boundary, TestKVCacheShape

feature/reduce-control-token-kv-overhead branch — 2 files

• tests/vllm/test_model_forward.py
  • TestForwardOutputShape::test_suite
  • TestAdapterIndicesWiring::test_suite
  • TestKVVisibility::test_suite
• tests/vllm/test_single_switch.py — full parametrized suite across:
  • TestTokenMatching, TestAdapterRetrieval, TestEdgeCases,
    TestShapeCorrectness, TestContextLengthSweep, TestGainSensitivity,
    TestFA3Boundary, TestKVCacheShape, TestFallbackGeometry

Reproduction

make test-all

Final lines:

• main: ================= 377 failed, 50 passed in 5177.09s (1:26:17) ==================
• feature: ============= 374 failed, 47 passed, 1 error in 4801.89s (1:20:01) =============

Suggested fix directions

• Confirm the GPU architecture (nvidia-smi) and rebuild vllm-flash-attn with a TORCH_CUDA_ARCH_LIST that covers it, or
• Force a non-FA3 backend in the vLLM test harness via VLLM_ATTENTION_BACKEND=FLASHINFER (or TRITON_ATTN / FLEX_ATTENTION) when the FA3 Hopper kernel isn't usable, or
• Have the vLLM worker fail fast with a clear error message when the auto-selected backend's kernel image is incompatible, instead of crashing mid-forward and cascading into hundreds of BrokenPipeErrors.

Once the environment is fixed, re-run on both branches to verify there is no real behavioural regression — the current data cannot establish that.

Environment

• Branches observed: main and feature/reduce-control-token-kv-overhead
• Date observed: 2026-05-18
• Platform: Darwin 24.6.0 (test execution host runs Linux containers — see /workspace/.deps/... paths in the traceback)
• GPU / CUDA / torch versions: to be filled in by the reporter (nvidia-smi, python -c "import torch; print(torch.__version__, torch.cuda.get_device_name(0), torch.cuda.get_device_capability(0))")

Overcoming this issue paths

We should try those 4 paths to check if this solves the problem and we are able to run all the tests, possible only run it from the main branch

• Update the torch version to match vllm 0.19.1:

dependencies = [
    "torch>=2.10.0",

• Install a compatible cuda version:
  pip install nvidia-cuda-runtime-cu12
• Set a compatible kernel / versioning with vllm configurations via flags
• Run the test suite on a compatible kernel

[antonpibm]

This issue is local to the environment where the tests are running as it has an old CUDA version:

Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131

This causes a mismatch between this and the torch version installed by vllm 0.19.1 (2.10.0)
This is not a bug in Granite Switch, let's explore the fix paths presented in the issue description