Jetson AGX Thor (sm_110), CUDA 13.2: 32B NVFP4 serves, first throughput datapoints, and a Tekken tokenizer blocker

[filipsajdak]

Hi — first, thanks for the unusually honest capability labelling in this project. BACKEND-CUDA-SM110 being scoped to exactly "portable bf16 Llama-1B greedy" is what made it obvious where a useful contribution was, so this report tries to fill that gap rather than restate it.

I have a Jetson AGX Thor and ran a window on it. Summary up front:

#	Result
1	Qwen3-32B-NVFP4A16 loads and serves on sm_110 via the portable dequant-GEMM
2	First serving-throughput datapoints on sm_110 — and they are flat with concurrency
3	CUDA 13.2 builds and passes the CUDA gates on sm_110 (you verified with 13.0)
4	Blocker: Mistral-Nemo (Tekken) is rejected at the tokenizer
5	The default async runner does not fault at op level here (unlike sm_87)

Everything below is from one session; I have kept the caveats attached to each claim rather than in a footnote.

Environment

• NVIDIA Jetson AGX Thor, aarch64, nvidia-smi compute_cap = 11.0, 122 GB unified memory
• Built from 0e3bf3c
• Built and run inside nvcr.io/nvidia/vllm:26.04-py3 — the Jetson host carries only the driver (no /usr/local/cuda*, no nvcc), so a native host build was not possible without installing a toolkit. The container supplies nvcc cuda_13.2.r13.2, cmake 3.31.6, ninja 1.13, g++ 13.3.0.
• Container run standalone: --runtime=nvidia --network=host, no MPS pipe mount, no --ipc=host.

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release \
  -DVLLM_CPP_CUDA=ON -DVLLM_CPP_TRITON=OFF -DVLLM_CPP_CUDA_ARCHITECTURES=110
cmake --build build -j 8

1124/1124 objects, 0 errors, 0 warnings. Codegen verified as
--generate-code=arch=compute_110,code=[compute_110,sm_110] and strings on
cuda_backend.cu.o contains sm_110.

As expected, every accelerated path resolved empty for [110]: cutlass-nvfp4,
cutlass-nvfp4-sm100, cutlass-fp8, scaledmm-c3x-sm90, scaledmm-c3x-sm100,
marlin-nvfp4, fa2.

1. Qwen3-32B-NVFP4A16 serves on sm_110

RedHatAI/Qwen3-32B-NVFP4A16 (Qwen3ForCausalLM, 64 layers, compressed-tensors /
nvfp4-pack-quantized) loads and produces coherent output:

--model <local> --port 30803 --gpu-memory-utilization 0.35 \
--max-model-len 4096 --max-num-seqs 16      # VT_ASYNC_RUNNER=0

Two things this establishes on sm_110, if I am reading BACKEND-CUDA-SM110,
docs/BUILD.md and docs/BENCHMARKS.md correctly — please correct me if not:

• NVFP4 computes on sm_110 through the portable kMatmulNvfp4 dequant-GEMM. I had
  initially assumed NVFP4 was unavailable here because the CUTLASS/Marlin/fp4-MMA fast
  paths are all EMPTY for 110; that turned out to be wrong, and the README's
  Qwen3-32B-NVFP4A16 line is what pointed the right way.
• A >1B model runs on sm_110 — 32B rather than the 1B the matrix row is scoped to.

2. First throughput datapoints on sm_110 — and they do not batch

64 max tokens, temperature=0, N concurrent chat-completions:

concurrency	wall	tokens	throughput
1	45.9 s	64	1.39 tok/s
2	88.3 s	128	1.45 tok/s
4	142.3 s	256	1.80 tok/s

The shape is the interesting part: c=4 buys only 1.3x over c=1, where your published
GB10 curve scales 86 -> 292 tok/s (~3.4x) over the same range. On this arch the portable
path appears bound by the unaccelerated GEMM rather than by parallelism — i.e. the fast
paths look like they buy batching, not just peak rate. That may be the more useful half
of the observation.

Caveats, and they are not small:

• Crude harness (concurrent curl), one run per point, no warm-up excluded, wall time
  includes HTTP and tokenization.
• Qwen3 emits thinking tokens, so this is total generated tokens, not useful-answer tokens.
• Other GPU tenants were live on the box (two unrelated vLLM containers under MPS,
  ~19 GB). This was not an isolated device, so treat these as a lower bound.
• 0.35 memory fence, max-model-len 4096, max-num-seqs 16 — untuned.
• Not comparable to the GB10 numbers: different arch, different quant path, different model.

Happy to re-run properly isolated, with a warm-up and repeats, if that would be more useful
than an indicative first datapoint.

3. CUDA 13.2

test_cuda_backend and test_cuda_ops both pass on sm_110 built with CUDA 13.2,
under both VT_ASYNC_RUNNER=0 and the default. Since the SM110 runtime proof was on
13.0.48, this is at least weak evidence the sm_110 support is toolchain-robust across the
13.0/13.2 boundary. I saw #82 touching the same axis, which is why I mention it.

I did not get the llama_paged_engine parity gate run — the checkpoint download was
still in flight when my maintenance window closed. That is the arm that would actually test
strict token-exactness under 13.2, and I would like to finish it.

4. Blocker: Mistral-Nemo is rejected at the tokenizer

mistralai/Mistral-Nemo-Instruct-2407 (BF16, MistralForCausalLM, 40 layers, GQA 32/8,
head_dim 128, untied lm_head, sliding_window: null) fails immediately after the model
loads:

server: loading model from <path> (config .../config.json, tokenizer .../tokenizer.json)
server: fatal: tokenizer: unrecognized pre-tokenizer split regex:
[^\r\n\p{L}\p{N}]?[\p{Lu}\p{Lt}\p{Lm}\p{Lo}\p{M}]*[\p{Ll}\p{Lm}\p{Lo}\p{M}]+|...

This is the Tekken pre-tokenizer regex. It looks architecture-independent — the shape class
matches your own Mistral parity gate, so this is a tokenizer-support gap rather than an
sm_110 issue. I can open it separately if you would prefer it not be bundled with the Thor
report.

5. Async runner

VT_ASYNC_RUNNER default-ON does not fault at op level on sm_110 — test_cuda_backend
and test_cuda_ops pass identically with and without it. Given docs/STATUS.md scopes the
sm_87 verification to VT_ASYNC_RUNNER=0 because of an illegal memory access, this at least
suggests that fault does not reproduce on Thor at this level. I would not call the async
runner verified on sm_110 from this — the paged-engine forward is the test that matters and
I have not run it yet.

Two small documentation notes

• docs/STATUS.md:1482 still describes sm_110 as build-only, which contradicts the
  BACKEND-CUDA-SM110 row and docs/BUILD.md:80. Happy to PR the one-line fix.
• For anyone else building in a CUDA container: CMakeCache.txt reports
  CMAKE_CUDA_ARCHITECTURES=75 even on a correct -DVLLM_CPP_CUDA_ARCHITECTURES=110 build,
  because CMakeLists.txt:363 uses a non-cache set() that shadows the cache entry. Reading
  the cache to confirm the target arch is misleading; the build.ninja gencode line is
  accurate. Not a bug, but it cost me a while.

Offer

I have ongoing access to this board. If useful I can: finish the llama_paged_engine parity
run under 13.2, redo the throughput sweep isolated and with repeats, test the decode CUDA
graph path, or try other checkpoints. Tell me what would be most valuable and I will run it.

[filipsajdak]

Follow-up: the llama_paged_engine parity gate — anchor drift on sm_110 at HEAD

I got the parity arm run after all. Reporting it as a question rather than a conclusion,
because I cannot isolate the variable.

Result: test_llama_paged_engine fails on sm_110, 14/15 assertions passing:

vllm.cpp: Asynchronous scheduling is disabled (max_concurrent_batches=1)
tests/parity/test_llama_paged_engine.cpp:205: FATAL ERROR: REQUIRE( first_div < 0 ) is NOT correct!
  values: REQUIRE( 12 <  0 )
  logged: 1 anchor drift prompt[0] tok=12 engine=3224 committed anchor=4219
          — re-run qwen3-neartie-gap.py to refresh the gap golden

So prompt[0] tracks the committed GB10 anchor for 11 tokens, then emits 3224 where the
anchor has 4219. VT_ASYNC_RUNNER=0, greedy.

Checkpoint drift is ruled out. My snapshot is unsloth/Llama-3.2-1B @
9535bd9b1d1dea6acafbdc4813b728796aeb28da, which is current main and per the HF API has
not changed since 2025-06-02 — comfortably before the goldens were captured. Same repo,
same bytes.

What I cannot separate, and why I am not calling this a CUDA 13.2 finding:

1. Toolchain — CUDA 13.2 here vs 13.0.48 on your sm_110 run (also g++ 13.3.0 in
   the NGC container vs the 13.4 mentioned elsewhere in your notes).
2. Engine drift — I built 0e3bf3c (HEAD at the time), whereas CLAIM-CUDA-SM110-RUNTIME
   was captured at 0f07fe34. That is a lot of engine commits, including the TP work.
3. Possibly just a stale golden at HEAD — your own error string suggests refreshing it is
   a normal operation.

The question that would settle it in one step: does test_llama_paged_engine currently
pass on GB10 at HEAD? If it also drifts there, this is engine-vs-golden and nothing to do
with Thor or 13.2. If it passes on GB10 at HEAD, then the divergence is specific to
sm_110-at-HEAD and worth bisecting — and I am happy to do the bisection on this board, since
it is the arch in question.

If it would help, I can also re-run at 0f07fe34 to hold the engine constant and isolate the
toolchain variable. Say which is more useful and I will run it.

(Correction to my report above, for accuracy: I originally listed the async runner as
"passes at op level". That still holds — test_cuda_backend / test_cuda_ops pass under both
runners — but this paged-engine run was sync-only, so the async runner remains untested in a
full forward pass on sm_110. I did not want that gap to be papered over by the op-level pass.)