Qwen3.5/3.8 text-only checkpoints cannot load: Qwen3_5MoeForCausalLM is unregistered and the loader hardcodes the VL model.language_model. prefix

[localai-bot]

Qwen3.8 (Qwen/Qwen3.8-2.4T-A95B, 2.4T total / A95B active) is not a new architecture. Its config declares Qwen3_5MoeForCausalLM / model_type: qwen3_5_moe_text — the same GDN-hybrid MoE backbone we already run token-exact (315/315 on Qwen3.6-35B-A3B). Every structural knob matches: head_dim 256, linear_key/value_head_dim 128, linear_num_key_heads 16, full_attention_interval 4, attn_output_gate true, partial_rotary_factor 0.25, rope_theta 1e7, mtp_num_hidden_layers 1, and the same tokenizer (vocab_size 248320). The deltas are pure scale (hidden 2048->8192, layers 40->92, experts 256->512, top-k 8->10, moe/shared intermediate 512->2048, linear V-heads 32->128), all config-driven.

Two things block loading it:

1. The architecture string is unregistered. ModelRegistry::Resolve is an exact string match with no aliasing (src/vllm/model_executor/models/model_registry.cpp:217-231); we register only Qwen3_5MoeForConditionalGeneration (qwen3_5_moe.cpp:215) and Qwen3_5ForConditionalGeneration (qwen3_5_dense.cpp:243). A text-only config falls through to RaiseForUnsupported.

2. The loader hardcodes the VL weight prefix. qwen3_5_weights.cpp:560,632,633,659 (and qwen3_5_dense_weights.cpp, 3 sites) look up model.language_model.*. Verified against the real checkpoints:

	Qwen3.6-35B-A3B	Qwen3.8-2.4T-A95B
embed	model.language_model.embed_tokens.weight	model.embed_tokens.weight
layer	model.language_model.layers.0.linear_attn.*	model.layers.0.linear_attn.*
experts	...mlp.experts.gate_up_proj (3D stacked)	...mlp.experts.gate_up_proj (3D stacked)
head	lm_head.weight	lm_head.weight

The tensor names are identical modulo the prefix — same 3D-stacked experts, same shared_expert_gate, same top-level lm_head. So the existing loader is correct once the prefix is tolerant.

Upstream solves exactly this with a mapper on the text-only base class:

# vllm/model_executor/models/qwen3_5.py:296-300 @ origin/main
# Some community text-only checkpoints keep the extraneous
# `model.language_model.` prefix inherited from the VL training stack.
hf_to_vllm_mapper = WeightsMapper(orig_to_new_prefix={"model.language_model.": "model."})

Qwen3_5MoeForCausalLM is a thin text-only class over the same backbone (qwen3_5.py:443-449), and Qwen3_5ForCausalLM is pass over the shared base (:439).

Ahead-of-pin, and stated as such. Our parity pin is 555967922, whose registry has only the ForConditionalGeneration entries. The text-only arms were added upstream by PR #50210 / ad5d29db7 ("[Model] Support Qwen3.5 text-only dense and MoE models"), which is post-pin. This is therefore a deliberate forward port, not a mirror of the pin.

We cannot gate it on our hardware, and that is not hidden. 2.4T bf16 is ~4.8 TB; the only other released variant is Qwen/Qwen3.8-2.4T-A95B-FP8 at ~2.4 TB. GB10 has 128 GB unified, and no smaller Qwen3.8 sibling exists. So there is no token-exact oracle run for the 2.4T checkpoint. What IS gateable: config resolution, architecture dispatch, weight-name mapping, and byte-identical inertness of the existing 27B/35B/Coder gates. The run gate stays explicitly OWED and the row must not claim DONE on the 3.8 checkpoint.

[localai-bot]

Correction to this issue's premise, found during review repair and verified directly.

I claimed above that "the tensor names are identical modulo the prefix ... so the existing loader is correct once the prefix is tolerant." The first half is true; the second half is wrong for the bf16 checkpoint.

The routed-expert tensors are published as 3D-STACKED (model.layers.N.mlp.experts.gate_up_proj / .down_proj) by both Qwen/Qwen3.8-2.4T-A95B and Qwen/Qwen3.6-35B-A3B. But LoadQwen3_5Moe reads PER-EXPERT names — qwen3_5_weights.cpp:526 builds mlp.experts.<e>.{gate,up,down}_proj. There is no stacked reader in that file; repo-wide the only stacked expert readers are gemma4_weights.cpp:183-184, qwen3_5_mtp.cpp:116-117 (the MTP head) and laguna_weights.cpp.

Why this was invisible: our gated 35B row does not load the bf16 checkpoint. It loads nvidia/Qwen3.6-35B-A3B-NVFP4 (include/vllm/model_executor/models/qwen3_5_weights.h:6, tests/parity/test_qwen36_paged_engine.cpp:116), whose quantized experts are per-expert. So the MoE loader has only ever been exercised against the per-expert spelling, and reading the 3.8 index alone could not reveal the mismatch.

Consequence. Registering the architecture and resolving the backbone namespace — what this row implements — does NOT make the published bf16 Qwen/Qwen3.8-2.4T-A95B loadable end to end. Its routed experts are in a layout LoadQwen3_5Moe cannot read, and it would fail at expert loading with a tensor-not-found naming the per-expert spelling.

What the row legitimately delivers is unchanged: the two architecture strings dispatch, the backbone namespace resolves for both spellings, a mixed index is refused, and the gated rows stay byte-identical. The spec was already scoped to claim exactly that and nothing about generated tokens, so no landed claim is false — but the stacked-expert arm is owed work that was not previously identified, and it is a prerequisite for the bf16 checkpoint on any hardware that could hold it.

Recording it here rather than widening the row: a stacked-expert reader for LoadQwen3_5Moe needs its own RED-first test and an inertness proof for the NVFP4 path, which is a separate change.

[localai-bot]

Second correction, and it goes deeper than the first. Found by independent review of 0097a014, verified against the published index.

My earlier comment said the blocker was the 3D-stacked expert layout. That is real but it is not the first thing that fails, and the underlying problem is broader.

LoadQwen3_5Moe is not a loader that happens to read per-expert names — it is a loader for a quantized checkpoint. LoadMoeExpertsInto (qwen3_5_weights.cpp:519-530) routes every expert through LoadNvfp4Raw (:433-462), which hard-requires .weight = U8 plus .weight_scale = F8_E4M3 plus .weight_scale_2.

The published Qwen/Qwen3.8-2.4T-A95B index (1609 tensors) contains 0 tensors matching weight_scale or input_scale, and lm_head is lm_head.weight alone. So the load dies at w.lm_head_fp4 = LoadNvfp4Raw(get, "lm_head") (:679) with expected U8 for lm_head.weight — before the experts are ever reached — and would die again at the FP8 attention, the routed experts, and the shared expert.

The same is true of Qwen/Qwen3.6-35B-A3B: also stacked, also zero scale tensors. Our gated 35B row reads the requantized nvidia/Qwen3.6-35B-A3B-NVFP4 variant. The MoE loader has never read a published Qwen bf16 repo.

Two further consequences worth stating plainly:

The debt was recorded inverted. Several surfaces say "MTP, quantized and GGUF arms for 3.8 are NOT implemented and are recorded as owed". The quantized arm is the only one implemented; the missing arm is bf16/stacked, and it was named nowhere.

The stated closing condition was false. Records said the run gate "closes only when a text-only Qwen3_5[Moe]ForCausalLM checkpoint that fits GB10 appears". A bf16 checkpoint that fit would still refuse at load.

One asymmetry that no record distinguished: LoadQwen3_5Dense does route BF16-vs-NVFP4 by tensor presence, so the dense text-only arm may genuinely load a flat bf16 checkpoint. It is specifically the MoE arm that cannot.

Responsibility for this is mine — the spec's Design table was my analysis, and its own left-hand column (35B publishing stacked, unquantized tensors) refuted its conclusion. The review caught that the claim had also propagated into product source (qwen3_5_weights.h:499-500), docs/STATUS.md, model-matrix.md and three commit messages.

Being repaired now: strike the false claims from every surface, record the bf16/stacked MoE arm as a named owed gap, make the refusal name the missing piece instead of surfacing a bare tensor-not-found, and correct the closing condition. No new loader capability is being added under this row — a stacked/bf16 expert reader needs its own spec, RED-first test, and an inertness proof for the NVFP4 path.