DeepSeek-V4 + MTP: `AttributeError: '_num_tables'` in `DeepseekV4CacheManager.copy_batch_block_offsets` during warmup

[thorjohnsen]

Component: tensorrt_llm/_torch/attention_backend/sparse/deepseek_v4/cache_manager.py
Version: TensorRT-LLM 1.3.0rc21
Verified against: NVIDIA/TensorRT-LLM main — this file does differ from 1.3.0rc21, so it was checked directly; the defect is present on main at cache_manager.py:1157 / :1227 (see below).
Hardware: 4x NVIDIA GB300, driver 580.159.04, CUDA 13.2
Model: DeepSeek-V4-Flash (FP8, model_type: deepseek_v4, num_nextn_predict_layers: 1)
Severity: High — MTP speculative decoding cannot be enabled for DeepSeek-V4; deterministic failure on all ranks during warmup

Summary

Enabling MTP speculative decoding on DeepSeek-V4 crashes during executor
initialization. DeepseekV4CacheManager.copy_batch_block_offsets() reads
self._num_tables, which is only ever assigned inside a different method
(compute_sliding_block_tables()) and is never initialized in __init__.

The MTP path invokes copy_batch_block_offsets() on the draft KV cache
manager without compute_sliding_block_tables() having run on that instance, so
the attribute does not exist.

Reproduction

cat > /tmp/mtp.yml <<'EOF'
kv_cache_config:
  tokens_per_block: 128
  dtype: fp8
  free_gpu_memory_fraction: 0.9
cuda_graph_config:
  enable_padding: true
moe_config:
  backend: TRTLLM
speculative_config:
  decoding_type: MTP
  num_nextn_predict_layers: 1
EOF

trtllm-bench --model $M --model_path $M throughput \
  --tp 4 --ep 4 --dataset /tmp/dsv4_1k1k.txt \
  --max_batch_size 64 --max_num_tokens 8192 \
  --concurrency 32 --num_requests 128 \
  --kv_cache_free_gpu_mem_fraction 0.8 --config /tmp/mtp.yml

Fails on all 4 ranks, deterministically, ~80 s in.

Traceback

[executor][RANK 0] Failed to initialize executor on rank 0:
    'DeepseekV4CacheManager' object has no attribute '_num_tables'

  File ".../_torch/pyexecutor/model_engine.py", line 1082, in warmup
    self._run_attention_warmup(resource_manager, can_run_general_warmup)
  File ".../_torch/pyexecutor/model_engine.py", line 1273, in _run_attention_warmup
    self.forward(batch, ...)
  File ".../_torch/pyexecutor/model_engine.py", line 5481, in forward
    inputs, gather_ids = self._prepare_inputs(...)
  File ".../_torch/pyexecutor/model_engine.py", line 4189, in _prepare_tp_inputs
    attn_metadata.prepare()
  File ".../attention_backend/sparse/deepseek_v4/deepseek_v4.py", line 679, in prepare
    TrtllmAttentionMetadata.prepare(self)
  File ".../attention_backend/trtllm.py", line 604, in prepare
    self.draft_kv_cache_manager.copy_batch_block_offsets(
  File ".../attention_backend/sparse/deepseek_v4/cache_manager.py", line 1314, in copy_batch_block_offsets
    dst_tensor[:, : self._num_tables, 0, :].copy_(
AttributeError: 'DeepseekV4CacheManager' object has no attribute '_num_tables'

Root cause

_num_tables is a lazily-created per-batch attribute with no default. In
DeepseekV4CacheManager:

• assigned only in compute_sliding_block_tables():

  def compute_sliding_block_tables(self, request_ids, num_contexts) -> None:
      copy_idx = self.index_mapper.get_copy_index(request_ids, num_contexts, 1)
      num_tables = copy_idx.size(0)
      self._num_tables = num_tables          # <-- only assignment

• read, unguarded, in copy_batch_block_offsets():

  dst_tensor[:, : self._num_tables, 0, :].copy_(
      self._precomputed_sliding_block_tables[
          :, DeepseekV4AttentionType.SWA.value, : self._num_tables, :], ...)

• never initialized in __init__.

So copy_batch_block_offsets() carries an implicit, undocumented ordering
dependency: compute_sliding_block_tables() must have run on the same instance
first. The non-speculative path apparently satisfies this. The MTP path does
not — trtllm.py calls copy_batch_block_offsets() on
self.draft_kv_cache_manager, a separate instance that never had
compute_sliding_block_tables() invoked.

Line numbers on main

	main	1.3.0rc21
compute_sliding_block_tables def	1149	1241
self._num_tables = num_tables	1157	1249
copy_batch_block_offsets def	1210	1302
unguarded read of self._num_tables	1227	1314

_num_tables occurs exactly 5 times in the file on main (1157, 1227, 1229,
1247, 1248) — none in __init__.

Suggested fix

Any of:

1. Initialize the attribute in DeepseekV4CacheManager.__init__ (e.g.
   self._num_tables = 0) so the state is always well-defined. Note this makes
   the copy a no-op rather than a crash, which may mask the real problem below.
2. Ensure compute_sliding_block_tables() is invoked on the draft KV cache
   manager before copy_batch_block_offsets() in the speculative path. This is
   probably the actual fix — if the draft manager's sliding block tables were
   never computed, the copy has nothing valid to read regardless of whether the
   attribute exists.
3. Assert explicitly, so the ordering contract fails loudly and legibly instead
   of as an AttributeError deep in warmup.

The broader issue is that a per-batch lazy attribute is being read across an
implicit ordering contract with no enforcement.

Bisect

Isolating which config feature triggers this — all runs tp4/ep4, concurrency 32,
128 requests, same model and build:

MTP	attention DP	result
on	on	crash — CUDA_ERROR_ILLEGAL_ADDRESS in CUDAGraph::replay() (filed separately)
off	on	OK — 2382 tok/s output
on	off	crash — this issue
off	off	OK — 3129 tok/s output

MTP is present in both failures and absent from both successes. Turning
attention DP off does not fix MTP; it changes how it fails.

[thorjohnsen]

Companion issue filed: #17025 — same MTP trigger on DeepSeek-V4, but with enable_attention_dp: true it manifests as CUDA_ERROR_ILLEGAL_ADDRESS inside CUDAGraph::replay() rather than this AttributeError. Possibly the same broken draft-KV bookkeeping surfacing as memory corruption once CUDA graphs are in play; worth triaging together.

Bisect summary (tp4/ep4, concurrency 32, 128 requests, identical model + build):

MTP	attention DP	result
on	on	crash — CUDA_ERROR_ILLEGAL_ADDRESS (#17025)
off	on	OK — 2382 tok/s output
on	off	crash — this issue
off	off	OK — 3129 tok/s output

MTP is necessary for the failure; attention DP only determines which failure occurs.