Vulkan on unified memory holds TWO copies of the weights: 27B peaks at 100.8 GiB RSS and OOM-reboots a Spark

[localai-bot]

On a unified-memory device (DGX Spark / GB10), loading a model on the Vulkan
backend holds two full copies of the weights in system RAM. For Qwen3.6-27B
bf16 that is an extra 50 GiB, and it is enough to take the whole machine down:
the box hard-reboots with

NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY]
(0x00000051) returned from _memdescAllocInternal(pMemDesc) @ mem_desc.c:1359

This happened twice in one session on an otherwise idle machine.

Measured

Qwen3.6-27B bf16 (50.89 GiB on disk), GB10, one binary, both arms:

arm	Vulkan live	buffers	VmRSS	MemAvailable floor	outcome
double copy	50.755 GiB	863 of 894	100.759 GiB	13.85 GiB	killed by a watchdog, still allocating
single copy	50.756 GiB	894	53.413 GiB	47.33 GiB	completed

The gap is a flat 50.003 GiB across the last four high-water samples — one
whole extra copy of the model. Qwen3-4B shows the same shape: VmHWM 16.392 →
9.607 GiB.

The unfixed 27B arm never completed, so 100.759 GiB is a lower bound.

Cause

ResidentWeight uploads each weight into a Vulkan buffer and then also keeps
the host-side OwnedTensor.bytes. On a discrete GPU those are different
memories and holding both is merely wasteful of host RAM. On GB10 the Vulkan
allocation is HOST_VISIBLE | HOST_COHERENT | DEVICE_LOCAL on a single unified
heap and is persistently mapped, so both copies are the same RAM and the
cost is doubled.

src/vllm/platforms/vulkan.cpp had reasoned that "there is exactly one copy of
the bytes"; include/vllm/model_executor/models/dense_attn_block.h:190 made a
second one.

Not the cause (checked)

• Windowed page release is not at fault and does fire. It is default ON and
  the Qwen3.5/3.6 loader calls it; during load the RSS holds one copy, not the
  mmap as well.
• No allocator waste. requested == committed exactly, every run: 50.756
  GiB over 894 buffers. No rounding excess, nothing to win from suballocation,
  and maxMemoryAllocationCount is nowhere near 894.
• No transient held too long. A staging or dequant scratch would appear as a
  bump; the excess is a constant equal to the model size.
• Page cache is not the trigger. It tracks copied bytes 1:1 and is
  reclaimable. Note MADV_DONTNEED on a private file mapping does not evict
  page cache — that needs POSIX_FADV_DONTNEED.
• The 89.72 GiB Vulkan heap was never the binding constraint. The machine
  was.

Fix

Re-point bytes at the device allocation through the existing OwnedBytes
borrow, keyed alive by the device allocation's control block — an adoption
rather than a release, so every .bytes reader sees the same bytes and it needs
no "device path committed" proof. Gated on a new
Backend::DeviceMemoryIsHostAddressable(), default false, deliberately narrower
than UnifiedMemory() because CUDA on GB10 is unified yet cudaMalloc is not
host-dereferenceable. Discrete and non-Vulkan paths are byte-identical.

Related

Adjacent but not this: peak is now the load phase, where the host build
reaches ~51 GiB before the first upload. Copying from the mmap straight into the
device buffer would cut that too, and is left open.

Also see #83 (memory budgeting), which is about auto-sizing to the workload
rather than this 2x waste.