[Bug]: rocm/vllm:rocm7.0.0_vllm_0.11.1_20251103 has an error with flash attention

[capteen-hook]

Your current environment

rocm/vllm:rocm7.0.0_vllm_0.11.1_20251103

(Note, this does work with rocm/vllm:rocm7.0.0_vllm_0.10.2_20251006)

Docker:

ARG BASE_IMAGE=python:3.12-slim # passed in rocm/latest
FROM ${BASE_IMAGE}

WORKDIR /app

# Install Rust, required build tools, pkg-config, and OpenSSL development libraries
RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    gcc \
    build-essential \
    pkg-config \
    libssl-dev \
    poppler-utils \
    && curl https://sh.rustup.rs -sSf | sh -s -- -y \
    && apt-get clean \
    && rm -rf /var/lib/apt/lists/*
    
ENV PATH="/root/.cargo/bin:${PATH}"

COPY ./requirements.txt /app/flask_server/requirements.txt

RUN pip install --upgrade pip
RUN pip install --no-cache-dir -r /app/flask_server/requirements.txt

COPY . /app/flask_server

ENV PORT=8000
EXPOSE $PORT

HEALTHCHECK --interval=300s --timeout=5s --start-period=10s --retries=3 \
  CMD curl --fail http://localhost:$PORT/$BASE_URL || exit 1

CMD ["python", "-m", "flask_server"]

environment: (some of these are likely unhelpful :/)

      QWEN_CHECKPOINT: ${QWEN_CHECKPOINT:-Qwen/Qwen2.5-VL-3B-Instruct}
      MAX_MODEL_LEN: ${MAX_MODEL_LEN:-45000}
      MAX_NUM_SEQS: ${MAX_NUM_SEQS:-1}
      GPU_MEMORY_UTILIZATION: ${GPU_MEMORY_UTILIZATION:-0.9}
      MAX_TOKENS: ${MAX_TOKENS:-8192}
      TEMPERATURE: ${TEMPERATURE:-0.0}

      HIP_VISIBLE_DEVICES: ${HIP_VISIBLE_DEVICES:-0,1}
      TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL: ${TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL:-1}
      VLLM_USE_TRITON_FLASH_ATTN: ${VLLM_USE_TRITON_FLASH_ATTN:-0}
      MIOPEN_DEBUG_CONV_DIRECT: ${MIOPEN_DEBUG_CONV_DIRECT:-0}
      MIOOPEN_DEBUG_GCN_ASM_KERNELS: ${MIOOPEN_DEBUG_GCN_ASM_KERNELS:-0}
      VLLM_CUGAGRAPH_MODE: ${VLLM_CUGAGRAPH_MODE:-0}
      USE_CHAT_API: ${USE_CHAT_API:-1}

Host rocm:

rocminfo
ROCk module version 6.14.14 is loaded
=====================
HSA System Attributes
=====================
Runtime Version:         1.18
Runtime Ext Version:     1.11
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE
System Endianness:       LITTLE
Mwaitx:                  DISABLED
XNACK enabled:           NO
DMAbuf Support:          YES
VMM Support:             YES

==========
HSA Agents
==========
*******
Agent 1
*******
  Name:                    AMD Ryzen Threadripper PRO 9965WX 24-Cores
  Uuid:                    CPU-XX
  Marketing Name:          AMD Ryzen Threadripper PRO 9965WX 24-Cores
  Vendor Name:             CPU
  Feature:                 None specified
  Profile:                 FULL_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        0(0x0)
  Queue Min Size:          0(0x0)
  Queue Max Size:          0(0x0)
  Queue Type:              MULTI
  Node:                    0
  Device Type:             CPU
  Cache Info:
    L1:                      49152(0xc000) KB
  Chip ID:                 0(0x0)
  ASIC Revision:           0(0x0)
  Cacheline Size:          64(0x40)
  Max Clock Freq. (MHz):   5489
  BDFID:                   0
  Internal Node ID:        0
  Compute Unit:            48
  SIMDs per CU:            0
  Shader Engines:          0
  Shader Arrs. per Eng.:   0
  WatchPts on Addr. Ranges:1
  Memory Properties:
  Features:                None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: FINE GRAINED
      Size:                    395385652(0x17911b34) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    395385652(0x17911b34) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 3
      Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
      Size:                    395385652(0x17911b34) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
    Pool 4
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    395385652(0x17911b34) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:4KB
      Alloc Alignment:         4KB
      Accessible by all:       TRUE
  ISA Info:
*******
Agent 2
*******
  Name:                    gfx1100
  Uuid:                    GPU-f106daae55da882a
  Marketing Name:          AMD Radeon PRO W7800
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    1
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      6144(0x1800) KB
    L3:                      65536(0x10000) KB
  Chip ID:                 29790(0x745e)
  ASIC Revision:           0(0x0)
  Cacheline Size:          128(0x80)
  Max Clock Freq. (MHz):   1895
  BDFID:                   25344
  Internal Node ID:        1
  Compute Unit:            70
  SIMDs per CU:            2
  Shader Engines:          6
  Shader Arrs. per Eng.:   2
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          32(0x20)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    1024(0x400)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        2147483647(0x7fffffff)
    y                        65535(0xffff)
    z                        65535(0xffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 552
  SDMA engine uCode::      24
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    31440896(0x1dfc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
      Size:                    31440896(0x1dfc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 3
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx1100
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        2147483647(0x7fffffff)
        y                        65535(0xffff)
        z                        65535(0xffff)
      FBarrier Max Size:       32
    ISA 2
      Name:                    amdgcn-amd-amdhsa--gfx11-generic
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        2147483647(0x7fffffff)
        y                        65535(0xffff)
        z                        65535(0xffff)
      FBarrier Max Size:       32
*******
Agent 3
*******
  Name:                    gfx1201
  Uuid:                    GPU-ad0031ab2e3ff4d7
  Marketing Name:          AMD Radeon AI PRO R9700
  Vendor Name:             AMD
  Feature:                 KERNEL_DISPATCH
  Profile:                 BASE_PROFILE
  Float Round Mode:        NEAR
  Max Queue Number:        128(0x80)
  Queue Min Size:          64(0x40)
  Queue Max Size:          131072(0x20000)
  Queue Type:              MULTI
  Node:                    2
  Device Type:             GPU
  Cache Info:
    L1:                      32(0x20) KB
    L2:                      8192(0x2000) KB
    L3:                      65536(0x10000) KB
  Chip ID:                 30033(0x7551)
  ASIC Revision:           1(0x1)
  Cacheline Size:          256(0x100)
  Max Clock Freq. (MHz):   2350
  BDFID:                   8960
  Internal Node ID:        2
  Compute Unit:            64
  SIMDs per CU:            2
  Shader Engines:          4
  Shader Arrs. per Eng.:   2
  WatchPts on Addr. Ranges:4
  Coherent Host Access:    FALSE
  Memory Properties:
  Features:                KERNEL_DISPATCH
  Fast F16 Operation:      TRUE
  Wavefront Size:          32(0x20)
  Workgroup Max Size:      1024(0x400)
  Workgroup Max Size per Dimension:
    x                        1024(0x400)
    y                        1024(0x400)
    z                        1024(0x400)
  Max Waves Per CU:        32(0x20)
  Max Work-item Per CU:    1024(0x400)
  Grid Max Size:           4294967295(0xffffffff)
  Grid Max Size per Dimension:
    x                        2147483647(0x7fffffff)
    y                        65535(0xffff)
    z                        65535(0xffff)
  Max fbarriers/Workgrp:   32
  Packet Processor uCode:: 58
  SDMA engine uCode::      380
  IOMMU Support::          None
  Pool Info:
    Pool 1
      Segment:                 GLOBAL; FLAGS: COARSE GRAINED
      Size:                    33406976(0x1fdc000) KB
      Allocatable:             TRUE
      Alloc Granule:           4KB
      Alloc Recommended Granule:2048KB
      Alloc Alignment:         4KB
      Accessible by all:       FALSE
    Pool 2
      Segment:                 GROUP
      Size:                    64(0x40) KB
      Allocatable:             FALSE
      Alloc Granule:           0KB
      Alloc Recommended Granule:0KB
      Alloc Alignment:         0KB
      Accessible by all:       FALSE
  ISA Info:
    ISA 1
      Name:                    amdgcn-amd-amdhsa--gfx1201
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        2147483647(0x7fffffff)
        y                        65535(0xffff)
        z                        65535(0xffff)
      FBarrier Max Size:       32
    ISA 2
      Name:                    amdgcn-amd-amdhsa--gfx12-generic
      Machine Models:          HSA_MACHINE_MODEL_LARGE
      Profiles:                HSA_PROFILE_BASE
      Default Rounding Mode:   NEAR
      Default Rounding Mode:   NEAR
      Fast f16:                TRUE
      Workgroup Max Size:      1024(0x400)
      Workgroup Max Size per Dimension:
        x                        1024(0x400)
        y                        1024(0x400)
        z                        1024(0x400)
      Grid Max Size:           4294967295(0xffffffff)
      Grid Max Size per Dimension:
        x                        2147483647(0x7fffffff)
        y                        65535(0xffff)
        z                        65535(0xffff)
      FBarrier Max Size:       32
*** Done ***

🐛 Describe the bug

Stack Trace

worker        | Visible devices: 0
worker        | Using tensor parallel size: 1
worker        | [Celery] Initializing model in worker...
worker        | INFO 11-04 15:56:04 [utils.py:243] non-default args: {'trust_remote_code': True, 'max_model_len': 10960, 'gpu_memory_utilization': 0.94, 'max_num_seqs': 1, 'disable_log_stats': True, 'limit_mm_per_prompt': {'image': 1, 'video': 0}, 'model': 'Qwen/Qwen3-VL-8B-Instruct'}
worker        | INFO 11-04 15:56:11 [model.py:658] Resolved architecture: Qwen3VLForConditionalGeneration
worker        | INFO 11-04 15:56:11 [model.py:1745] Using max model len 10960
worker        | The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
worker        | INFO 11-04 15:56:12 [scheduler.py:225] Chunked prefill is enabled with max_num_batched_tokens=8192.
worker        | WARNING 11-04 15:56:12 [vllm.py:498] No piecewise cudagraph for executing cascade attention. Will fall back to eager execution if a batch runs into cascade attentions
worker        | WARNING 11-04 15:56:13 [__init__.py:1993] We must use the `spawn` multiprocessing start method. Overriding VLLM_WORKER_MULTIPROC_METHOD to 'spawn'. See https://docs.vllm.ai/en/latest/usage/troubleshooting.html#python-multiprocessing for more information. Reasons: CUDA is initialized
worker        | INFO 11-04 15:56:14 [__init__.py:225] Automatically detected platform rocm.
worker        | /usr/local/lib/python3.12/dist-packages/transformers/utils/hub.py:110: FutureWarning: Using `TRANSFORMERS_CACHE` is deprecated and will be removed in v5 of Transformers. Use `HF_HOME` instead.
worker        |   warnings.warn(
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:56:15 [core.py:730] Waiting for init message from front-end.
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:56:15 [core.py:97] Initializing a V1 LLM engine (v0.11.1rc2.dev141+g38f225c2a) with config: model='Qwen/Qwen3-VL-8B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen3-VL-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=10960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-VL-8B-Instruct, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': None, 'mode': 3, 'debug_dump_path': None, 'cache_dir': '', 'backend': 'inductor', 'custom_ops': ['+rms_norm', '+silu_and_mul', '+quant_fp8', 'none', '+rms_norm'], 'splitting_ops': [], 'use_inductor': None, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL: 2>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [2, 1], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_capture_size': 2, 'local_cache_dir': None}
worker        | [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
worker        | [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
worker        | [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
worker        | [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
worker        | [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
worker        | [Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:56:18 [parallel_state.py:1325] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:56:21 [gpu_model_runner.py:2843] Starting to load model Qwen/Qwen3-VL-8B-Instruct...
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:56:21 [rocm.py:298] Using Rocm Attention backend on V1 engine.
worker        | (EngineCore_DP0 pid=249) WARNING 11-04 15:56:21 [compilation.py:874] Op 'quant_fp8' not present in model, enabling with '+quant_fp8' has no effect
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:56:22 [weight_utils.py:419] Using model weights format ['*.safetensors']
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:58:35 [weight_utils.py:440] Time spent downloading weights for Qwen/Qwen3-VL-8B-Instruct: 133.725200 seconds
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:01<00:04,  1.43s/it]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:03<00:03,  1.51s/it]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:04<00:01,  1.53s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:05<00:00,  1.29s/it]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:05<00:00,  1.37s/it]
worker        | (EngineCore_DP0 pid=249)
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:58:41 [default_loader.py:314] Loading weights took 5.54 seconds
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:58:42 [gpu_model_runner.py:2904] Model loading took 16.8730 GiB and 139.878708 seconds
worker        | (EngineCore_DP0 pid=249) INFO 11-04 15:58:42 [gpu_model_runner.py:3670] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 image items of the maximum feature size.
worker        | (EngineCore_DP0 pid=249) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/utils.py:117: UserWarning: Failed validator: GCN_ARCH_NAME (Triggered internally at /app/pytorch/aten/src/ATen/hip/tunable/Tunable.cpp:366.)
worker        | (EngineCore_DP0 pid=249)   return torch.nn.functional.linear(x, weight, bias)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793] EngineCore failed to start.
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793] Traceback (most recent call last):
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 784, in run_engine_core
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     engine_core = EngineCoreProc(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 552, in __init__
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     super().__init__(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 113, in __init__
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 224, in _initialize_kv_caches
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     available_gpu_memory = self.model_executor.determine_available_memory()
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 88, in determine_available_memory
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return self.collective_rpc("determine_available_memory")
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 74, in collective_rpc
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return [run_method(self.driver_worker, method, args, kwargs)]
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2089, in run_method
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return func(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return func(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 280, in determine_available_memory
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     self.model_runner.profile_run()
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3686, in profile_run
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     dummy_encoder_outputs = self.model.get_multimodal_embeddings(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 1604, in get_multimodal_embeddings
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     image_embeddings = self._process_image_input(multimodal_input)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 1416, in _process_image_input
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     image_embeds = self.visual(pixel_values, grid_thw=grid_thw_list)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return self._call_impl(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return forward_call(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 547, in forward
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     hidden_states = blk(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                     ^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return self._call_impl(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return forward_call(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 231, in forward
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     x = x + self.attn(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]             ^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return self._call_impl(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return forward_call(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 409, in forward
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     output = self.flash_attn_varlen_func(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/flash_attn/flash_attn_interface.py", line 1443, in flash_attn_varlen_func
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return FlashAttnVarlenFunc.apply(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 581, in apply
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return super().apply(*args, **kwargs)  # type: ignore[misc]
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/flash_attn/flash_attn_interface.py", line 925, in forward
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_varlen_forward(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1254, in __call__
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return self._op(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 342, in backend_impl
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     result = self._backend_fns[device_type](*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/_compile.py", line 53, in inner
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return disable_fn(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1005, in _fn
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return fn(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 375, in wrapped_fn
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     return fn(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]            ^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]   File "/usr/local/lib/python3.12/dist-packages/flash_attn/flash_attn_interface.py", line 165, in _flash_attn_varlen_forward
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]     out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.varlen_fwd(
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793]                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) ERROR 11-04 16:00:28 [core.py:793] RuntimeError: HIP Function Failed (/app/flash-attention/csrc/composable_kernel/include/ck_tile/host/kernel_launch_hip.hpp,65) invalid device function
worker        | (EngineCore_DP0 pid=249) Process EngineCore_DP0:
worker        | (EngineCore_DP0 pid=249) Traceback (most recent call last):
worker        | (EngineCore_DP0 pid=249)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
worker        | (EngineCore_DP0 pid=249)     self.run()
worker        | (EngineCore_DP0 pid=249)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
worker        | (EngineCore_DP0 pid=249)     self._target(*self._args, **self._kwargs)
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 797, in run_engine_core
worker        | (EngineCore_DP0 pid=249)     raise e
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 784, in run_engine_core
worker        | (EngineCore_DP0 pid=249)     engine_core = EngineCoreProc(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 552, in __init__
worker        | (EngineCore_DP0 pid=249)     super().__init__(
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 113, in __init__
worker        | (EngineCore_DP0 pid=249)     num_gpu_blocks, num_cpu_blocks, kv_cache_config = self._initialize_kv_caches(
worker        | (EngineCore_DP0 pid=249)                                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 224, in _initialize_kv_caches
worker        | (EngineCore_DP0 pid=249)     available_gpu_memory = self.model_executor.determine_available_memory()
worker        | (EngineCore_DP0 pid=249)                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 88, in determine_available_memory
worker        | (EngineCore_DP0 pid=249)     return self.collective_rpc("determine_available_memory")
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 74, in collective_rpc
worker        | (EngineCore_DP0 pid=249)     return [run_method(self.driver_worker, method, args, kwargs)]
worker        | (EngineCore_DP0 pid=249)             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2089, in run_method
worker        | (EngineCore_DP0 pid=249)     return func(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context
worker        | (EngineCore_DP0 pid=249)     return func(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 280, in determine_available_memory
worker        | (EngineCore_DP0 pid=249)     self.model_runner.profile_run()
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 3686, in profile_run
worker        | (EngineCore_DP0 pid=249)     dummy_encoder_outputs = self.model.get_multimodal_embeddings(
worker        | (EngineCore_DP0 pid=249)                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 1604, in get_multimodal_embeddings
worker        | (EngineCore_DP0 pid=249)     image_embeddings = self._process_image_input(multimodal_input)
worker        | (EngineCore_DP0 pid=249)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 1416, in _process_image_input
worker        | (EngineCore_DP0 pid=249)     image_embeds = self.visual(pixel_values, grid_thw=grid_thw_list)
worker        | (EngineCore_DP0 pid=249)                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
worker        | (EngineCore_DP0 pid=249)     return self._call_impl(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
worker        | (EngineCore_DP0 pid=249)     return forward_call(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 547, in forward
worker        | (EngineCore_DP0 pid=249)     hidden_states = blk(
worker        | (EngineCore_DP0 pid=249)                     ^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
worker        | (EngineCore_DP0 pid=249)     return self._call_impl(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
worker        | (EngineCore_DP0 pid=249)     return forward_call(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_vl.py", line 231, in forward
worker        | (EngineCore_DP0 pid=249)     x = x + self.attn(
worker        | (EngineCore_DP0 pid=249)             ^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
worker        | (EngineCore_DP0 pid=249)     return self._call_impl(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl
worker        | (EngineCore_DP0 pid=249)     return forward_call(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 409, in forward
worker        | (EngineCore_DP0 pid=249)     output = self.flash_attn_varlen_func(
worker        | (EngineCore_DP0 pid=249)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/flash_attn/flash_attn_interface.py", line 1443, in flash_attn_varlen_func
worker        | (EngineCore_DP0 pid=249)     return FlashAttnVarlenFunc.apply(
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/autograd/function.py", line 581, in apply
worker        | (EngineCore_DP0 pid=249)     return super().apply(*args, **kwargs)  # type: ignore[misc]
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/flash_attn/flash_attn_interface.py", line 925, in forward
worker        | (EngineCore_DP0 pid=249)     out_padded, softmax_lse, S_dmask, rng_state = _wrapped_flash_attn_varlen_forward(
worker        | (EngineCore_DP0 pid=249)                                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1254, in __call__
worker        | (EngineCore_DP0 pid=249)     return self._op(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 342, in backend_impl
worker        | (EngineCore_DP0 pid=249)     result = self._backend_fns[device_type](*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/_compile.py", line 53, in inner
worker        | (EngineCore_DP0 pid=249)     return disable_fn(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/_dynamo/eval_frame.py", line 1005, in _fn
worker        | (EngineCore_DP0 pid=249)     return fn(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/torch/_library/custom_ops.py", line 375, in wrapped_fn
worker        | (EngineCore_DP0 pid=249)     return fn(*args, **kwargs)
worker        | (EngineCore_DP0 pid=249)            ^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249)   File "/usr/local/lib/python3.12/dist-packages/flash_attn/flash_attn_interface.py", line 165, in _flash_attn_varlen_forward
worker        | (EngineCore_DP0 pid=249)     out, softmax_lse, S_dmask, rng_state = flash_attn_gpu.varlen_fwd(
worker        | (EngineCore_DP0 pid=249)                                            ^^^^^^^^^^^^^^^^^^^^^^^^^^
worker        | (EngineCore_DP0 pid=249) RuntimeError: HIP Function Failed (/app/flash-attention/csrc/composable_kernel/include/ck_tile/host/kernel_launch_hip.hpp,65) invalid device function
worker        | [rank0]:[W1104 16:00:28.050688195 ProcessGroupNCCL.cpp:1522] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[github-actions]

CC @hongxiayang @tjtanaa @vllmellm for ROCm-related issue

[capteen-hook]

sorry if this is not the right forum for issues with the image rocm/vllm from AMD

[tjtanaa]

@capteen-hook The CK flash attention implementation is not compatible with Radeon GPUs. Can you reinstall the latest vLLM source code. The code path for Radeon GPU is fixed. It will now go into the TORCH_SDPA code path.

[ianbmacdonald]

It does fire up on gfx1151 using v0.11.1rc6.dev209+g781f5ebf5 ; I did manually tweak the aiter JIT core.py to include gfx1151 (in addition to gfx1100) and used the new aiter commit from rocm/vllm-dev:preview7.1_1117_rc1_20251107 to avoid missing symbols on the aiter JIT which I assumed were fixed by aligning some ck_tile bits between the shipped ROCm 7.1 libraries (composablekernel-dev 1.1.0.70100-20~24.04) and the ck submodule synced in aiter commit 59bd8ff2. Doing this at least seems to allow vLLM to load, which otherwise just hangs up during aiter enumeration when I try to use the similar prebuilt image noted above, at least on gfx1151. I have a gfx1100 that seemed to behave the same way with the prebuilt; have not tried from source yet.

(vllm_11_1) imac@ai2:~/src/vllm_11_1$ vllm serve Qwen/Qwen3-VL-8B-Instruct --chat-template-content-format openai
(APIServer pid=4301) INFO 11-08 17:13:49 [api_server.py:1959] vLLM API server version 0.11.1rc6.dev209+g781f5ebf5
(APIServer pid=4301) INFO 11-08 17:13:49 [utils.py:253] non-default args: {'model_tag': 'Qwen/Qwen3-VL-8B-Instruct', 'chat_template_content_format': 'openai', 'model': 'Qwen/Qwen3-VL-8B-Instruct'}
(APIServer pid=4301) INFO 11-08 17:13:54 [model.py:630] Resolved architecture: Qwen3VLForConditionalGeneration
(APIServer pid=4301) INFO 11-08 17:13:54 [model.py:1728] Using max model len 262144
(APIServer pid=4301) INFO 11-08 17:13:54 [scheduler.py:254] Chunked prefill is enabled with max_num_batched_tokens=8192.
(EngineCore_DP0 pid=4448) INFO 11-08 17:13:58 [core.py:91] Initializing a V1 LLM engine (v0.11.1rc6.dev209+g781f5ebf5) with config: model='Qwen/Qwen3-VL-8B-Instruct', speculative_config=None, tokenizer='Qwen/Qwen3-VL-8B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen/Qwen3-VL-8B-Instruct, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={'level': None, 'mode': 3, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none', '+rms_norm'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': True, 'use_inductor': None, 'compile_sizes': [], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'use_cudagraph': True, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'full_cuda_graph': True, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {}, 'max_cudagraph_capture_size': 512, 'local_cache_dir': None}
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_DP0 pid=4448) INFO 11-08 17:13:59 [parallel_state.py:1325] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:04 [gpu_model_runner.py:2944] Starting to load model Qwen/Qwen3-VL-8B-Instruct...
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:05 [rocm.py:288] Using Triton Attention backend.
Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:02,  1.37it/s]
Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:01<00:01,  1.33it/s]
Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:02<00:00,  1.31it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.55it/s]
Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:02<00:00,  1.46it/s]
(EngineCore_DP0 pid=4448) 
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:08 [default_loader.py:314] Loading weights took 2.79 seconds
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:09 [gpu_model_runner.py:3023] Model loading took 16.8861 GiB and 3.993779 seconds
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:09 [gpu_model_runner.py:3808] Encoder cache will be initialized with a budget of 153600 tokens, and profiled with 1 video items of the maximum feature size.
(EngineCore_DP0 pid=4448) /home/imac/src/vllm_11_1/.venv/lib/python3.12/site-packages/vllm/model_executor/layers/utils.py:149: UserWarning: Failed validator: ROCBLAS_VERSION (Triggered internally at /home/imac/src/vllm_11_1/pytorch/aten/src/ATen/hip/tunable/Tunable.cpp:366.)
(EngineCore_DP0 pid=4448)   return torch.nn.functional.linear(x, weight, bias)
(EngineCore_DP0 pid=4448) WARNING 11-08 17:14:14 [common.py:90] flash_attn is not installed. Falling back to PyTorch implementation for rotary embeddings.
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:54 [backends.py:618] Using cache directory: /mnt/models/vllm-cache/torch_compile_cache/4bd6b98c77/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:54 [backends.py:634] Dynamo bytecode transform time: 3.22 s
(EngineCore_DP0 pid=4448) INFO 11-08 17:14:56 [backends.py:250] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=4448) INFO 11-08 17:15:02 [backends.py:281] Compiling a graph for dynamic shape takes 7.47 s
(EngineCore_DP0 pid=4448) [aiter] import [module_aiter_enum] under /mnt/models/aiter/module_aiter_enum.so
(EngineCore_DP0 pid=4448) [2025-11-08 17:15:02] INFO core.py:353: import [module_aiter_enum] under /mnt/models/aiter/module_aiter_enum.so
(EngineCore_DP0 pid=4448) [aiter] import [module_rmsnorm] under /mnt/models/aiter/module_rmsnorm.so
(EngineCore_DP0 pid=4448) [2025-11-08 17:15:02] INFO core.py:353: import [module_rmsnorm] under /mnt/models/aiter/module_rmsnorm.so
(EngineCore_DP0 pid=4448) INFO 11-08 17:15:03 [monitor.py:34] torch.compile takes 10.69 s in total
(EngineCore_DP0 pid=4448) INFO 11-08 17:15:10 [gpu_worker.py:349] Available KV cache memory: 83.88 GiB
(EngineCore_DP0 pid=4448) INFO 11-08 17:15:10 [kv_cache_utils.py:1229] GPU KV cache size: 610,816 tokens
(EngineCore_DP0 pid=4448) INFO 11-08 17:15:10 [kv_cache_utils.py:1234] Maximum concurrency for 262,144 tokens per request: 2.33x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:15<00:00,  3.27it/s]
Capturing CUDA graphs (decode, FULL): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:18<00:00,  2.77it/s]
(EngineCore_DP0 pid=4448) INFO 11-08 17:15:50 [gpu_model_runner.py:3964] Graph capturing finished in 35 secs, took 0.44 GiB
(EngineCore_DP0 pid=4448) INFO 11-08 17:15:50 [core.py:240] init engine (profile, create kv cache, warmup model) took 101.21 seconds
(APIServer pid=4301) INFO 11-08 17:15:53 [api_server.py:1724] Supported tasks: ['generate']
(APIServer pid=4301) WARNING 11-08 17:15:53 [model.py:1558] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=4301) INFO 11-08 17:15:53 [serving_responses.py:168] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=4301) INFO 11-08 17:15:53 [serving_chat.py:131] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=4301) INFO 11-08 17:15:53 [serving_completion.py:73] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=4301) INFO 11-08 17:15:53 [serving_chat.py:131] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=4301) INFO 11-08 17:15:53 [api_server.py:2034] Starting vLLM API server 0 on http://0.0.0.0:8000
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:38] Available routes are:
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=4301) INFO 11-08 17:15:53 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=4301) INFO:     Started server process [4301]
(APIServer pid=4301) INFO:     Waiting for application startup.
(APIServer pid=4301) INFO:     Application startup complete.

(vllm_11_1) imac@ai2:~/src/vllm_11_1$ cat pyproject.toml 
[project]
name = "vllm-11-1"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "aiter",
    "amdsmi>=7.0.2",
    "hf-transfer>=0.1.9",
    "hf-xet>=1.2.0",
    "huggingface-hub[cli]>=0.36.0",
    "torch",
    "torchvision",
    "triton",
    "uv>=0.9.8",
    "vllm",
]

[tool.uv.sources]
triton = { path = "../triton-3.4.0-cp312-cp312-linux_x86_64.whl" }
torch = { path = "../torch-2.9.0a0+git1c57644-cp312-cp312-linux_x86_64.whl" }
aiter = { path = "../aiter-0.1.6.post2.dev55+g59bd8ff2c.d20251108-cp312-cp312-linux_x86_64.whl" }
torchvision = { path = "../torchvision-0.23.0a0+824e8c8-cp312-cp312-linux_x86_64.whl" }
vllm = { path = "../vllm-0.11.1rc6.dev209+g781f5ebf5.rocm710-cp312-cp312-linux_x86_64.whl" }

[hongxiayang]

@capteen-hook can you try to run with latest vllm code?

[westers]

Root Cause Analysis - gfx1103 Support

I've successfully reproduced and investigated this issue on an AMD Radeon 780M (gfx1103).

Root Cause

The Docker image rocm/vllm:rocm7.0.0_vllm_0.11.1_20251103 was built with:

PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151

gfx1103 is missing from this list!

This means PyTorch/vLLM binaries don't include compiled kernels for gfx1103, causing the HIP error: invalid device function when trying to execute any GPU operation.

Why This Happens

• The error occurs during torch.zeros_like() in CpuGpuBuffer.__init__ (vllm/v1/utils.py:117)
• This is before any attention backend selection
• It's a fundamental PyTorch kernel compatibility issue, not specific to flash attention

Solutions

For Users:

1. Build vLLM from source with your GPU architecture:

   export PYTORCH_ROCM_ARCH="gfx1103"
   pip install -e .

2. Wait for updated Docker image that includes gfx1103 support

For Maintainers:
Add gfx1103 to the PYTORCH_ROCM_ARCH environment variable when building Docker images. This is related to PR #31060 which improves GPU target propagation during builds.

Testing

• ✅ Reproduced error on gfx1103 (AMD Radeon 780M)
• ❌ Setting VLLM_USE_V1=0 doesn't help (V1 is forced)
• ❌ Setting VLLM_ATTENTION_BACKEND=TRITON_ATTN doesn't help (error occurs before backend selection)
• ✅ Confirmed Docker image environment shows missing gfx1103

Hope this helps other Radeon users encountering this issue!