[BUG] RuntimeError: [_efficient_attention_forward] Unsupported mask type on ROCM, for now

[ZhenyaPav]

OS

Linux

GPU Library

AMD ROCm

Python version

3.11

Describe the bug

Text generation fails on AMD GPU (7900 XT)

Reproduction steps

• Install tabbyAPI as described in the wiki (tested with python 3.11 and 3.13) on Arch Linux with ROCm 6.2.4 and a gfx1100 GPU.
• Start the server
• Make a text generation request

Expected behavior

Text is generated

Logs

2025-14-02 21:20:59.411 ERROR: Traceback (most recent call last):
2025-14-02 21:20:59.411 ERROR: File "/home/zhenyapav/Projects/tabbyAPI/endpoints/OAI/utils/completion.py", line 219, in
stream_generate_completion
2025-14-02 21:20:59.411 ERROR: raise generation
2025-14-02 21:20:59.411 ERROR: File "/home/zhenyapav/Projects/tabbyAPI/endpoints/OAI/utils/completion.py", line 100, in
_stream_collector
2025-14-02 21:20:59.411 ERROR: async for generation in new_generation:
2025-14-02 21:20:59.411 ERROR: File "/home/zhenyapav/Projects/tabbyAPI/backends/exllamav2/model.py", line 1523, in
generate_gen
2025-14-02 21:20:59.411 ERROR: raise ex
2025-14-02 21:20:59.411 ERROR: File "/home/zhenyapav/Projects/tabbyAPI/backends/exllamav2/model.py", line 1431, in
generate_gen
2025-14-02 21:20:59.411 ERROR: async for result in job:
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/generator/dynamic_async.py", line 97, in
aiter
2025-14-02 21:20:59.411 ERROR: raise result
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/generator/dynamic_async.py", line 28, in
_run_iteration
2025-14-02 21:20:59.411 ERROR: results = self.generator.iterate()
2025-14-02 21:20:59.411 ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-14-02 21:20:59.411 ERROR: return func(*args, **kwargs)
2025-14-02 21:20:59.411 ERROR: ^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/generator/dynamic.py", line 980, in iterate
2025-14-02 21:20:59.411 ERROR: job.prefill(results)
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/generator/dynamic.py", line 2434, in prefill
2025-14-02 21:20:59.411 ERROR: self.generator.model.forward_chunk(
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-14-02 21:20:59.411 ERROR: return func(*args, **kwargs)
2025-14-02 21:20:59.411 ERROR: ^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/model.py", line 1012, in forward_chunk
2025-14-02 21:20:59.411 ERROR: x = module.forward(x, cache = cache, attn_params = attn_params, past_len = past_len,
loras = loras, **kwargs)
2025-14-02 21:20:59.411 ERROR:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/attn.py", line 1165, in forward
2025-14-02 21:20:59.411 ERROR: attn_output = attn_func(batch_size, q_len, q_states, k_states, v_states, attn_params,
cfg)
2025-14-02 21:20:59.411 ERROR:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/exllamav2/attn.py", line 891, in _attn_torch
2025-14-02 21:20:59.411 ERROR: attn_output = F.scaled_dot_product_attention(
2025-14-02 21:20:59.411 ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/torch/nn/attention/bias.py", line 290, in
torch_function
2025-14-02 21:20:59.411 ERROR: return cls._dispatch(*args, **kwargs)
2025-14-02 21:20:59.411 ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File
"/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/torch/nn/attention/bias.py", line 248, in _dispatch
2025-14-02 21:20:59.411 ERROR: return torch.ops.aten._efficient_attention_forward(
2025-14-02 21:20:59.411 ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: File "/home/zhenyapav/Projects/tabbyAPI/venv/lib/python3.11/site-packages/torch/_ops.py",
line 1123, in call
2025-14-02 21:20:59.411 ERROR: return self._op(*args, **(kwargs or {}))
2025-14-02 21:20:59.411 ERROR: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-14-02 21:20:59.411 ERROR: RuntimeError: [_efficient_attention_forward] Unsupported mask type on ROCM, for now
2025-14-02 21:20:59.418 ERROR: Sent to request: Completion c7e197066aa946c6a360e456ab562f15 aborted. Please check the server
console.

Additional context

The same version of exllamav2 worked on text-generation-webui (though not without issues)

Acknowledgements

• I have looked for similar issues before submitting this one.
• I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[ZhenyaPav]

This is likely due to can_use_efficient_attention() incorrectly returning true in torch/nn/attention/bias.py. Replacing can_use_efficient_attention with False, to use the fallback logic, fixes the issue.