🐛 [Bug] Error with Cask conv when using Hardware Compatible flag

[nMoussa]

Bug Description

When compiling on one GPU Arch, and then running on a different GPU Arch, I'm getting the following error:

ERROR: [Torch-TensorRT] - IExecutionContext::enqueueV3: Error Code 1: Cask (Cask convolution execution In execute at /_src/runtime/gpu/cask/convBaseRunner.cpp:312)

I'm using the dynamo path. The flag hardware_compatible is set to True.
The error happens at runtime, it doesn't raise exception, it gives dummy outputs.

repro_cask_hw_compatible.py

I compiled on A100 and executed on H100/L4 --> got the error.
I compiled on H100 and executed on A100/B200 --> got the error.

I checked that there is no cuda context conflicts.

To Reproduce

Steps to reproduce the behavior:

Attached script: repro_cask_hw_compatible.py. It isolates the crash down to
torchaudio's public WAV2VEC2_BASE bundle's convolutional feature extractor (no
proprietary model needed) — a stack of Conv1d + GroupNorm/LayerNorm + GELU
layers, FP16, use_explicit_typing=True.

1. On an Ampere or Hopper GPU (e.g. A100, H100):

   python repro_cask_hw_compatible.py build --hw-compatible -o engine_hwcompat.pt2
   

2. Copy engine_hwcompat.pt2 to a Blackwell GPU (e.g. RTX 5060), then:

   python repro_cask_hw_compatible.py run -i engine_hwcompat.pt2
   

3. Observe the Cask (Cask convolution execution ...) ERROR printed on the first
   inference call (stderr), with no Python exception raised.

Optional control, to see the contrast described above:

python repro_cask_hw_compatible.py build -o engine_control.pt2   # hardware_compatible=False
python repro_cask_hw_compatible.py run -i engine_control.pt2     # fails at load, not inference

The relevant compile call inside the script:

compiled = torch_tensorrt.dynamo.compile(
    exported,
    inputs=[waveforms],
    use_explicit_typing=True,
    min_block_size=1,
    hardware_compatible=True,
    require_full_compilation=True,
    truncate_double=True,
)

Expected behavior

The engine built with hardware_compatible=True on an Hopper GPU should either:

• run correctly on the Blackwell GPU (the contract hardware_compatible=True advertises), or
• fail explicitly and loudly at load time, the same way the hardware_compatible=False
  control does, if a specific Cask tactic genuinely cannot be made to work cross-architecture.

It should never silently corrupt output at runtime with no raised exception.

Environment

Build information about Torch-TensorRT can be found by turning on debug messages

• Torch-TensorRT Version (e.g. 1.0.0): 2.11.0
• PyTorch Version (e.g. 1.0): 2.11.0
• CPU Architecture: x86-64
• OS (e.g., Linux): Linux
• How you installed PyTorch (conda, pip, libtorch, source): pip
• Build command you used (if compiling from source):
• Are you using local sources or building from archives:
• Python version: 3.11.15
• CUDA version: cuda-toolkit --> 13.0.2
• GPU models and configuration: H100 / A100-80GB / L4 / B200
• Any other relevant information:

Additional context

The model is explicitly in FP16 (.half())