softmax_type != vanilla support thd format

[nicosouth1]

hello, i find in TransformerEngine, it will close Fusedattn when i use softmax_type==learnable and thd format

[ptrendx]

@cyanguwa, could you take a look?
@nicosouth1, what are the TE and cuDNN versions you use?

[nicosouth1]

thank you！
i use 2.10.0 TE and 9.15.1 cudnn.

[cyanguwa]

Hi @nicosouth1, thanks for bringing this up. We had some numerical issues with sink attention in THD format, and we're working with cuDNN on fixing it. The fix is likely going into cuDNN 9.18 or 9.19, and TE 2.12 or TE 2.13 (releasing in Jan or Feb 2026) judging by the current timelines. Sorry about the inconvenience and thanks for your patience.

[nicosouth1]

thank you !

[jordane95]

I'm facing the same issue NVIDIA-NeMo/Megatron-Bridge#1782