convert float16 weight to bfloat16 for FP8 models#4276
convert float16 weight to bfloat16 for FP8 models#4276lvhan028 merged 2 commits intoInternLM:mainfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR addresses issue #4261 by adding support for converting float16 weights to bfloat16 format in FP8 models, specifically for the Qwen/Qwen3-4B-Instruct-2507-FP8 model where parameters like *.weight_scale_inv are stored in half precision but require bfloat16 for compatibility with turbomind FP8 kernels.
Changes:
- Added float16 to bfloat16 conversion in the
process_fp8function
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lmdeploy/turbomind/deploy/policy.py
Outdated
| return x.view(dtype=torch.uint8) | ||
| elif kind != 'weight_scale_inv' and x.dtype == torch.float: | ||
| return x.to(dtype=torch.bfloat16) | ||
| elif x.dtype == torch.float16: |
There was a problem hiding this comment.
The new float16 to bfloat16 conversion does not respect the 'weight_scale_inv' exclusion that exists for float32 tensors (line 65). This inconsistency means that weight_scale_inv parameters will be converted from float16 to bfloat16, but not from float32 to bfloat16. Consider whether this condition should also check kind != 'weight_scale_inv' to maintain consistency with the existing logic.
| elif x.dtype == torch.float16: | |
| elif kind != 'weight_scale_inv' and x.dtype == torch.float16: |
fix #4261
In the model Qwen/Qwen3-4B-Instruct-2507-FP8, some parameters like "*.weight_scale_inv" are in half precision. However, the turbomind FP8 kernel is only compatible with bfloat16.
This PR implements a temporary workaround by converting Half-precision weights to the bfloat16 format.