Add FLOAT8E8M0 data type support in ONNX Runtime - #28381
Conversation
- Add Float8E8M0 struct to float8.h with conversion routines - Add ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT8E8M0 to C API enum - Register Float8E8M0 in type lists, dispatch macros, and type registration - Add support in tensorprotoutils, tensor_type_and_shape, map_type_info - Update shared library provider bridge interfaces Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/69c13152-073e-4f2f-91df-13d0804982be Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/69c13152-073e-4f2f-91df-13d0804982be Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/69c13152-073e-4f2f-91df-13d0804982be Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
- Fix ToFloat() to correctly return 2^(-127) when val=0. Previously, it mapped val=0 directly to float32 exponent field 0 which encodes +0.0 in IEEE 754. Now handles val=0 as a special case using the denormalized float32 representation (exponent=0, mantissa=2^22). - Regenerate docs/OperatorKernels.md to include float8e8m0 in operator type constraints after adding the type to AllFixedSizeIRv9/AllIRv9/ AllFloat8 lists.
tianleiwu
left a comment
There was a problem hiding this comment.
Review Summary
The PR provides comprehensive FLOAT8E8M0 support following established patterns. The framework integration (type dispatch, registration, serialization, shared-library bridge) is thorough and correct.
However, there is a correctness bug in the float→Float8E8M0 conversion that causes negative NaN inputs to silently produce valid outputs instead of preserving NaN. This should be fixed before merge.
Additional notes
- The
ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT8E8M0enum insertion at ordinal 24 correctly aligns with the ONNX protobuf spec (FLOAT8E8M0 = 24,UINT2 = 25,INT2 = 26). This shifts the previously-incorrect UINT2/INT2 ordinals onmain— language bindings (JavaTensorInfo.java, C#OnnxMl.cs, JSwasm-common.ts) will need corresponding updates. - Python pybind numpy dtype mapping in
onnxruntime_pybind_mlvalue.ccis not updated, which will be needed for end-to-end Python usage.
Move NaN and infinity checks before the sign check so that negative NaN (0xFFC00000) correctly produces NaN output (0xFF) instead of being silently converted to the smallest positive value. Also handle negative infinity separately from positive infinity. Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/dfd9000d-bd25-40b2-ba51-a12485bbabb6 Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
tianleiwu
left a comment
There was a problem hiding this comment.
Review Summary
The FLOAT8E8M0 data type support is well-implemented and follows established patterns for other Float8 types. The framework integration is comprehensive (tensor, sparse tensor, sequence, optional types + provider bridge). The conversion logic is correct with proper handling of NaN, infinity, and negative values.
Positives:
- Correct alignment with ONNX spec (enum value 24)
- Thorough framework plumbing following existing Float8 patterns
- Good unit test coverage including edge cases
- Proper fix for negative NaN priority (e07a9dd) and val=0 denorm (3acbcf9)
Suggestions (non-blocking):
- Add a test for
-0.0fconversion — the code correctly maps it to0x00via theexponent == 0path, but an explicit test documents this behavior - Add a round-trip test documenting that
Float8E8M0(0.0f).ToFloat()returns~5.88e-39(not0.0f) since E8M0 cannot represent zero - A brief comment on the rounding mode in the conversion constructor would help future spec compliance verification
- Add OE-MX spec rounding mode comment to Float8E8M0 conversion - Add NegativeZero test: -0.0f maps to 0x00 like +0.0f - Add ZeroRoundTrip test: documents that val=0 round-trips to 2^(-127)
Description
This PR adds support for the
FLOAT8E8M0data type in ONNX Runtime, following the same pattern used for existing Float8 types (Float8E4M3FN, Float8E4M3FNUZ, Float8E5M2, Float8E5M2FNUZ).FLOAT8E8M0 Format
FLOAT8E8M0 is an 8-bit floating point format with:
2^(val - 127)This format is useful as a scaling factor type in microscaling (MX) quantization formats.
Changes
Core Type Definition
include/onnxruntime/core/common/float8.h- AddedFloat8E8M0struct with float conversion routines andstd::numeric_limitsspecialization. The float→Float8E8M0 conversion correctly handles NaN (including negative NaN) and infinity by checking these special values before sign-dependent logic, consistent with howFloat8E4M3FNhandles them.C API
include/onnxruntime/core/session/onnxruntime_c_api.h- AddedONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT8E8M0enum value at ordinal 24 to match the ONNX protobuf spec (FLOAT8E8M0 = 24). Note:UINT2andINT2shift to ordinals 25 and 26 respectively to align with the spec. Language bindings (Java, C#, JS) will need corresponding updates in a follow-up PR.Framework Integration
onnxruntime/core/framework/element_type_lists.h- Added to AllFixedSizeIRv9, AllIRv9, AllFloat8 type listsinclude/onnxruntime/core/framework/to_tensor_proto_element_type.h- Added template specializationinclude/onnxruntime/core/framework/data_types_internal.h- Added dispatch casesinclude/onnxruntime/core/framework/data_types.h- Added to IsTensorContainedType and IsSparseTensorContainedTypeonnxruntime/core/framework/utils.h- Added GetONNXTensorElementDataType specializationonnxruntime/core/framework/data_types.cc- Full type registration (tensor, sparse, seq, optional, prim)onnxruntime/core/framework/tensorprotoutils.cc- UnpackTensor, size calculation, serializationonnxruntime/core/framework/tensor_type_and_shape.cc- Type mappingonnxruntime/core/framework/onnxruntime_map_type_info.cc- Type mappingShared Library Provider Bridge
onnxruntime/core/providers/shared_library/provider_interfaces.honnxruntime/core/providers/shared_library/provider_bridge_provider.cconnxruntime/core/providers/shared_library/provider_wrappedtypes.honnxruntime/core/session/provider_bridge_ort.ccTests
onnxruntime/test/framework/float8e8m0_test.cc- Unit tests for Float8E8M0 type including conversion, rounding, NaN (positive and negative), infinity (positive and negative), negative values, operators, and batch conversiononnxruntime/test/onnx/tensorprotoutils.cc- Test infrastructure updatesonnxruntime/test/unittest_util/checkers.cc- Test checker updatesMotivation and Context
ONNX has added the FLOAT8E8M0 data type. This PR adds the corresponding support in ONNX Runtime so that models using this data type can be loaded and processed correctly.