Skip to content

Add FLOAT8E8M0 data type support in ONNX Runtime - #28381

Merged
tianleiwu merged 12 commits into
mainfrom
copilot/add-float8e8m0-support
May 9, 2026
Merged

Add FLOAT8E8M0 data type support in ONNX Runtime#28381
tianleiwu merged 12 commits into
mainfrom
copilot/add-float8e8m0-support

Conversation

Copilot AI commented May 6, 2026

Copy link
Copy Markdown
Contributor

Description

This PR adds support for the FLOAT8E8M0 data type in ONNX Runtime, following the same pattern used for existing Float8 types (Float8E4M3FN, Float8E4M3FNUZ, Float8E5M2, Float8E5M2FNUZ).

FLOAT8E8M0 Format

FLOAT8E8M0 is an 8-bit floating point format with:

  • 8 exponent bits
  • 0 mantissa bits
  • No sign bit (all values are positive)
  • Values are powers of two: 2^(val - 127)
  • 0xFF is reserved for NaN
  • No infinity, no negative values, no denormals

This format is useful as a scaling factor type in microscaling (MX) quantization formats.

Changes

Core Type Definition

  • include/onnxruntime/core/common/float8.h - Added Float8E8M0 struct with float conversion routines and std::numeric_limits specialization. The float→Float8E8M0 conversion correctly handles NaN (including negative NaN) and infinity by checking these special values before sign-dependent logic, consistent with how Float8E4M3FN handles them.

C API

  • include/onnxruntime/core/session/onnxruntime_c_api.h - Added ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT8E8M0 enum value at ordinal 24 to match the ONNX protobuf spec (FLOAT8E8M0 = 24). Note: UINT2 and INT2 shift to ordinals 25 and 26 respectively to align with the spec. Language bindings (Java, C#, JS) will need corresponding updates in a follow-up PR.

Framework Integration

  • onnxruntime/core/framework/element_type_lists.h - Added to AllFixedSizeIRv9, AllIRv9, AllFloat8 type lists
  • include/onnxruntime/core/framework/to_tensor_proto_element_type.h - Added template specialization
  • include/onnxruntime/core/framework/data_types_internal.h - Added dispatch cases
  • include/onnxruntime/core/framework/data_types.h - Added to IsTensorContainedType and IsSparseTensorContainedType
  • onnxruntime/core/framework/utils.h - Added GetONNXTensorElementDataType specialization
  • onnxruntime/core/framework/data_types.cc - Full type registration (tensor, sparse, seq, optional, prim)
  • onnxruntime/core/framework/tensorprotoutils.cc - UnpackTensor, size calculation, serialization
  • onnxruntime/core/framework/tensor_type_and_shape.cc - Type mapping
  • onnxruntime/core/framework/onnxruntime_map_type_info.cc - Type mapping

Shared Library Provider Bridge

  • onnxruntime/core/providers/shared_library/provider_interfaces.h
  • onnxruntime/core/providers/shared_library/provider_bridge_provider.cc
  • onnxruntime/core/providers/shared_library/provider_wrappedtypes.h
  • onnxruntime/core/session/provider_bridge_ort.cc

Tests

  • onnxruntime/test/framework/float8e8m0_test.cc - Unit tests for Float8E8M0 type including conversion, rounding, NaN (positive and negative), infinity (positive and negative), negative values, operators, and batch conversion
  • onnxruntime/test/onnx/tensorprotoutils.cc - Test infrastructure updates
  • onnxruntime/test/unittest_util/checkers.cc - Test checker updates

Motivation and Context

ONNX has added the FLOAT8E8M0 data type. This PR adds the corresponding support in ONNX Runtime so that models using this data type can be loaded and processed correctly.

Copilot AI and others added 3 commits May 6, 2026 09:06
- Add Float8E8M0 struct to float8.h with conversion routines
- Add ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT8E8M0 to C API enum
- Register Float8E8M0 in type lists, dispatch macros, and type registration
- Add support in tensorprotoutils, tensor_type_and_shape, map_type_info
- Update shared library provider bridge interfaces

Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/69c13152-073e-4f2f-91df-13d0804982be

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
tianleiwu added 2 commits May 6, 2026 12:39
- Fix ToFloat() to correctly return 2^(-127) when val=0. Previously,
  it mapped val=0 directly to float32 exponent field 0 which encodes
  +0.0 in IEEE 754. Now handles val=0 as a special case using the
  denormalized float32 representation (exponent=0, mantissa=2^22).
- Regenerate docs/OperatorKernels.md to include float8e8m0 in operator
  type constraints after adding the type to AllFixedSizeIRv9/AllIRv9/
  AllFloat8 lists.
@tianleiwu
tianleiwu marked this pull request as ready for review May 6, 2026 21:51
@tianleiwu
tianleiwu requested review from hariharans29 and xadupre May 6, 2026 21:56
@tianleiwu
tianleiwu marked this pull request as draft May 6, 2026 22:13
@tianleiwu
tianleiwu marked this pull request as ready for review May 7, 2026 17:30

@tianleiwu tianleiwu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

The PR provides comprehensive FLOAT8E8M0 support following established patterns. The framework integration (type dispatch, registration, serialization, shared-library bridge) is thorough and correct.

However, there is a correctness bug in the float→Float8E8M0 conversion that causes negative NaN inputs to silently produce valid outputs instead of preserving NaN. This should be fixed before merge.

Additional notes

  • The ONNX_TENSOR_ELEMENT_DATA_TYPE_FLOAT8E8M0 enum insertion at ordinal 24 correctly aligns with the ONNX protobuf spec (FLOAT8E8M0 = 24, UINT2 = 25, INT2 = 26). This shifts the previously-incorrect UINT2/INT2 ordinals on main — language bindings (Java TensorInfo.java, C# OnnxMl.cs, JS wasm-common.ts) will need corresponding updates.
  • Python pybind numpy dtype mapping in onnxruntime_pybind_mlvalue.cc is not updated, which will be needed for end-to-end Python usage.

Comment thread include/onnxruntime/core/common/float8.h
Comment thread include/onnxruntime/core/session/onnxruntime_c_api.h Outdated
Move NaN and infinity checks before the sign check so that
negative NaN (0xFFC00000) correctly produces NaN output (0xFF)
instead of being silently converted to the smallest positive value.
Also handle negative infinity separately from positive infinity.

Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/dfd9000d-bd25-40b2-ba51-a12485bbabb6

Co-authored-by: tianleiwu <30328909+tianleiwu@users.noreply.github.com>
Copilot AI requested a review from tianleiwu May 7, 2026 17:57

@tianleiwu tianleiwu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

The FLOAT8E8M0 data type support is well-implemented and follows established patterns for other Float8 types. The framework integration is comprehensive (tensor, sparse tensor, sequence, optional types + provider bridge). The conversion logic is correct with proper handling of NaN, infinity, and negative values.

Positives:

  • Correct alignment with ONNX spec (enum value 24)
  • Thorough framework plumbing following existing Float8 patterns
  • Good unit test coverage including edge cases
  • Proper fix for negative NaN priority (e07a9dd) and val=0 denorm (3acbcf9)

Suggestions (non-blocking):

  1. Add a test for -0.0f conversion — the code correctly maps it to 0x00 via the exponent == 0 path, but an explicit test documents this behavior
  2. Add a round-trip test documenting that Float8E8M0(0.0f).ToFloat() returns ~5.88e-39 (not 0.0f) since E8M0 cannot represent zero
  3. A brief comment on the rounding mode in the conversion constructor would help future spec compliance verification

Comment thread include/onnxruntime/core/common/float8.h
Comment thread onnxruntime/test/framework/float8e8m0_test.cc
tianleiwu added 2 commits May 7, 2026 16:42
- Add OE-MX spec rounding mode comment to Float8E8M0 conversion
- Add NegativeZero test: -0.0f maps to 0x00 like +0.0f
- Add ZeroRoundTrip test: documents that val=0 round-trips to 2^(-127)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants