[tmva][sofie] Generate ONNX models and refs instead of binaries in repo#22799
Open
guitargeek wants to merge 1 commit into
Open
[tmva][sofie] Generate ONNX models and refs instead of binaries in repo#22799guitargeek wants to merge 1 commit into
guitargeek wants to merge 1 commit into
Conversation
The SOFIE ONNX unit tests relied on 145 binary .onnx files checked into tmva/sofie/test/input_models, plus ~108 frozen reference headers (input_models/references/*.ref.hxx) whose expected outputs were computed once from the exact weights inside those binaries. This made it impossible to tell from the repository what the models contain, and impossible to regenerate them (some were exported with pytorch versions as old as 1.5). Replace both with a single script, tmva/sofie/test/generate_input_models.py: * Each model is built with the onnx helper API in a make_<Name>() function, so graph structure, attributes, shapes and initializers are readable and reviewable. Large weight tensors that used to be opaque random blobs are seeded-random via _random_tensor(), with pytorch-like 1/sqrt(fan_in) scaling. * The inputs for the value-based tests are defined in TEST_INPUTS, and the expected outputs are computed with onnx's ReferenceEvaluator and written to references/<Name>.ref next to the generated models. Where the evaluator is wrong or unimplemented (MaxPool with asymmetric padding, Mean with multidirectional broadcasting, bidirectional or batchwise RNN/LSTM/GRU), numpy fallbacks implement the ONNX operator definitions directly. * The script runs as the new SofieGenerateModels_ONNX ctest, which the other SOFIE ONNX tests require as a fixture. CMake gets the model list at configure time from `generate_input_models.py --list`, which works without the onnx module. TestCustomModelsFromONNX.cxx and TestCladAutodiff.cxx read the inputs and expected outputs at runtime through a small reader in test_helpers.h instead of including frozen headers. SOFIE's results are thus checked against an independent implementation instead of a snapshot of its own past output. While migrating, all computed references were validated to reproduce the frozen .ref.hxx values with the original weights before switching to seeded-random weights. The ONNX model tests now require the onnx python package; they are disabled with a warning if it is missing or has the broken version 1.19.0. The TMVA_SOFIE_ONNX.C tutorial input is likewise generated at configure time instead of being copied from the test sources. 🤖 Done with the help of AI.
Test Results 23 files 23 suites 3d 15h 55m 55s ⏱️ Results for commit b8d4f28. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SOFIE ONNX unit tests relied on 145 binary .onnx files checked into tmva/sofie/test/input_models, plus ~108 frozen reference headers (input_models/references/*.ref.hxx) whose expected outputs were computed once from the exact weights inside those binaries. This made it impossible to tell from the repository what the models contain, and impossible to regenerate them (some were exported with pytorch versions as old as 1.5).
Replace both with a single script, tmva/sofie/test/generate_input_models.py:
generate_input_models.py --list, which works without the onnx module.TestCustomModelsFromONNX.cxx and TestCladAutodiff.cxx read the inputs and expected outputs at runtime through a small reader in test_helpers.h instead of including frozen headers. SOFIE's results are thus checked against an independent implementation instead of a snapshot of its own past output.
While migrating, all computed references were validated to reproduce the frozen .ref.hxx values with the original weights before switching to seeded-random weights.
The ONNX model tests now require the onnx python package; they are disabled with a warning if it is missing or has the broken version 1.19.0. The TMVA_SOFIE_ONNX.C tutorial input is likewise generated at configure time instead of being copied from the test sources.
🤖 Done with the help of AI.