Output Tensor Shape Validation b/w ONNX inference and ORT - #7252
Conversation
…transpose test. Ignoring pad and reduction test
|
/azp run Linux CPU CI Pipeline, Linux CPU x64 NoContribops CI Pipeline, Linux GPU CI Pipeline, Linux GPU TensorRT CI Pipeline, Linux OpenVINO CI Pipeline, MacOS CI Pipeline, MacOS NoContribops CI Pipeline, Windows CPU CI Pipeline, Linux CPU Minimal Build E2E CI Pipeline |
|
Commenter does not have sufficient privileges for PR 7252 in repo microsoft/onnxruntime |
|
/azp run Windows GPU CI Pipeline, WIndows GPU TensorRT CI Pipeline, centos7_cpu, centos7_cpu (linux_centos_ci Debug), centos7_cpu (linux_centos_ci Release), orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-distributed, orttraining-amd-gpu-ci-pipeline, Linux Nuphar CI Pipeline |
|
Commenter does not have sufficient privileges for PR 7252 in repo microsoft/onnxruntime |
|
/azp run Windows GPU CI Pipeline, WIndows GPU TensorRT CI Pipeline, centos7_cpu, centos7_cpu (linux_centos_ci Debug), centos7_cpu (linux_centos_ci Release), orttraining-linux-ci-pipeline, orttraining-linux-gpu-ci-pipeline, orttraining-distributed, orttraining-amd-gpu-ci-pipeline, Linux Nuphar CI Pipeline |
|
Azure Pipelines successfully started running 6 pipeline(s). |
…ORT tests which are failing because of this validation
…ead of throwing an exception
…ORT tests which are failing because of this validation
…ead of throwing an exception
…s/onnxruntime into output-shape-validation
23b88ca to
484293f
Compare
fdwr
left a comment
fdwr
left a comment
There was a problem hiding this comment.
Reapproving. ORT already signed off, but Sumit encountered a bad merge and rewound it to an earlier commit.
Remove duplicate function "GetLogger()"
Fixed typo in method name "TestConvTransposeOpInitializer"
fdwr
left a comment
fdwr
left a comment
There was a problem hiding this comment.
Reapproving after bad merge fixed (duplicate OptimizerExecutionFrame::GetLogger and test case name) and CI's pass. No other changes since ORT (Pranav and Hari) last signed off.
4 participants
Background:
During the execution of a model on CPU, output tensor shape generated by ONNX inference needs to be validated against the output tensor shape generated by ORT.
Today, this validation happens only if the space for output tensor is pre-allocated. But, to make the behavior consistent with GPU execution provider (DML), this validation needs to happen irrespective of the space pre-allocation.
Motivation:
Bug raised by a user, when a in-consistency has been seen b/w the CPU ep and GPU ep. Github Bug: 6075 ,
Implementation:
Before the actual execution of a operator on cpu, this validation will happen. As of now, if this validation fails, it will display/log a warning message stating the validation failure and continue with the execution. But ideally to make it consistent with GPU behavior, ORT should also stop the execution then and there if the validation fails. It is done like this for now because there are multiples existing tests fail because of this validation failure. First those tests need to be fixed, then ORT should throw an exception and stop the execution. Known tests which were failing because of validation failure:
• PadOpTest.Pad_Edge_DimWithZeroInput
• ReductionOpTest.ReduceDimWithZero
• Loop.SubgraphInputShadowsOuterScopeValue
There were more tests which were failing, but actual reason needs to be investigated. Few of them were:
• CudaKernelTest.LayerNorm_SmallSizeTensor_IntermediateAxis
• CudaKernelTest.SimplifiedLayerNorm_SmallSizeTensor_IntermediateAxis
This was the Azure build but it is not present anymore in the pipeline.