Bug report: cached Torch-TensorRT safetensors upscaler engine produces corrupted frames
Summary
Persistent Torch-TensorRT engine reuse for safetensors upscalers can produce corrupted output frames on the second and subsequent jobs. In my tests with 2x-AnimeSharpV4_RCAN, the first job after a fresh TensorRT compile produced correct preview and video output, while the next job using the cached/refitted upscaler engine produced solid red frames and a very small but otherwise valid video file (frame count OK, mp4 OK, just all frames were just a red picture).
Reproduction
Tested with:
- TensorRT backend, FP16
2x-AnimeSharpV4_RCAN.safetensors- RIFE v4.26 interpolation
- 1672x936 source, 24 -> 60 FPS
- 2x and 4x upscale with the same result
- RTX 5090 / 64GB RAM / 9800 X3D / Windows 11 with latest nvidia drivers
- Clear the relevant TensorRT caches and run the first job.
- The upscaler performs a fresh TensorRT compile (about 39-45 s in my tests).
- Preview and final output are correct.
- Run a second job with the same upscaler/settings.
- Cached/refitted upscaler setup is much faster (about 13 s).
- The log contains warnings such as:
torch_tensorrt.dynamo._refit:/mul_rhs CONSTANT is not found in weight mapping.
torch_tensorrt.dynamo._refit:/div_rhs CONSTANT is not found in weight mapping.
- Preview and encoded frames become corrupted / solid red.
Isolation test
I disabled persistent Dynamo cache reuse only for the safetensors upscaler while leaving the RIFE TensorRT cache enabled.
Result:
- RIFE loaded its cached engine successfully.
- The upscaler compiled fresh for each job.
- Both the first and second jobs produced correct previews and correct video frames.
This isolates the problem to the cached/refitted Torch-TensorRT safetensors upscaler path rather than the RIFE cache or the video encoder.
Current workaround
Returning no persistent cache options from tensorrt_engine_cache_kwargs() avoids the corruption, at the cost of recompiling the upscaler TensorRT engine for every backend job.
It would be great if the durable upscaler cache could eventually be restored with a safe cache/refit configuration rather than permanently disabling the optimization.
Bug report: cached Torch-TensorRT safetensors upscaler engine produces corrupted frames
Summary
Persistent Torch-TensorRT engine reuse for safetensors upscalers can produce corrupted output frames on the second and subsequent jobs. In my tests with
2x-AnimeSharpV4_RCAN, the first job after a fresh TensorRT compile produced correct preview and video output, while the next job using the cached/refitted upscaler engine produced solid red frames and a very small but otherwise valid video file (frame count OK, mp4 OK, just all frames were just a red picture).Reproduction
Tested with:
2x-AnimeSharpV4_RCAN.safetensorstorch_tensorrt.dynamo._refit:/mul_rhs CONSTANT is not found in weight mapping.torch_tensorrt.dynamo._refit:/div_rhs CONSTANT is not found in weight mapping.Isolation test
I disabled persistent Dynamo cache reuse only for the safetensors upscaler while leaving the RIFE TensorRT cache enabled.
Result:
This isolates the problem to the cached/refitted Torch-TensorRT safetensors upscaler path rather than the RIFE cache or the video encoder.
Current workaround
Returning no persistent cache options from
tensorrt_engine_cache_kwargs()avoids the corruption, at the cost of recompiling the upscaler TensorRT engine for every backend job.It would be great if the durable upscaler cache could eventually be restored with a safe cache/refit configuration rather than permanently disabling the optimization.