Let a C++ application use the ExecuTorch TensorRT delegate from an install - #4458
Draft
shoumikhin wants to merge 1 commit into
Draft
Let a C++ application use the ExecuTorch TensorRT delegate from an install#4458shoumikhin wants to merge 1 commit into
shoumikhin wants to merge 1 commit into
Conversation
shoumikhin
force-pushed
the
executorch-wheel-consumable
branch
10 times, most recently
from
August 2, 2026 22:47
8b572f6 to
c2e8f0e
Compare
shoumikhin
force-pushed
the
executorch-wheel-consumable
branch
16 times, most recently
from
August 3, 2026 16:37
0ac7b67 to
0f8c1ad
Compare
shoumikhin
force-pushed
the
executorch-wheel-consumable
branch
5 times, most recently
from
August 4, 2026 16:59
542bb47 to
c30c1cd
Compare
Two problems in the new verification path. The install check looked for the delegate under a hardcoded lib directory, but the install honors the platform library directory, which is lib64 on several distributions. A completely successful build would have been reported as a failed install. The configure now pins the directory and the check searches for the file rather than assuming where it landed. The device-input path handed a device pointer to the runtime without checking how that input is supplied. A memory-planned input is copied into the plan with a host memcpy, so a device pointer there would be read from the host. Only a non-planned input has its pointer aliased, which is what makes device memory safe. The runner now reports that clearly instead of corrupting memory. Test plan: confirmed the install check finds the library when it lands in either lib or lib64, and that a host-side copy of a device pointer is what the runtime would do for a memory-planned input.
shoumikhin
force-pushed
the
executorch-wheel-consumable
branch
from
August 5, 2026 04:13
c30c1cd to
f23ba48
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Torch-TensorRT can already export an ExecuTorch model that runs through TensorRT. What it cannot
do is give you a prebuilt delegate: the runtime piece that actually executes the TensorRT parts
of the model has to be compiled from source, which means a full C++ build with the TensorRT SDK
installed.
This adds a prebuilt delegate and a CMake package for it, so a C++ program can link it from the
installed wheel.
What you get
How you use it
In your build, alongside the ExecuTorch runtime:
In Python, importing the subpackage loads the delegate so it registers itself:
Version pairing
A delegate built against one ExecuTorch release cannot be assumed to work with another, because
the two share C++ types that are free to change between releases. So the delegate records the
ExecuTorch version it was built against and requires exactly that version, both in CMake and in
its Python metadata.
That is stricter than a normal lower bound, and deliberately so: a mismatched pair fails at
find_packagetime with a clear message rather than at run time with something confusing.Status
The prebuilt delegate is off by default. Turning it on requires an ExecuTorch release that
publishes the GPU wheels it pins, which is not available yet, so a release build that enabled it
would produce a wheel whose dependency no package index can satisfy. Once those wheels publish,
enabling it is a build-flag change plus the release job wiring.
Everything else here is ready: the delegate builds, the CMake package exports correctly, the
version pairing is enforced, and the Python import path loads it.
Tested
Built the delegate on Linux aarch64 with a Jetson device and on x86_64, then from a clean
environment with no source checkout reachable:
find_package(torch_tensorrt_executorch)and linked a C++ program against the delegatealongside the ExecuTorch runtime and CUDA delegate;
program references a symbol from it;
imports when no prebuilt delegate is present.