[PyTorch] torch.compile for an OperationFuser group holding one operation - #29
Open
pggPL wants to merge 3 commits into
Open
[PyTorch] torch.compile for an OperationFuser group holding one operation#29pggPL wants to merge 3 commits into
pggPL wants to merge 3 commits into
Conversation
register_custom_op now defines an operation's forward and backward as two independent two-tier custom ops and hands back both, leaving autograd to the caller. That is what lets a pipeline-level autograd.Function decide how the two are wired, and so group the forward and backward passes differently -- which is what ops.OperationFuser does. The variant that wires autograd itself keeps the old behaviour under register_custom_op_with_autograd, and is now built on the same registration: the pair is the primitive, autograd is what the other one adds. About two thirds of the two bodies were the same code before. BasicOperation gains the plumbing an operation needs to opt in: declare two argument containers and implement four compute classmethods, and __init_subclass__ registers the custom ops while op_forward / op_backward are written once in the base. compile_unsupported_reason lets an operation say why it cannot be compiled -- it sits here rather than on the args, as Linear has it, because in ops/ the compile boundary is the fuser group, not the operation. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
A group whose operations declare their compute halves now runs through their custom ops under torch.compile(fullgraph=True). The pipeline-level autograd.Function is traced as a higher-order op, which is what will later let its forward and backward walk different op groupings. Four side effects reached outside the higher-order op's scope and had to go: - OperationContext objects are created in the forward, but the backward is a separate subgraph, so writing to them there mutates an enclosing scope; the backward copies them into its own scope instead; - requires_grad_ on an output, which AOTAutograd's functionalization drops anyway -- autograd marks the outputs of an apply() itself; - _do_not_clear on inputs and outputs; - warnings.warn from the gate, which is not traceable, so the reason is reported from the eager path only. They are gated on being traced rather than on using the custom ops. Under fullgraph there is no leaving the graph, so an unsupported operation does not fall back: the pipeline is traced either way and only the choice of implementation changes. Sequential builds its module groups outside the forward pass, since that constructs nn.Modules. Tested with a test-only operation, so the fuser's path does not depend on which real operations happen to declare their halves. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This was referenced Aug 5, 2026
An operation lists the forward kwargs it takes in fwd_kwarg_names. They are resolved into its args container like any other config, in the traced Python where Dynamo guards them, so they reach the custom op through the existing schema -- a value is guarded, a tensor is lifted into the graph, and a quantized one crosses as its inner buffers. An undeclared kwarg still sends the whole group to eager. That is not a schema limitation, as the old message implied: the kwargs that remain are the grouped operations' preallocated buffers, which the op writes to, and a custom op may not mutate a tensor from an enclosing scope. A kwarg carries no gradient. This matches the eager path, where kwargs never entered the autograd graph either, and is why only read-only ones are accepted. The fuser test helper now builds a separate model for the eager and the compiled pass. Previously both shared one model and the eager pass ran first to produce the reference, so the compiled pass was always traced on a model whose module groups, fusions and pre_first_fuser_forward had already run. Those paths are now traced as well. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
First step towards
torch.compile(fullgraph=True)support fortransformer_engine.pytorch.ops.The approach: keep the pipeline-level
_OperationFuserAutogradFunctionand let Dynamo trace it as ahigher-order op, with each fusible operation calling its own custom op inside. That keeps the forward
and backward fusion layouts independent (so the backward-only fusions survive), keeps
OperationContextinside the traced scope, and bounds op registration to one entry per op class.This PR makes that work for an
OperationFusergroup holding one operation. It converts no realoperation -- the fuser's path is exercised by a test-only operation, so it does not depend on which
operations happen to declare their compute halves. Converting the operations themselves is a follow-up.
Type of change
Changes
register_custom_opnow defines an operation's forward and backward as two independenttwo-tier custom ops and hands back both, leaving autograd to the caller. That is what lets a
pipeline-level
autograd.Functiondecide how the two are wired. The variant that wires autograditself keeps the old behaviour under
register_custom_op_with_autogradand is built on the sameregistration -- the pair is the primitive, autograd is what the other one adds. About two thirds of
the two bodies were the same code before.
BasicOperationgains the plumbing an operation needs to opt in: declare two argument containersand implement four compute classmethods, and
__init_subclass__registers the custom ops whileop_forward/op_backwardare written once in the base.compile_unsupported_reasonlets anoperation say why it cannot be compiled; it sits on the operation rather than on the args (where
Linearkeeps it) because inops/the compile boundary is the fuser group.OperationFuserruns a supported group through its operations' custom ops under compile.Sequentialbuilds its module groups outside the forward pass, since that constructsnn.Modules.Four side effects had to go
All of them reached outside the higher-order op's scope, and each was found by running the thing:
OperationContextobjects are created in the forward, but the backward is a separate subgraph, sowriting to them there mutates an enclosing scope. The backward copies them into its own scope.
requires_grad_on an output -- AOTAutograd's functionalization drops it anyway, and autograd marksthe outputs of an
apply()itself._do_not_clearon inputs and outputs.warnings.warnfrom the gate, which is not traceable; the reason is reported from the eager pathonly, so a configuration that is never run eagerly falls back silently.
They are gated on being traced, not on using the custom ops. Under
fullgraph=Truethere is noleaving the graph, so an unsupported operation does not "fall back": the pipeline is traced either way
and only the choice of implementation changes.
Testing
test_torch_compile.pygains a test-only_ScaleOpand two tests: a single-operation group compilesand matches eager (output, input gradient, parameter gradient), and an operation without the compute
halves still runs, on its eager implementation.
test_torch_compile.py+test_fusible_ops.py: 1587 passed, 1187 skipped. Lint 10.00/10. RTX Ada.Gated out and untouched: multi-operation groups, fused operations, grouped operations, userbuffers,
delayed scaling, FP8 block scaling.
Checklist: