[PyTorch] Declare fusible ops as custom ops (1/3): framework + Bias - #27
Closed
pggPL wants to merge 3 commits into
Closed
[PyTorch] Declare fusible ops as custom ops (1/3): framework + Bias#27pggPL wants to merge 3 commits into
pggPL wants to merge 3 commits into
Conversation
… glue Registers an op's forward and backward as two independent two-tier custom ops and returns callables for both, leaving autograd to the caller. This lets a pipeline-level autograd.Function (which Dynamo traces as a higher-order op) group the forward and backward passes differently, as ops.OperationFuser does. Reuses the existing schema/adapter/TensorSpec machinery; register_custom_op is untouched. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Splits Bias into config resolution, pure compute and ctx saving, then registers the compute halves via register_op_halves. resolve_fwd_args reads module config and global FP8 state, so it stays in the traced region where Dynamo guards those reads; the impls take everything as arguments and never touch self. op_forward/op_backward keep their signatures and drive the same impls, so the eager path is unchanged. Adds tests/pytorch/test_ops_custom_ops.py with the fake-vs-real conformance harness every subsequent op will reuse. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
A custom op may not return one of its own inputs, which Bias did whenever the grad input is grad_output unchanged. Cloning would cost a full-size copy on the common unquantized path, so the impl returns None for that slot and the caller substitutes grad_output. Pass-through grads are common (Identity, Reshape, ConstantScale, Quantize), so this is the convention those ops will follow too. Signed-off-by: Pawel Gadzinski <pgadzinski@nvidia.com>
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
First step towards
torch.compile(fullgraph=True)support fortransformer_engine.pytorch.ops.Sequential.The plan is to keep the pipeline-level
_OperationFuserAutogradFunctionand let Dynamo trace it as ahigher-order op, with each fusible operation calling its own custom op inside. That keeps the forward
and backward fusion layouts independent (so the backward-only fusions survive), keeps
OperationContextinside the traced scope, and bounds op registration to one entry per op class.This PR does only the per-op half of that: make every basic/fused operation declarable as a
custom op, with tests. Wiring the fuser up is a separate PR, so nothing in the eager path changes yet.
Type of change
Changes
register_op_halves(dynamo/custom_op.py): registers an op's forward and backward as twoindependent two-tier custom ops and returns callables for both, without registering autograd.
A caller wiring the halves into its own
autograd.Functioncan then group the forward and backwardpasses differently, which is exactly what
OperationFuserdoes. Reuses the existingschema/adapter/
TensorSpecmachinery;register_custom_opis untouched.Biasconverted as the template: split intoresolve_fwd_args(reads module config and globalFP8 state, so it stays in the traced region where Dynamo guards those reads), pure impls that take
everything as arguments and never touch
self, and data-free fakes.op_forward/op_backwardkeep their signatures and drive the same impls, so the eager path is unchanged.
tests/pytorch/test_ops_custom_ops.py: the fake-vs-real conformance harness every subsequent opwill reuse, plus numerics-against-eager and
fullgraph=Truetests forBias.Note on the pass-through-gradient convention
A custom op may not return one of its own inputs.
Bias's backward returnsgrad_outputunchangedwhenever there is no grad-input quantizer, which trips that check, and cloning would cost a full-size
copy on the common path. The impl therefore returns
Nonefor that slot and the caller substitutesgrad_output. Pass-through grads are common (Identity,Reshape,ConstantScale,Quantize), sothe remaining ops will follow the same convention.
Checklist: