Skip to content

Add support for overlapping wgrad NCCL AG with dgrad GEMM - #1849

Merged
timmoon10 merged 3 commits into
NVIDIA:mainfrom
djns99:djns99/mxfp8_naive_wgrad_overlap
Jun 13, 2025
Merged

Add support for overlapping wgrad NCCL AG with dgrad GEMM#1849
timmoon10 merged 3 commits into
NVIDIA:mainfrom
djns99:djns99/mxfp8_naive_wgrad_overlap

Conversation

@djns99

@djns99 djns99 commented Jun 4, 2025

Copy link
Copy Markdown
Contributor

Description

Adds support for overlapping the wgrad All Gather operation with dgrad GEMM

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

  • Add new API to UB object for obtaining reference to communication stream
  • Update *Linear layers to start wgrad allgather after dgrad communication completes

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@djns99
djns99 force-pushed the djns99/mxfp8_naive_wgrad_overlap branch from e48df51 to 8120e54 Compare June 4, 2025 02:14
@timmoon10

timmoon10 commented Jun 4, 2025

Copy link
Copy Markdown
Member

/te-ci pytorch L1

@timmoon10
timmoon10 self-requested a review June 4, 2025 19:00
erhoo82
erhoo82 reviewed Jun 6, 2025
View reviewed changes
Comment thread transformer_engine/pytorch/module/linear.py Outdated
@djns99
djns99 force-pushed the djns99/mxfp8_naive_wgrad_overlap branch from 8120e54 to a2382fd Compare June 9, 2025 01:52
@djns99

djns99 commented Jun 9, 2025

Copy link
Copy Markdown
Contributor Author

/te-ci pytorch L1

1 similar comment
@timmoon10

timmoon10 commented Jun 10, 2025

Copy link
Copy Markdown
Member

/te-ci pytorch L1

@djns99
djns99 force-pushed the djns99/mxfp8_naive_wgrad_overlap branch 2 times, most recently from 6baa1e3 to 8b0d377 Compare June 10, 2025 01:37
timmoon10
timmoon10 previously approved these changes Jun 11, 2025
View reviewed changes

@timmoon10 timmoon10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread transformer_engine/pytorch/module/linear.py Outdated
Comment thread setup.py Outdated
dastokes and others added 2 commits June 11, 2025 14:11
dastokes @djns99
Add support for overlapping wgrad NCCL AG with dgrad GEMM
d704f3e 
Signed-off-by: djns99 <40156487+djns99@users.noreply.github.com>
@djns99
Remove unused wait on memcpy API from UB
7e734f6 
Signed-off-by: djns99 <40156487+djns99@users.noreply.github.com>
@djns99
djns99 force-pushed the djns99/mxfp8_naive_wgrad_overlap branch from bc49936 to 7e734f6 Compare June 11, 2025 21:11
@djns99
Add better commenting to MXFP8 overlap
68c7bb2 
Signed-off-by: djns99 <40156487+djns99@users.noreply.github.com>
@djns99 djns99 changed the title Draft: Add support for overlapping wgrad NCCL AG with dgrad GEMM Add support for overlapping wgrad NCCL AG with dgrad GEMM Jun 11, 2025
@timmoon10

timmoon10 commented Jun 11, 2025

Copy link
Copy Markdown
Member

/te-ci pytorch L1

@timmoon10
timmoon10 merged commit d90ced7 into NVIDIA:main Jun 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timmoon10 timmoon10 timmoon10 approved these changes
+1 more reviewer
@erhoo82 erhoo82 erhoo82 left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@djns99 @timmoon10 @erhoo82