Skip to content

Use internal quantizer for input to the modules - #1551

Merged
ptrendx merged 1 commit into
NVIDIA:mainfrom
ptrendx:pr_internal_input_quantizer
Mar 10, 2025
Merged

Use internal quantizer for input to the modules#1551
ptrendx merged 1 commit into
NVIDIA:mainfrom
ptrendx:pr_internal_input_quantizer

Conversation

@ptrendx

@ptrendx ptrendx commented Mar 7, 2025

Copy link
Copy Markdown
Member

Description

FP8 input to the modules is temporary and fully internal variable.

Type of change

  • Documentation change (change only to the documentation, either a fix or a new content)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Infra/Build change
  • Code refactoring

Changes

Please list the changes introduced in this PR:

  • Set input_quantizer to internal for Linear, LayerNormLinear and LayerNormMLP

Checklist:

  • I have read and followed the contributing guidelines
  • The functionality is complete
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes

@ptrendx
Internal quantizer for input to the modules
cf8fdb8 
Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
@ptrendx

ptrendx commented Mar 7, 2025

Copy link
Copy Markdown
Member Author

/te-ci pytorch

timmoon10
timmoon10 approved these changes Mar 8, 2025
View reviewed changes

@timmoon10 timmoon10 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ptrendx
ptrendx merged commit b3e7035 into NVIDIA:main Mar 10, 2025
@ptrendx ptrendx mentioned this pull request Mar 10, 2025
Merged
ptrendx added a commit that referenced this pull request Mar 10, 2025
@ptrendx
Revert "Use internal quantizer for input to the modules (#1551)"
aa0ee4e 
This reverts commit b3e7035.

Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
ptrendx added a commit that referenced this pull request Mar 10, 2025
@ptrendx
Revert "Use internal quantizer for input to the modules" (#1555)
f3a009d 
Revert "Use internal quantizer for input to the modules (#1551)"

This reverts commit b3e7035.

Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
@ksivaman

ksivaman commented Mar 11, 2025

Copy link
Copy Markdown
Member

Will also need some work to be supported with columnwise only tensors (#1505). See pipeline 25177951

yaox12 added a commit to yaox12/TransformerEngine that referenced this pull request Mar 13, 2025
@yaox12
Squashed commit of the following:
c898f84 
commit 25b38220158c7fda26704d1e51baf4ddd05bda86
Author: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Date:   Thu Mar 13 00:07:58 2025 +0000

    [pre-commit.ci] auto fixes from pre-commit.com hooks

    for more information, see https://pre-commit.ci

commit 365a4d99be7671c3f5f6c5487d8961a109b18da5
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Wed Mar 12 17:00:15 2025 -0700

    Update FIXME to TODO with linked issue.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit bdf01d891b5b75219b14ba28610d595436b22bb2
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Tue Mar 11 20:01:11 2025 -0700

    Run against tensor dumps from internal library.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 733e2a27f96c634238c0b1b73ae67177558f57af
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Tue Mar 11 11:35:42 2025 -0700

    Revert "Use internal quantizer for input to the modules (NVIDIA#1551)"

    This reverts commit b3e7035.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 5c4234802e6cc98065c166679b67d336901edac6
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Tue Mar 11 13:18:15 2025 -0700

    Use 12.9 feature test.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 5e2a5600a62e62e26ddb839493feaea2222790c1
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Thu Mar 6 16:21:28 2025 -0800

    Recipe setup for Linear modules.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 88ac78cae1d7250f39b43d2ad42207410dcf3d77
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Tue Mar 11 17:47:46 2025 -0700

    GEMM simplifications from recipe branch.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit dc2264b222a7fdef4dd645df1a155f945df27eaa
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Tue Mar 11 13:24:05 2025 -0700

    Reflect epilogue availability and test supported epilogues.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit cb9824975e0f800007fb3cf038f64096fbf06b9f
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 17:40:24 2025 -0700

    Remove unecessary type converter change.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 3225ebe2d7fa56e58505aac38e2c7a82c0e79888
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 18:18:18 2025 -0700

    Gemm typo.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 311e85107edc20c330199a64dea482a51539b2d0
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Thu Mar 6 11:17:27 2025 -0800

    Gate feature on CUDA 12.9

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit d9f11cc0721dd715308bd34b94742b64e65af351
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 16:50:56 2025 -0700

    Update NVTE_BLOCK_SCALING for GEMM.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 06aa77d7a786da89108456e7e953d553936cd84b
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Fri Feb 28 15:30:07 2025 -0800

    Add GEMM logic for blockwise quantized tensors.

    GEMM test cases included in pytorch integration.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 2bbb9960c469c49319fe7ebb70604216381c7861
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Wed Mar 12 16:19:38 2025 -0700

    Save memory by dropping reference to saved tensors.

    Issues previously observed are solved.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit ff1b378e742ab8bcf2874456f7b2cef010f284b2
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Wed Mar 12 16:47:18 2025 -0700

    Reuse scale calculation between quantizer refs.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit b4482a45f15e0e8758c4caa5fca8ec4c7dc5fb02
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Wed Mar 12 12:22:55 2025 -0700

    Tests for torch tensor API surface.

    Since the quantized tensor is a tensor
    subclass, these tests exercise torch hooks.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 1f242465581103719f589c2240e9a987062a8a4a
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 18:04:15 2025 -0700

    Update some call sites.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 863a16b71358bdeafb111b20bac28a5d796050b2
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 17:52:10 2025 -0700

    Fix typo.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 7a1bc870b895fd2dc2b40f059e2c559c42bd9985
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 16:06:22 2025 -0700

    Change API to use two scaling mode enums.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit ee425e299b801fe0c384f4647f5b1013265524af
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 15:32:29 2025 -0700

    Use common tensor map API.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 6ca3cadbc81884c1813021913b5049c9c2795517
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 15:23:53 2025 -0700

    Fix merge conflict with name change.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 408f29c73b5e66c5a688d136fe3da5b138863a2f
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 15:20:44 2025 -0700

    Change API name.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 602b984e9484cd11a3b5e20c8bbab7d67eb2e139
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Mon Mar 10 14:16:23 2025 -0700

    MR feedback.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 7149c8b03fbc112e2eeacf42b4200a27bb113f40
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Thu Feb 27 14:01:33 2025 -0800

    Alignment for 1D scaling for GEMM edge case.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 47a3edeca2dc48a63067b8f85bcfa87e6ccad1bc
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Thu Mar 6 09:25:26 2025 -0800

    Apply linting changes.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>

commit 8157fd978d997673cfc9d2551f818d913a2d923f
Author: Keith Wyss <kwyss@nvidia.com>
Date:   Wed Feb 12 13:50:52 2025 -0800

    Blockwise float8 quantizer and quantized tensor class.

    The classes are configurable for 128x128 blocksize
    and 1x128 blocksize via setting block_scaling_dim == 2,1 respectively.

    Scale tensors are stored in a format emenable for matrix multiplication,
    however the integration of matmul is deferred as a separate story.

    Fusions of quantization and DBIAS or activation functions are not yet
    implemented, and the dequantization is currently implemented in torch.

    Tests for quantization are included in C++ and pytorch layers, with
    exact comparison to reference quantizer behavior as well as an attempt
    to hit interesting branches through the API such as tensor creation
    in pytorch and CPP and dequantization of row and columnwise usage.

    Two CUDA kernels for quantization are included, and are direct ports
    of equivalents in the kitchen repository, where a subchannel recipe
    has been used for end to end training.

    Signed-off-by: Keith Wyss <kwyss@nvidia.com>
negvet pushed a commit to negvet/TransformerEngine that referenced this pull request Mar 18, 2025
@ptrendx @negvet
Use internal quantizer for input to the modules (NVIDIA#1551)
5fb359e 
Internal quantizer for input to the modules

Signed-off-by: Przemek Tredak <ptredak@nvidia.com>
Signed-off-by: Evgeny Tsykunov <etsykunov@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@timmoon10 timmoon10 timmoon10 approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@ptrendx @ksivaman @timmoon10