Skip to content

bugfix/incorrect lora out dims - #2484

Closed
akhoroshev wants to merge 2 commits into
NVIDIA:mainfrom
akhoroshev:bugfix/lora-module-fix-dims
Closed

bugfix/incorrect lora out dims#2484
akhoroshev wants to merge 2 commits into
NVIDIA:mainfrom
akhoroshev:bugfix/lora-module-fix-dims

Conversation

@akhoroshev

@akhoroshev akhoroshev commented Nov 22, 2024

Copy link
Copy Markdown
Contributor

I have in-house model for which hidden_size != num_heads * head_size (attention dimension is higher then hidden_size) and lora doesn't work properly. But after this fix, everything works exactly the same as in Python (same generations).

My target modules:

  "target_modules": [
    "v_proj",
    "q_proj",
    "k_proj",
    "down_proj"
  ],

It probably makes sense to make similar changes for kCROSS_ATTN_Q,kCROSS_ATTN_V,kCROSS_ATTN_K

@hello-11 hello-11 added the triaged Issue has been triaged by maintainers label Nov 25, 2024
@DanBlanaru DanBlanaru mentioned this pull request Feb 6, 2025
@juney-nvidia juney-nvidia added the Community want to contribute PRs initiated from Community label Mar 28, 2025
@juney-nvidia

Copy link
Copy Markdown
Collaborator

Close this since it already lands into 0.17.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Community want to contribute PRs initiated from Community triaged Issue has been triaged by maintainers

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants