Skip to content

[https://nvbugs/5527956][fix] AutoDeploy: fix IMA due to outdated metadata - #140

Merged
lucaslie merged 1 commit into
feat/ad_linear_attentionfrom
ll/fix_ima_for_feat
Sep 25, 2025
Merged

[https://nvbugs/5527956][fix] AutoDeploy: fix IMA due to outdated metadata#140
lucaslie merged 1 commit into
feat/ad_linear_attentionfrom
ll/fix_ima_for_feat

Conversation

@lucaslie

Copy link
Copy Markdown

cherry-picked from NVIDIA#8002

…es with better reset/sizing

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 25, 2025 20:18

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes an issue in AutoDeploy where outdated metadata was causing problems with IMA (Inflight Batching Attention). The changes primarily focus on improving cache management, parameter validation, and logging consistency.

  • Updates KV cache resizing logic to use more specific cache name matching
  • Improves parameter handling for max_num_tokens and cache page calculations
  • Adds better error checking and logging for cache operations

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
kvcache.py Removes exception handling around cache resizing and unused logger import
cleanup_input_constraints.py Adds heuristic multiplier to constraint calculations for safety margin
interface.py Updates cache name matching to be more specific (k_cache/v_cache vs generic "cache")
ad_executor.py Removes redundant logging statement
attention_interface.py Improves max_num_tokens parameter handling, page calculation logic, and adds validation checks

Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.

Comment thread tensorrt_llm/_torch/auto_deploy/transform/library/kvcache.py
Comment thread tensorrt_llm/_torch/auto_deploy/custom_ops/attention_interface.py
@lucaslie
lucaslie merged commit 75b53f2 into feat/ad_linear_attention Sep 25, 2025
3 checks passed
@lucaslie
lucaslie deleted the ll/fix_ima_for_feat branch September 26, 2025 17:12
lucaslie added a commit that referenced this pull request Sep 29, 2025
…es with better reset/sizing (#140)

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
nvchenghaoz pushed a commit that referenced this pull request Oct 1, 2025
…es with better reset/sizing (#140)

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
nvchenghaoz pushed a commit that referenced this pull request Oct 3, 2025
…es with better reset/sizing (#140)

Signed-off-by: Lucas Liebenwein <11156568+lucaslie@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants