[None][fix] fix FA4 install in devel docker

[o-stoner]

Summary by CodeRabbit

• Chores
  • Added flash-attn-4 (v4.0.0b11) to the development container build pipeline. This new dependency is automatically provisioned during container setup, with optional support for mirror configuration.

Description

FA4's wheel ships only flash_attn/cute/ (no top-level flash_attn/__init__.py), so it must share a directory with the base image's FlashAttention. A non-root pip install introduced in #13788 lands FA4 in ~/.local, separate from the system FlashAttention, so flash_attn.cute can't be found (ModuleNotFoundError) in VisualGen. Pre-install FA4 in the devel image as root, co-located with FlashAttention, so it is functional.

Test Coverage

PR Checklist

Please review the following before submitting your PR:

• PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.

• PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.

• Test cases are provided for new code paths (see test instructions)

• If PR introduces API changes, an appropriate PR label is added - either api-compatible or api-breaking. For api-breaking, include BREAKING in the PR title.

• Any new dependencies have been scanned for license and vulnerabilities

• CODEOWNERS updated if ownership changes

• Documentation updated as needed

• Update tava architecture diagram if there is a significant design change in PR.

• The reviewers assigned automatically/manually are appropriate for the PR.

• Please check this after reviewing the above items as appropriate for this PR.

GitHub Bot Help

To see a list of available CI bot commands, please comment /bot help.

[o-stoner]

/bot run --disable-fail-fast

[coderabbitai]

Actionable comments posted: 0

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dc922068-8e1f-4c6c-b823-84e5ba2bd150

📥 Commits

Reviewing files that changed from the base of the PR and between f6ba936 and beb9bfa.

📒 Files selected for processing (2)

• docker/Dockerfile.multi
• docker/common/install_fa4.sh

---

📝 Walkthrough

Walkthrough

This PR adds flash-attn-4 support to the Docker build pipeline. A new installer script is created to pin flash-attn-4 to version 4.0.0b11 with optional NVIDIA Artifactory mirror routing. The Dockerfile.multi devel stage is updated to invoke this script after the existing etcd installation step.

Changes

Flash-Attn-4 Installation Support

Layer / File(s)	Summary
Flash-Attn-4 installer and Dockerfile integration docker/common/install_fa4.sh, docker/Dockerfile.multi	New install_fa4.sh script with strict tracing, fixed version pinning to 4.0.0b11, conditional NVIDIA Artifactory PyPI mirror routing via GITHUB_MIRROR, and pip3 installation. Dockerfile devel stage invokes the script after etcd setup, passing GITHUB_MIRROR through the environment.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Description check	⚠️ Warning	The PR description includes a clear explanation of the issue and solution but leaves the 'Test Coverage' section empty, which is a required section per the template.	Complete the 'Test Coverage' section by listing the relevant tests that safeguard these Docker image changes or explain why no new tests are needed.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title '[None][fix] fix FA4 install in devel docker' clearly describes the main change: fixing FA4 installation in the development Docker image.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[tensorrt-cicd]

PR_Github #50876 [ run ] triggered by Bot. Commit: beb9bfa Link to invocation

[o-stoner]

/bot run --stage-list "Build-Docker-Images"

[tensorrt-cicd]

PR_Github #50912 [ run ] triggered by Bot. Commit: beb9bfa Link to invocation

[tensorrt-cicd]

PR_Github #50876 [ run ] completed with state ABORTED. Commit: beb9bfa

Link to invocation

[tensorrt-cicd]

PR_Github #50912 [ run ] completed with state FAILURE. Commit: beb9bfa
/LLM/main/L0_MergeRequest_PR pipeline #40375 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51118 [ run ] triggered by Bot. Commit: a5b68e7 Link to invocation

[tensorrt-cicd]

PR_Github #51118 [ run ] completed with state SUCCESS. Commit: a5b68e7
/LLM/main/L0_MergeRequest_PR pipeline #40556 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51403 [ run ] triggered by Bot. Commit: ac45238 Link to invocation

[tensorrt-cicd]

PR_Github #51403 [ run ] completed with state SUCCESS. Commit: ac45238
/LLM/main/L0_MergeRequest_PR pipeline #40813 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #51655 [ run ] triggered by Bot. Commit: ac45238 Link to invocation

[tensorrt-cicd]

PR_Github #51655 [ run ] completed with state FAILURE. Commit: ac45238
/LLM/main/L0_MergeRequest_PR pipeline #41039 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --stage-list "Build-Docker-Images"

[tensorrt-cicd]

PR_Github #51664 [ run ] triggered by Bot. Commit: 5ea8fb5 Link to invocation

[tensorrt-cicd]

PR_Github #51664 [ run ] completed with state FAILURE. Commit: 5ea8fb5
/LLM/main/L0_MergeRequest_PR pipeline #41047 (Partly Tested) completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[tensorrt-cicd]

PR_Github #53622 [ run ] completed with state SUCCESS. Commit: af8c860
/LLM/main/L0_MergeRequest_PR pipeline #42770 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53671 [ run ] triggered by Bot. Commit: af8c860 Link to invocation

[tensorrt-cicd]

PR_Github #53671 [ run ] completed with state SUCCESS. Commit: af8c860
/LLM/main/L0_MergeRequest_PR pipeline #42814 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #53708 [ run ] triggered by Bot. Commit: af8c860 Link to invocation

[tensorrt-cicd]

PR_Github #53708 [ run ] completed with state SUCCESS. Commit: af8c860
/LLM/main/L0_MergeRequest_PR pipeline #42839 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #54360 [ run ] triggered by Bot. Commit: 0dee979 Link to invocation

[tensorrt-cicd]

PR_Github #54360 [ run ] completed with state FAILURE. Commit: 0dee979
/LLM/main/L0_MergeRequest_PR pipeline #43430 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #54629 [ run ] triggered by Bot. Commit: 0dee979 Link to invocation

[tensorrt-cicd]

PR_Github #54629 [ run ] completed with state FAILURE. Commit: 0dee979
/LLM/main/L0_MergeRequest_PR pipeline #43676 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #55044 [ run ] triggered by Bot. Commit: 5389d75 Link to invocation

[tensorrt-cicd]

PR_Github #55044 [ run ] completed with state SUCCESS. Commit: 5389d75
/LLM/main/L0_MergeRequest_PR pipeline #44035 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #55080 [ run ] triggered by Bot. Commit: 5389d75 Link to invocation

[tensorrt-cicd]

PR_Github #55080 [ run ] completed with state FAILURE. Commit: 5389d75
/LLM/main/L0_MergeRequest_PR pipeline #44067 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

Link to invocation

[o-stoner]

/bot run --disable-fail-fast

[tensorrt-cicd]

PR_Github #55330 [ run ] triggered by Bot. Commit: 5389d75 Link to invocation

[tensorrt-cicd]

PR_Github #55330 [ run ] completed with state FAILURE. Commit: 5389d75
/LLM/main/L0_MergeRequest_PR pipeline #44281 completed with status: 'FAILURE'

CI Report

⚠️ Action Required:

• Please check the failed tests and fix your PR
• If you cannot view the failures, ask the CI triggerer to share details
• Once fixed, request an NVIDIA team member to trigger CI again

CI Agent Failure Analysis

Link to invocation

[o-stoner]

/bot run

[tensorrt-cicd]

PR_Github #55555 [ run ] triggered by Bot. Commit: 36132aa Link to invocation

[tensorrt-cicd]

PR_Github #55555 [ run ] completed with state SUCCESS. Commit: 36132aa
/LLM/main/L0_MergeRequest_PR pipeline #44477 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

CI Report

Link to invocation