[TRTLLMINF-102][fix] Surface SLURM device faults to the failure classifier - #16557
Conversation
…ifier Device / driver / interconnect faults (CUDA, NVLink, ECC, driver/NVML, GPU off the bus) print into the SLURM job output log but never reach the stage exception chain: the job tracker squashes a failed job to `exit 1`, so FailureClassifier.classify() sees only a generic failure and cannot steer the retry off the bad node (confirmed in OpenSearch stage data -- these faults land with s_infra_failure_patterns empty). On a terminal FAILED state, scrape job-output.log for a device-fault signature and, on a hit, fold the matched line into a fresh exception so the retry loop's existing classify(SLURM) + rememberAvoidedSlurmNodeLists path recognizes it and relaunches on a different node. The scrape is only a gate: the catalog remains authoritative, so a line it does not recognize falls through to the normal rethrow (no retry). App-induced CUDA errors (illegal memory access, launch failure, OOM) are excluded -- the stage data shows those are code regressions, not node faults. Signed-off-by: Derek Pitman <dpitman@nvidia.com>
📝 WalkthroughWalkthroughAdds SLURM log scanning for CUDA, NVLink, ECC, driver, and device-loss signatures. Failed sbatch jobs now raise an enriched exception containing the matched fault line so failure classification can select device-aware retry behavior. ChangesSLURM device-fault detection
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant Runner as SLURM sbatch runner
participant Scraper as scrapeSlurmLogForDeviceFault
participant Log as SLURM job log
participant Classifier as failure-classification layer
Runner->>Scraper: Scrape failed job log
Scraper->>Log: Search fault signatures
Log-->>Scraper: Return last matching line
Scraper-->>Runner: Return matched or empty line
Runner->>Classifier: Raise enriched or original failure
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/L0_Test.groovy`:
- Around line 250-256: Update the command in the Utils.exec call to preserve the
deviceFaultRegex match when limiting output to 500 characters. Replace the
current start-of-line cut with logic that returns the matched expression itself,
or truncates the surrounding text while retaining that match, so the resulting
deviceHit remains recognized by FailureClassifier.
- Around line 257-260: Update the SSH scrape exception handling around the catch
block to catch and immediately rethrow InterruptedException before the generic
Exception handler. Keep the existing ignorable-warning and empty-string behavior
for other scrape failures, preserving pipeline cancellation semantics.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 26cba290-7718-4b89-8f95-d0f4898df687
📒 Files selected for processing (1)
jenkins/L0_Test.groovy
…crape - Use grep -o so the scrape returns the matched signature itself, not the whole line: a long log line can no longer push the signature past the truncation boundary and hide it from classify(). Two alternatives are made catalog-exact so grep -o output still contains the catalog substring (CUDA_ERROR_UNKNOWN: 999, and the full "couldn't communicate with the NVIDIA driver" via wildcards). - Rethrow InterruptedException before the generic catch so a pipeline abort during the SSH scrape is not downgraded to an ignorable warning. Signed-off-by: Derek Pitman <dpitman@nvidia.com>
|
/bot run --disable-fail-fast |
|
PR_Github #60056 [ run ] triggered by Bot. Commit: |
|
PR_Github #60056 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60383 [ run ] triggered by Bot. Commit: |
|
PR_Github #60383 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60394 [ run ] triggered by Bot. Commit: |
|
PR_Github #60394 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60423 [ run ] triggered by Bot. Commit: |
|
PR_Github #60423 [ run ] completed with state |
…h login shells The scrape command used bash test/pipe/redirection syntax that the remote login shell runs directly. Cluster login shells are often csh/tcsh, which can't parse `if [ -f ... ]; then ... 2>/dev/null; fi`. Wrap the body in `bash -c '...'` (the same idiom echoRemoteLogTail already uses) so the login shell only invokes bash with the command body. Inner quoting switched to escaped double quotes to avoid colliding with the bash -c single quotes. Signed-off-by: Derek Pitman <dpitman@nvidia.com>
|
/bot run |
|
PR_Github #60767 [ run ] triggered by Bot. Commit: |
|
PR_Github #60767 [ run ] completed with state
|
|
/bot run |
|
PR_Github #60990 [ run ] triggered by Bot. Commit: |
|
PR_Github #60990 [ run ] completed with state
|
|
/bot run |
|
PR_Github #61017 [ run ] triggered by Bot. Commit: |
|
PR_Github #61017 [ run ] completed with state
|
|
/bot run |
|
PR_Github #61049 [ run ] triggered by Bot. Commit: |
|
PR_Github #61049 [ run ] completed with state |
…retry-on-terminal-state Resolve jenkins/L0_Test.groovy: fold the device-fault log scrape (NVIDIA#16557) into this PR's verdict-file-based SLURM result handling. The non-terminal-state (job-still-running) infra classification runs first, then on a terminal FAILED verdict the log is scraped for a device/interconnect signature and surfaced to the classifier; a miss falls through to the "Pytest failed" rethrow. Dropped the old catch-block's `e`/`e.message` reference, which no longer exists in the verdict-file structure. Signed-off-by: Derek Pitman <dpitman@nvidia.com>
…ifier (NVIDIA#16557) Signed-off-by: Derek Pitman <dpitman@nvidia.com>
3 participants
Summary by CodeRabbit
Description
Device / driver / interconnect faults (CUDA, NVLink, ECC, driver/NVML, GPU off the bus) print into the SLURM job output log but never reach the stage exception chain: the job tracker squashes a failed job to
exit 1, so FailureClassifier.classify() sees only a generic failure and cannot steer the retry off the bad node (confirmed in OpenSearch stage data -- these faults land with s_infra_failure_patterns empty).On a terminal FAILED state, scrape job-output.log for a device-fault signature and, on a hit, fold the matched line into a fresh exception so the retry loop's existing classify(SLURM) + rememberAvoidedSlurmNodeLists path recognizes it and relaunches on a different node. The scrape is only a gate: the catalog remains authoritative, so a line it does not recognize falls through to the normal rethrow (no retry). App-induced CUDA errors (illegal memory access, launch failure, OOM) are excluded.
Test Coverage
N/A, this is a CI change
PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.