From 9f4038b63b414f930c83a6eb324b88b6ca921876 Mon Sep 17 00:00:00 2001 From: Derek Pitman Date: Fri, 17 Jul 2026 18:32:14 -0400 Subject: [PATCH 1/3] [TRTLLMINF-101][fix] Surface SLURM device faults to the failure classifier Device / driver / interconnect faults (CUDA, NVLink, ECC, driver/NVML, GPU off the bus) print into the SLURM job output log but never reach the stage exception chain: the job tracker squashes a failed job to `exit 1`, so FailureClassifier.classify() sees only a generic failure and cannot steer the retry off the bad node (confirmed in OpenSearch stage data -- these faults land with s_infra_failure_patterns empty). On a terminal FAILED state, scrape job-output.log for a device-fault signature and, on a hit, fold the matched line into a fresh exception so the retry loop's existing classify(SLURM) + rememberAvoidedSlurmNodeLists path recognizes it and relaunches on a different node. The scrape is only a gate: the catalog remains authoritative, so a line it does not recognize falls through to the normal rethrow (no retry). App-induced CUDA errors (illegal memory access, launch failure, OOM) are excluded -- the stage data shows those are code regressions, not node faults. Signed-off-by: Derek Pitman --- jenkins/L0_Test.groovy | 53 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 53 insertions(+) diff --git a/jenkins/L0_Test.groovy b/jenkins/L0_Test.groovy index cc7281206ece..79b9d5f9181c 100644 --- a/jenkins/L0_Test.groovy +++ b/jenkins/L0_Test.groovy @@ -224,6 +224,42 @@ def echoRemoteLogTail(def pipeline, Map remote, String remotePath, int lines = 2 } } +// Scrape the SLURM job output log for a device / driver / interconnect fault +// signature and return the last matching line (truncated), or "" for no match. +// +// Device faults (CUDA/NVLink/ECC/driver) print into job-output.log but never +// reach the stage exception chain -- the tracker squashes a failed job to +// `exit 1` -- so classify() otherwise sees only a generic failure and cannot +// steer the retry off the bad node. This is a GATE only: the returned line is +// folded into a fresh exception so FailureClassifier.PATTERN_CATALOG (the +// authoritative list) makes the real retry/severity decision. The regex mirrors +// the machine-fault catalog rows; a line the catalog does not recognize simply +// falls through to a normal rethrow. App-induced CUDA errors (illegal memory +// access, unspecified launch failure, OOM) are deliberately excluded -- the +// OpenSearch stage data shows those are overwhelmingly code regressions, not +// node faults, and must not trigger a node-avoiding retry. +def scrapeSlurmLogForDeviceFault(def pipeline, Map remote, String remoteLogPath) { + def deviceFaultRegex = "cudaErrorMapBufferObjectFailed|mapping of buffer object failed|" + + "uncorrectable NVLink error|cudaErrorNvlinkUncorrectable|CUDA_ERROR_SYSTEM_NOT_READY|" + + "uncorrectable ECC error|CUDA_ERROR_ECC_UNCORRECTABLE|has fallen off the bus|GPU is lost|" + + "Unable to determine the device handle for GPU|RmInitAdapter failed|Failed to initialize NVML|" + + "communicate with the NVIDIA driver|CUDA_ERROR_DEVICE_UNAVAILABLE|" + + "no CUDA-capable device is detected|CUDA_ERROR_UNKNOWN|CUDA unknown error|" + + "CUDA-capable device.s. is/are busy or unavailable" + try { + return Utils.exec( + pipeline, + script: Utils.sshUserCmd(remote, + "\"if [ -f '${remoteLogPath}' ]; then grep -aiE '${deviceFaultRegex}' '${remoteLogPath}' 2>/dev/null | tail -n 1 | cut -c1-500; fi\""), + returnStdout: true, + numRetries: 1, + )?.trim() + } catch (Exception scrapeEx) { + pipeline.echo("Ignorable warning: could not scrape ${remoteLogPath} for device faults on ${remote.host}: ${scrapeEx.message}") + return "" + } +} + // `postTag` uniquifies the uploaded tar filename, the Artifactory guard key and // the locally-staged result XMLs when the same stageName is uploaded more than // once in a build (e.g. SLURM infra-failure retries). First attempt passes "". @@ -1871,6 +1907,23 @@ def runLLMTestlistWithSbatch(pipeline, platform, testList, config=VANILLA_CONFIG "Original failure: ${e.message}", e) } + // A terminal FAILED state may be a node/device fault whose + // signature (CUDA/NVLink/ECC/driver) printed only into the SLURM + // job output log, never into this exception chain. Scrape the log + // and, on a hit, surface the matched line into a fresh exception + // so the authoritative catalog (FailureClassifier.classify at the + // runLLMTestlistWithSbatch caller) can match it and steer the retry + // off the bad node. A miss falls through to the plain rethrow. + if (slurmState == "FAILED") { + def deviceHit = scrapeSlurmLogForDeviceFault(pipeline, remote, slurmJobLogPath) + if (deviceHit) { + echo "[INFRA-RETRY] ${stageName}: device-fault signature in SLURM job ${slurmJobId} log; " + + "surfacing to classifier: ${deviceHit}" + throw new Exception( + "Device/interconnect fault on SLURM node during job ${slurmJobId} for ${stageName}: " + + "${deviceHit} | original: ${e.message}") + } + } echo "[INFRA-RETRY] ${stageName}: SLURM job ${slurmJobId} terminal state=${slurmState ?: 'unknown'}; " + "deferring to failure classifier." throw e From df610946afd26867eddd27e9a0bc9f22338c9b6b Mon Sep 17 00:00:00 2001 From: Derek Pitman Date: Fri, 17 Jul 2026 18:49:22 -0400 Subject: [PATCH 2/3] [TRTLLMINF-101][fix] Address CodeRabbit notes on SLURM device-fault scrape - Use grep -o so the scrape returns the matched signature itself, not the whole line: a long log line can no longer push the signature past the truncation boundary and hide it from classify(). Two alternatives are made catalog-exact so grep -o output still contains the catalog substring (CUDA_ERROR_UNKNOWN: 999, and the full "couldn't communicate with the NVIDIA driver" via wildcards). - Rethrow InterruptedException before the generic catch so a pipeline abort during the SSH scrape is not downgraded to an ignorable warning. Signed-off-by: Derek Pitman --- jenkins/L0_Test.groovy | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/jenkins/L0_Test.groovy b/jenkins/L0_Test.groovy index 79b9d5f9181c..22289d4490fe 100644 --- a/jenkins/L0_Test.groovy +++ b/jenkins/L0_Test.groovy @@ -225,35 +225,43 @@ def echoRemoteLogTail(def pipeline, Map remote, String remotePath, int lines = 2 } // Scrape the SLURM job output log for a device / driver / interconnect fault -// signature and return the last matching line (truncated), or "" for no match. +// signature and return the matched signature itself, or "" for no match. // // Device faults (CUDA/NVLink/ECC/driver) print into job-output.log but never // reach the stage exception chain -- the tracker squashes a failed job to // `exit 1` -- so classify() otherwise sees only a generic failure and cannot -// steer the retry off the bad node. This is a GATE only: the returned line is -// folded into a fresh exception so FailureClassifier.PATTERN_CATALOG (the -// authoritative list) makes the real retry/severity decision. The regex mirrors -// the machine-fault catalog rows; a line the catalog does not recognize simply -// falls through to a normal rethrow. App-induced CUDA errors (illegal memory -// access, unspecified launch failure, OOM) are deliberately excluded -- the -// OpenSearch stage data shows those are overwhelmingly code regressions, not -// node faults, and must not trigger a node-avoiding retry. +// steer the retry off the bad node. This is a GATE only: the returned signature +// is folded into a fresh exception so FailureClassifier.PATTERN_CATALOG (the +// authoritative list) makes the real retry/severity decision. A signature the +// catalog does not recognize simply falls through to a normal rethrow. +// App-induced CUDA errors (illegal memory access, unspecified launch failure, +// OOM) are deliberately excluded -- the OpenSearch stage data shows those are +// overwhelmingly code regressions, not node faults, and must not trigger a +// node-avoiding retry. +// +// grep -o returns only the matched signature (not the whole line), so a long +// log line cannot truncate the signature out of the result before it reaches +// classify(). Each alternative must therefore be catalog-exact: it must match +// (via `.` wildcards for shell-hostile chars) the full catalog substring, so +// grep -o emits text that still contains the catalog pattern. def scrapeSlurmLogForDeviceFault(def pipeline, Map remote, String remoteLogPath) { def deviceFaultRegex = "cudaErrorMapBufferObjectFailed|mapping of buffer object failed|" + "uncorrectable NVLink error|cudaErrorNvlinkUncorrectable|CUDA_ERROR_SYSTEM_NOT_READY|" + "uncorrectable ECC error|CUDA_ERROR_ECC_UNCORRECTABLE|has fallen off the bus|GPU is lost|" + "Unable to determine the device handle for GPU|RmInitAdapter failed|Failed to initialize NVML|" + - "communicate with the NVIDIA driver|CUDA_ERROR_DEVICE_UNAVAILABLE|" + - "no CUDA-capable device is detected|CUDA_ERROR_UNKNOWN|CUDA unknown error|" + + "could... communicate with the NVIDIA driver|CUDA_ERROR_DEVICE_UNAVAILABLE|" + + "no CUDA-capable device is detected|CUDA_ERROR_UNKNOWN: 999|CUDA unknown error|" + "CUDA-capable device.s. is/are busy or unavailable" try { return Utils.exec( pipeline, script: Utils.sshUserCmd(remote, - "\"if [ -f '${remoteLogPath}' ]; then grep -aiE '${deviceFaultRegex}' '${remoteLogPath}' 2>/dev/null | tail -n 1 | cut -c1-500; fi\""), + "\"if [ -f '${remoteLogPath}' ]; then grep -aioE '${deviceFaultRegex}' '${remoteLogPath}' 2>/dev/null | tail -n 1 | cut -c1-500; fi\""), returnStdout: true, numRetries: 1, )?.trim() + } catch (InterruptedException e) { + throw e } catch (Exception scrapeEx) { pipeline.echo("Ignorable warning: could not scrape ${remoteLogPath} for device faults on ${remote.host}: ${scrapeEx.message}") return "" From 3de68d6f8372309867738c4cc47e47fb88e7e7fa Mon Sep 17 00:00:00 2001 From: Derek Pitman Date: Tue, 21 Jul 2026 14:56:19 -0400 Subject: [PATCH 3/3] [TRTLLMINF-101][fix] Wrap SLURM device-fault scrape in bash -c for csh login shells The scrape command used bash test/pipe/redirection syntax that the remote login shell runs directly. Cluster login shells are often csh/tcsh, which can't parse `if [ -f ... ]; then ... 2>/dev/null; fi`. Wrap the body in `bash -c '...'` (the same idiom echoRemoteLogTail already uses) so the login shell only invokes bash with the command body. Inner quoting switched to escaped double quotes to avoid colliding with the bash -c single quotes. Signed-off-by: Derek Pitman --- jenkins/L0_Test.groovy | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/jenkins/L0_Test.groovy b/jenkins/L0_Test.groovy index ce55ffaa7bd9..9f4dae2f2f24 100644 --- a/jenkins/L0_Test.groovy +++ b/jenkins/L0_Test.groovy @@ -254,10 +254,13 @@ def scrapeSlurmLogForDeviceFault(def pipeline, Map remote, String remoteLogPath) "no CUDA-capable device is detected|CUDA_ERROR_UNKNOWN: 999|CUDA unknown error|" + "CUDA-capable device.s. is/are busy or unavailable" try { + // Wrap the body in `bash -c` so it is shell-agnostic: cluster login shells + // are often csh/tcsh, which can't parse this bash test/pipe/redirection + // syntax. The login shell only has to run `bash -c ''`. return Utils.exec( pipeline, script: Utils.sshUserCmd(remote, - "\"if [ -f '${remoteLogPath}' ]; then grep -aioE '${deviceFaultRegex}' '${remoteLogPath}' 2>/dev/null | tail -n 1 | cut -c1-500; fi\""), + "\"bash -c 'if [ -f \\\"${remoteLogPath}\\\" ]; then grep -aioE \\\"${deviceFaultRegex}\\\" \\\"${remoteLogPath}\\\" 2>/dev/null | tail -n 1 | cut -c1-500; fi'\""), returnStdout: true, numRetries: 1, )?.trim()