Skip to content

[None][infra] Log infra-retry classify declines instead of silent rethrow #17432

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
brnguyen2 merged 4 commits into from
Aug 10, 2026
+21 −3
Merged

[None][infra] Log infra-retry classify declines instead of silent rethrow #17432

Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
08fe068
[None][infra] Log infra-retry classify declines instead of silent ret…
brnguyen2 Aug 7, 2026
5eef699
Merge branch 'main' into infra/log-classify-decline
brnguyen2 Aug 8, 2026
3b2ff58
test waives: skip A100X-PyTorch-1/test_unittests.py::test_unittests_v…
brnguyen2 Aug 9, 2026
e283397
test waives: revert the part3 llm_pytorch waive (unblock full run)
brnguyen2 Aug 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 21 additions & 3 deletions jenkins/L0_Test.groovy
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2401,7 +2401,13 @@ def runLLMTestlistOnSlurm(pipeline, platform, testList, config=VANILLA_CONFIG, p
// ensures we only match catalog rows tagged SLURM or BOTH.
def c = FailureClassifier.classify(e, InfraFailure.SLURM)
if (c instanceof PipelineInterruption) throw e
if (!(c instanceof InfraFailure)) throw e // UserFailure -> don't retry
if (!(c instanceof InfraFailure)) {
// UserFailure -> don't retry, but leave a trace: a failure whose
// exception matches no catalog pattern lands here and would
// otherwise decline the retry with no log output at all.
echo "[INFRA-RETRY] ${stageName}: SLURM attempt failed with no infra pattern matched (classified as user failure); not retrying. Exception: ${e.toString()}"
throw e
}

rememberAvoidedSlurmNodeLists(avoidedSlurmNodeListsByCluster, attemptPlacementContext.lastSlurmClusterName, attemptPlacementContext.lastSlurmNodeList, stageName)

Expand Down Expand Up @@ -5068,7 +5074,14 @@ def runKubernetesPodWithInfraRetry(Map opts = [:], pipeline, podSpec, containerN
}
def c = FailureClassifier.classify(e, InfraFailure.K8S)
if (c instanceof PipelineInterruption) throw e
if (!(c instanceof InfraFailure)) throw e // UserFailure -> don't retry
if (!(c instanceof InfraFailure)) {
// UserFailure -> don't retry, but leave a trace: a pod/agent
// launch exception missing from PATTERN_CATALOG (e.g. a
// pod-scheduling timeout) lands here and would otherwise
// vanish with no log output at all.
echo "[INFRA-RETRY] ${stageName}: pod/agent launch failed before execution with no infra pattern matched (classified as user failure); not retrying. Exception: ${e.toString()}"
throw e
}

rememberAvoidedKubernetesHostNodes(avoidedKubernetesHostNodes, attemptPlacementContext.lastHostNode, stageName)

Expand Down Expand Up @@ -5153,7 +5166,12 @@ def runKubernetesPodWithInfraRetry(Map opts = [:], pipeline, podSpec, containerN
// exhausted its own budget) from being treated as K8s infra here.
def c = FailureClassifier.classify(e, InfraFailure.K8S)
if (c instanceof PipelineInterruption) throw e
if (!(c instanceof InfraFailure)) throw e // UserFailure -> don't retry
if (!(c instanceof InfraFailure)) {
// UserFailure -> don't retry, but leave a trace so the decline
// is diagnosable from the console (see the launch-loop twin above).
echo "[INFRA-RETRY] ${stageName}: stage failed with no infra pattern matched (classified as user failure); not retrying. Exception: ${e.toString()}"
throw e
}

rememberAvoidedKubernetesHostNodes(avoidedKubernetesHostNodes, attemptPlacementContext.lastHostNode, stageName)

Expand Down
Loading