Enable Helix Job Monitor for the runtime pipeline - #129690
Open
premun wants to merge 31 commits into
Open
Conversation
Contributor
|
Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR wires up Helix Job Monitor support across the runtime Azure DevOps pipelines by introducing a useHelixMonitor parameter through the relevant YAML templates, enabling it for the runtime pipeline test legs, and adding the Helix Job Monitor .NET tool/dependency so the monitor job can run.
Changes:
- Add a
HelixJobMonitorjob toeng/pipelines/runtime.yml(Build stage) using the sharedeng/commontemplate. - Plumb a
useHelixMonitorparameter through Helix-related pipeline templates and pass/p:EnableHelixJobMonitor=...through the various “send to helix” paths. - Add
Microsoft.DotNet.Helix.JobMonitortool/dependency and pin it in.config/dotnet-tools.json.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| eng/Version.Details.xml | Adds the Microsoft.DotNet.Helix.JobMonitor toolset dependency entry. |
| eng/Version.Details.props | Adds version properties for the Helix Job Monitor dependency. |
| eng/pipelines/runtime.yml | Adds the Helix Job Monitor job and enables useHelixMonitor: true across runtime pipeline test legs. |
| eng/pipelines/libraries/run-test-job.yml | Introduces useHelixMonitor parameter and forwards it into the libraries Helix submission step. |
| eng/pipelines/libraries/helix.yml | Accepts useHelixMonitor and passes /p:EnableHelixJobMonitor=... into sendtohelix.proj. |
| eng/pipelines/installer/helix.yml | Adds useHelixMonitor, but currently passes UseHelixMonitor into /eng/common/templates/steps/send-to-helix.yml (template parameter mismatch; needs fix). |
| eng/pipelines/coreclr/nativeaot-post-build-steps.yml | Adds useHelixMonitor pass-through to the shared runtime test+Helix submit template. |
| eng/pipelines/common/templates/wasm-runtime-tests.yml | Adds and forwards useHelixMonitor into the Helix submission path. |
| eng/pipelines/common/templates/wasm-library-tests.yml | Adds and forwards useHelixMonitor into the Helix submission path. |
| eng/pipelines/common/templates/wasm-library-aot-tests.yml | Adds and forwards useHelixMonitor into nested wasm templates. |
| eng/pipelines/common/templates/wasm-coreclr-library-tests.yml | Adds and forwards useHelixMonitor into the Helix submission path. |
| eng/pipelines/common/templates/simple-wasm-build-tests.yml | Adds and forwards useHelixMonitor into the Helix submission path for WBT scenarios. |
| eng/pipelines/common/templates/runtimes/send-to-helix-step.yml | Adds useHelixMonitor and includes /p:EnableHelixJobMonitor=... in the computed send parameters. |
| eng/pipelines/common/templates/runtimes/run-test-job.yml | Adds useHelixMonitor parameter and forwards it to send-to-helix-step.yml. |
| eng/pipelines/common/templates/runtimes/build-runtime-tests-and-send-to-helix.yml | Adds useHelixMonitor parameter and forwards it to send-to-helix-step.yml. |
| eng/pipelines/common/templates/browser-wasm-coreclr-build-tests.yml | Hard-enables useHelixMonitor: true for the CoreCLR Wasm.Build.Tests Helix submission. |
| eng/pipelines/common/templates/browser-wasm-build-tests.yml | Adds and forwards useHelixMonitor for non-CoreCLR WBT Helix submissions. |
| .config/dotnet-tools.json | Adds the microsoft.dotnet.helix.jobmonitor tool entry and pins its version. |
- Add useHelixMonitor parameter to all helix pipeline templates - Enable Helix Monitor in runtime.yml for all test legs - Add helix-job-monitor.yml template job at the top of the Build stage - Add Microsoft.DotNet.Helix.JobMonitor dependency (same version as Arcade) - Pass EnableHelixJobMonitor MSBuild property through send-to-helix Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
premun
force-pushed
the
prvysoky/helix-job-monitor-rebased
branch
from
June 22, 2026 13:35
20899d1 to
624f97d
Compare
This was referenced Jun 22, 2026
Open
3 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 18 out of 19 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
eng/pipelines/common/templates/browser-wasm-coreclr-build-tests.yml:145
- This template hard-codes
useHelixMonitor: truewhen invoking the libraries Helix template. Since other Helix templates were updated to accept auseHelixMonitorparameter (defaulting to false) andruntime.ymlis now explicitly setting it, this hard-codedtruemakes the behavior inconsistent and harder to roll back/experiment with.
Consider adding a useHelixMonitor parameter to this template (default false) and threading it through instead of hard-coding true.
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)_$(_hostedOs)
extraHelixArguments: /p:BrowserHost=$(_hostedOs)
scenarios:
- buildwasmapps
useHelixMonitor: true
Comment on lines
+70
to
+74
| - template: /eng/common/core-templates/job/helix-job-monitor.yml | ||
| parameters: | ||
| helixAccessToken: $(HelixApiAccessToken) | ||
| timeoutInMinutes: 540 | ||
|
|
Comment on lines
706
to
710
| testBuildArgs: nativeaot tree nativeaot | ||
| liveLibrariesBuildConfig: Release | ||
| testRunNamePrefixSuffix: NativeAOT_$(_BuildConfig) | ||
| useHelixMonitor: true | ||
| extraVariablesTemplates: |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
eng/pipelines/common/templates/browser-wasm-coreclr-build-tests.yml:145
- This template hardcodes
useHelixMonitor: true, while the other Wasm test templates were updated to accept auseHelixMonitorparameter. Hardcoding makes it harder to centrally enable/disable Helix monitor behavior (e.g., via enableHelixJobMonitor) and creates an inconsistency across Wasm legs.
creator: dotnet-bot
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)_$(_hostedOs)
extraHelixArguments: /p:BrowserHost=$(_hostedOs)
scenarios:
- buildwasmapps
useHelixMonitor: true
This version has updated logging
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (1)
eng/pipelines/common/templates/browser-wasm-coreclr-build-tests.yml:145
- This template hard-codes
useHelixMonitor: true, which bypasses the pipeline-level enableHelixJobMonitor toggle used elsewhere in runtime.yml. This makes it harder to roll back/experiment consistently across all Helix submissions.
testRunNamePrefixSuffix: CoreCLR_$(_BuildConfig)_$(_hostedOs)
extraHelixArguments: /p:BrowserHost=$(_hostedOs)
scenarios:
- buildwasmapps
useHelixMonitor: true
Contributor
|
Workflow state for the Holistic Review Orchestrator. {
"version": 5,
"last_dispatched_commit": "c0584e7112beb2068ac9c4ffb90a1f398672c844",
"last_dispatched_base_ref": "main",
"last_dispatched_base_sha": "a0ad638fb14e9f3ad2c11588f6fb3a1610c8be08",
"last_reviewed_commit": "c0584e7112beb2068ac9c4ffb90a1f398672c844",
"last_reviewed_base_ref": "main",
"last_reviewed_base_sha": "a0ad638fb14e9f3ad2c11588f6fb3a1610c8be08",
"last_recorded_worker_run_id": "29684912062",
"review_attempt_commit": "",
"review_attempt_base_ref": "",
"review_attempt_count": 0,
"max_review_attempts": 5,
"review_history_format": "holistic-review-disclosure-v1",
"review_history": [
{
"commit": "c0584e7112beb2068ac9c4ffb90a1f398672c844",
"review_id": 4730671433
}
]
} |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Holistic Review
Motivation: Enable the Helix Job Monitor (dotnet/arcade#16731) for the dotnet/runtime pipeline so that stuck or lost Helix jobs are detected and surfaced, improving CI reliability. This requires wiring a new Microsoft.DotNet.Helix.JobMonitor dependency plus a monitor job and a useHelixMonitor/EnableHelixJobMonitor plumbing parameter through the many Helix test templates.
Approach: The PR adds the Microsoft.DotNet.Helix.JobMonitor package/tool dependency, introduces a helix-job-monitor.yml template job at the top of the Build stage in runtime.yml, defines an enableHelixJobMonitor variable, and threads a useHelixMonitor parameter (defaulting to false) through every Helix send template so it becomes the EnableHelixJobMonitor / UseHelixMonitor MSBuild/Helix SDK property. The plumbing is mechanical and consistent across templates, and the parameter defaults are safe (opt-in per leg). The libraries/helix.yml change is largely a whitespace/indentation reflow with the single functional addition of the new property.
Summary: The pipeline plumbing is sound and follows existing conventions. However, the dependency metadata has two problems that should block merge until resolved: (1) the new Microsoft.DotNet.Helix.JobMonitor version is declared inconsistently in three files, and (2) the pre-existing Microsoft.DotNet.GenAPI dependency appears to have been accidentally modified (version downgrade + Sha repointed to the arcade commit), which looks like a bad rebase artifact. Both are flagged inline. Recommend regenerating the dependency files via darc to guarantee coherence. Verdict: Needs work before merge (dependency-metadata issues); the pipeline YAML changes themselves look good.
Detailed Findings
- Inconsistent JobMonitor version across three files (inline on
eng/Version.Details.xml):26363.117vs26330.2(Version.Details.props) vs26366.1(dotnet-tools.json). These should be a single version, normally kept in lockstep bydarc. - Likely-accidental
Microsoft.DotNet.GenAPIchange (inline oneng/Version.Details.xml): downgraded and repointed to the arcade Sha, inconsistent with its sibling dotnet/dotnet dependencies still at26365.101. Appears out of scope; please revert. - Non-blocking:
eng/pipelines/common/variables.ymlhardcodesenableHelixJobMonitor: truewhile every template parameter defaultsuseHelixMonitor: false. This is a reasonable global on-switch, but note there is now no single knob to disable the monitor for legs that reference the variable directly — acceptable if intentional.
Note
This review was generated by this repository's Holistic Review agentic workflow to complement the built-in Copilot review.
Generated by Holistic Review · 111.3 AIC · ⌖ 15.1 AIC · ⊞ 10K
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: d85d00dc-22a3-49cd-9534-3552ff2afabc
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (4)
eng/Version.Details.xml:73
- The Microsoft.DotNet.GenAPI dependency now references a SHA (b25932e8...) that doesn’t match its declared repository (Uri is dotnet/dotnet). This looks like a copy/paste from the new Arcade dependency and will break dependency flow because the SHA must exist in the referenced repo. Restore GenAPI’s version/SHA to the dotnet/dotnet source SHA used by the rest of the toolset deps (or otherwise ensure the SHA corresponds to a real commit in dotnet/dotnet).
<Dependency Name="Microsoft.DotNet.GenAPI" Version="11.0.0-beta.26363.117">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>b25932e8d128b5242f1d89691f8e8171b82774c0</Sha>
</Dependency>
eng/Version.Details.props:10
- There are three different versions being introduced for Helix Job Monitor: Version.Details.xml uses 11.0.0-beta.26363.117, Version.Details.props sets 11.0.0-beta.26330.2, and .config/dotnet-tools.json pins 11.0.0-beta.26373.5. Since the helix-job-monitor template restores the tool from .config/dotnet-tools.json by default, this version skew is likely unintentional and makes updates/repro harder. Recommend aligning these to a single version (ideally the same Arcade build used by the eng/common template) and/or removing the unused Version.Details.props entry if it’s not required.
<!-- dotnet-arcade dependencies -->
<MicrosoftDotNetHelixJobMonitorPackageVersion>11.0.0-beta.26330.2</MicrosoftDotNetHelixJobMonitorPackageVersion>
<!-- dotnet-dotnet dependencies -->
eng/pipelines/runtime.yml:74
- The HelixJobMonitor job is always injected into the Build stage, but the actual Helix submissions are toggled via useHelixMonitor (enableHelixJobMonitor). If enableHelixJobMonitor is ever turned off (or overridden per-pipeline), the monitor job will still run and may either no-op or fail the stage unexpectedly. Suggest gating the monitor job on the same enableHelixJobMonitor switch for consistent rollout/rollback behavior.
- template: /eng/common/core-templates/job/helix-job-monitor.yml
parameters:
helixAccessToken: $(HelixApiAccessToken)
timeoutInMinutes: 540
eng/pipelines/common/templates/browser-wasm-coreclr-build-tests.yml:149
- This template hardcodes useHelixMonitor: true when invoking /eng/pipelines/libraries/helix.yml, which makes it impossible to disable Helix Job Monitor for this leg (and inconsistent with the other wasm templates that pass useHelixMonitor through a parameter). Consider adding a useHelixMonitor parameter to this template (default false) and wiring it through instead of hardcoding true.
# concurrently instead of one-send-then-wait-for-completion serially (see issue #118938).
extraHelixArguments: /p:BrowserHost=$(_hostedOs) /maxcpucount:8
scenarios:
- buildwasmapps
useHelixMonitor: true
Member
|
Green now? Hrmmmmmm |
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (6)
eng/Version.Details.xml:73
- The Microsoft.DotNet.GenAPI dependency now has https://github.com/dotnet/dotnet but the was changed to b25932e8… (an Arcade commit referenced by the new Helix.JobMonitor dependency). Uri/Sha should refer to the same source repo/commit; otherwise dependency flow and provenance tracking become incorrect.
<Dependency Name="Microsoft.DotNet.Helix.JobMonitor" Version="11.0.0-beta.26363.117">
<Uri>https://github.com/dotnet/arcade</Uri>
<Sha>b25932e8d128b5242f1d89691f8e8171b82774c0</Sha>
</Dependency>
<Dependency Name="Microsoft.DotNet.GenAPI" Version="11.0.0-beta.26363.117">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>b25932e8d128b5242f1d89691f8e8171b82774c0</Sha>
</Dependency>
eng/Version.Details.props:10
- MicrosoftDotNetHelixJobMonitorPackageVersion (11.0.0-beta.26330.2) is out of sync with eng/Version.Details.xml, which pins Microsoft.DotNet.Helix.JobMonitor to 11.0.0-beta.26363.117. These files should agree, otherwise version-consuming props can drift from the tracked dependency.
<!-- dotnet-arcade dependencies -->
<MicrosoftDotNetHelixJobMonitorPackageVersion>11.0.0-beta.26330.2</MicrosoftDotNetHelixJobMonitorPackageVersion>
<!-- dotnet-dotnet dependencies -->
.config/dotnet-tools.json:34
- The helix job monitor tool version is pinned here to 11.0.0-beta.26373.5, but eng/Version.Details.xml adds Microsoft.DotNet.Helix.JobMonitor at 11.0.0-beta.26363.117 (and Version.Details.props currently has 26330.2). Since helix-job-monitor.yml restores from dotnet-tools.json by default, these version pins should be aligned to avoid surprising tool/version drift.
"microsoft.dotnet.helix.jobmonitor": {
"version": "11.0.0-beta.26373.5",
"commands": [
"dotnet-helix-job-monitor"
]
}
eng/pipelines/runtime.yml:74
- helix-job-monitor.yml receives helixAccessToken unconditionally. Most other Helix submission paths only provide HelixApiAccessToken for internal builds, and leaving this set for public builds risks passing a literal '$(HelixApiAccessToken)' / empty token depending on variable resolution. Consider only setting helixAccessToken when System.TeamProject == 'internal'.
- template: /eng/common/core-templates/job/helix-job-monitor.yml
parameters:
helixAccessToken: $(HelixApiAccessToken)
timeoutInMinutes: 540
eng/pipelines/runtime.yml:1010
- The browser_wasm_mac wasm-library-tests leg does not pass useHelixMonitor, so it will keep the default (false) even though other wasm legs now use ${{ variables.enableHelixJobMonitor }}. If the intent is to enable monitoring for all wasm test legs, this one is currently missed.
- template: /eng/pipelines/common/templates/wasm-library-tests.yml
parameters:
platforms:
- browser_wasm_win
alwaysRun: ${{ variables.isRollingBuild }}
extraBuildArgs: /p:AotHostArchitecture=x64 /p:AotHostOS=$(_hostedOS)
useHelixMonitor: ${{ variables.enableHelixJobMonitor }}
scenarios:
eng/pipelines/common/templates/browser-wasm-coreclr-build-tests.yml:149
- This template hard-codes
useHelixMonitor: truewhen invoking /eng/pipelines/libraries/helix.yml. That makes it impossible for callers (e.g., runtime.yml) to toggle Helix monitoring consistently via a parameter/variable, unlike the other wasm templates updated in this PR.
extraHelixArguments: /p:BrowserHost=$(_hostedOs) /maxcpucount:8
scenarios:
- buildwasmapps
useHelixMonitor: true
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
eng/Version.Details.xml:73
- The Microsoft.DotNet.GenAPI dependency was changed to a different version and to a SHA that matches the new Helix.JobMonitor dependency, but the Uri is still dotnet/dotnet. This makes the dependency metadata internally inconsistent (the SHA no longer corresponds to the declared source repo) and looks like an accidental copy/paste unrelated to enabling the Helix job monitor.
<Dependency Name="Microsoft.DotNet.GenAPI" Version="11.0.0-beta.26363.117">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>b25932e8d128b5242f1d89691f8e8171b82774c0</Sha>
</Dependency>
eng/Version.Details.props:10
- MicrosoftDotNetHelixJobMonitorPackageVersion is pinned to 11.0.0-beta.26330.2, but eng/Version.Details.xml adds Microsoft.DotNet.Helix.JobMonitor at 11.0.0-beta.26363.117. Version.Details.props is expected to reflect Version.Details.xml; this mismatch suggests the dependency flow outputs weren’t regenerated (and may cause the repo to consume a different tool version than intended).
<!-- dotnet-arcade dependencies -->
<MicrosoftDotNetHelixJobMonitorPackageVersion>11.0.0-beta.26330.2</MicrosoftDotNetHelixJobMonitorPackageVersion>
<!-- dotnet-dotnet dependencies -->
.config/dotnet-tools.json:33
- The Helix job monitor tool is pinned to 11.0.0-beta.26373.5 here, which doesn’t match the version introduced in eng/Version.Details.xml (11.0.0-beta.26363.117) or eng/Version.Details.props. If the intent is to keep the tool version aligned with the tracked dependency, these should be consistent to avoid restoring/running a different tool than the repo thinks it depends on.
"microsoft.dotnet.helix.jobmonitor": {
"version": "11.0.0-beta.26373.5",
"commands": [
"dotnet-helix-job-monitor"
]
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 20 out of 20 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
eng/Version.Details.xml:73
- The GenAPI dependency entry now has a different version/Sha than the other dotnet/dotnet toolset dependencies, and it no longer matches the GenAPI version pinned in eng/Version.Details.props. Unless this PR intentionally updates GenAPI, this looks like an accidental change (it also pairs a dotnet/dotnet Uri with a Sha that appears to be reused from the new Arcade dependency). Consider reverting GenAPI back to the previous Version/Sha (or, if GenAPI is intentionally coming from a different repo, update all related version pins consistently).
<Dependency Name="Microsoft.DotNet.GenAPI" Version="11.0.0-beta.26363.117">
<Uri>https://github.com/dotnet/dotnet</Uri>
<Sha>b25932e8d128b5242f1d89691f8e8171b82774c0</Sha>
</Dependency>
eng/Version.Details.props:10
- eng/Version.Details.xml adds Microsoft.DotNet.Helix.JobMonitor at 11.0.0-beta.26363.117, but eng/Version.Details.props pins MicrosoftDotNetHelixJobMonitorPackageVersion to 11.0.0-beta.26330.2. These should match (Version.Details.props is used for the actual package version during builds).
<!-- dotnet-arcade dependencies -->
<MicrosoftDotNetHelixJobMonitorPackageVersion>11.0.0-beta.26330.2</MicrosoftDotNetHelixJobMonitorPackageVersion>
<!-- dotnet-dotnet dependencies -->
.config/dotnet-tools.json:34
- The Helix job monitor template restores the tool from .config/dotnet-tools.json. To avoid running a different tool build than the one pinned via Version.Details.*, the tool manifest version should be kept in sync with Microsoft.DotNet.Helix.JobMonitor (added in this PR).
"microsoft.dotnet.helix.jobmonitor": {
"version": "11.0.0-beta.26373.5",
"commands": [
"dotnet-helix-job-monitor"
]
}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dotnet/arcade#16731