Take machine offline and notify when dpkg is corrupted during prereq install - #5267
Open
DrewScoggins wants to merge 1 commit into
Open
Take machine offline and notify when dpkg is corrupted during prereq install#5267DrewScoggins wants to merge 1 commit into
DrewScoggins wants to merge 1 commit into
Conversation
…install
Some prerequisite-install failures are not transient but indicate a broken
host. The canonical case is the apt/dpkg database getting stuck in an
interrupted state ("dpkg was interrupted, you must manually run
'sudo dpkg --configure -a'"), which currently just fails the run repeatedly.
When this specific state is detected during prerequisite installation:
- The machine is taken out of Helix rotation by creating the `offline`
sentinel file in $HELIX_CONFIG_ROOT.
- A semaphore blob named after the machine is written to the new
`offline-machines` container in pvscmdupload so the perf team is notified.
The semaphore blob is cleared only when the machine next uploads results
successfully, guaranteeing a machine must actually be working to have its
semaphore removed.
The notification blob write is best effort: it uses the already-available
azure libraries, falls back to bootstrapping a throwaway venv with the
dependencies, and if both fail the machine is still taken offline and a
message is printed to the console.
- scripts/machine_health.py: detection, offline sentinel, semaphore
write (with venv fallback) and clear.
- scripts/performance/constants.py: add OFFLINE_MACHINES_CONTAINER.
- scripts/run_performance_job.py: run the check on prereq failure.
- scripts/benchmarks_ci.py: clear the semaphore after a successful upload.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 418ccfe4-f031-42ad-92d8-38130011c784
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a machine-health hook to detect a specific non-transient prereq install failure mode (corrupted/interrupted apt/dpkg state), then automatically removes the affected Helix machine from rotation and emits an external “offline machine” signal via Azure Storage until the machine proves healthy again by successfully uploading results.
Changes:
- Add
scripts/machine_health.pyto detect interrupted dpkg state, create Helixofflinesentinel, and write/clear an “offline semaphore” blob. - Add
OFFLINE_MACHINES_CONTAINERconstant for the new Azure Storage container. - Invoke the health check on prereq-install failure and clear the semaphore after successful results upload.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| scripts/run_performance_job.py | Runs machine health check after prereq install failure (without affecting exit code). |
| scripts/performance/constants.py | Adds OFFLINE_MACHINES_CONTAINER constant for offline-machine semaphores. |
| scripts/machine_health.py | Implements dpkg corruption detection, Helix offline sentinel creation, and Azure blob semaphore write/clear. |
| scripts/benchmarks_ci.py | Clears the offline semaphore after a successful results upload. |
Comments suppressed due to low confidence (1)
scripts/machine_health.py:132
- The fallback sentinel creation path uses
sudo teewithout-n, which can block on an interactive sudo password prompt (up to the 30s timeout) and delay/offline-marking in the exact failure scenario this script is meant to handle. Adding-nmakes the operation non-interactive and fail fast.
try:
subprocess.run(
["sudo", "tee", offline_path],
input=content,
text=True,
stdout=subprocess.DEVNULL,
check=True,
timeout=30,
)
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
330
to
+334
| combined_prequisites = " && ".join(install_prerequisites) | ||
| helix_pre_commands += [ | ||
| 'echo "** Installing prerequistes **"', | ||
| f"{combined_prequisites} || export PERF_PREREQS_INSTALL_FAILED=1", | ||
| 'test "x$PERF_PREREQS_INSTALL_FAILED" = "x1" && echo "** Error: Failed to install prerequites **"' | ||
| 'test "x$PERF_PREREQS_INSTALL_FAILED" = "x1" && echo "** Error: Failed to install prerequites **"', |
Comment on lines
+82
to
+89
| try: | ||
| result = subprocess.run( | ||
| ["sudo", "apt-get", "check"], | ||
| stdout=subprocess.PIPE, | ||
| stderr=subprocess.STDOUT, | ||
| text=True, | ||
| timeout=120, | ||
| ) |
LoopedBard3
approved these changes
Jul 29, 2026
Member
|
We should fix the typos in the names, and to double check, deleting the helix offline file will be a manual thing we do? |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated by Copilot
Summary
Some prerequisite-install failures are not transient but indicate a broken host. The canonical case is the apt/dpkg database getting stuck in an interrupted state:
Today this just fails the run repeatedly on the same bad machine. This PR detects that specific state during prerequisite installation and:
offlinesentinel file in$HELIX_CONFIG_ROOT(Helix stops sending work to a machine with that file).offline-machinescontainer inpvscmdupload.The semaphore blob is cleared only when the machine next uploads results successfully, so a machine has to actually be working to have its semaphore removed.
The notification write is best effort: it first uses the already-available azure libraries, then falls back to bootstrapping a throwaway venv with the dependencies, and if both fail the machine is still taken offline and a message is printed to the console.
Changes
scripts/machine_health.py(new): dpkg-state detection, offline sentinel creation, semaphore write (with venv fallback), and semaphore clear.scripts/performance/constants.py: addOFFLINE_MACHINES_CONTAINER.scripts/run_performance_job.py: run the health check on prerequisite failure.scripts/benchmarks_ci.py: clear the semaphore after a successful results upload.Detection
Detection mirrors exactly what triggers the failure: a read-only
sudo apt-get checkis probed for thedpkg was interrupted/dpkg --configure -asignatures, so only this non-transient condition takes a machine offline (flaky network failures do not).Notes
dotnet-performance-infra(PerfMonitor) consumes theoffline-machinescontainer and alerts the team.offline-machinescontainer; PerfMonitor's identity needs read.