fix: use atomic counter for tempdir uniqueness in bench_tail tests - #22
Closed
rominf wants to merge 1 commit into
Closed
fix: use atomic counter for tempdir uniqueness in bench_tail tests#22rominf wants to merge 1 commit into
rominf wants to merge 1 commit into
Conversation
Windows SystemTime has ~15ms resolution, so nanosecond timestamps can collide when parallel tests run within the same timer tick. An atomic counter guarantees a unique directory name per call regardless of clock resolution.
rominf
force-pushed
the
fix-bench-tail-flaky-test
branch
from
June 17, 2026 13:56
edfaff3 to
d9244f0
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes a Windows-specific test flake in rocm-dash-collectors by ensuring each test invocation uses a uniquely named temporary directory even when multiple tests run within the same coarse SystemTime tick.
Changes:
- Replaced the
SystemTime::now().as_nanos()-based temp directory suffix with a process-localAtomicU64counter. - Ensured parallel unit tests in
bench_tailcannot collide on the same temp directory /results.csvpath on Windows.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Collaborator
Author
|
Closing as already-fixed. When rebasing onto current main, the only remaining diff is the tempdir prefix string — the atomic-counter fix that is the substance of this PR already landed on main via #15 ( |
fredespi
added a commit
that referenced
this pull request
Jul 12, 2026
…I300X first)
MI300X ran `install sdk` 9× (once per isolated scenario), each a multi-GB TheRock
cold install, pushing the job past its time cap (~37min, cancelled → no report).
Raising the cap failed twice; the fix is to install the runtime ONCE and share it.
- harness: `use_shared_runtimes()` opts a scenario into a shared runtimes tree by
symlinking its `data/runtimes` at `E2E_SHARED_RUNTIMES_DIR` (only `runtimes/`;
config/services/engines stay isolated). No-op when the env var is unset (local
+ mock runs stay fully isolated — verified mock still 7 pass / 2 xfail). Called
from the "a managed runtime is active" precondition only; the clean-slate
scenarios ("no CLI-managed runtimes", "Installing the SDK") deliberately do NOT
opt in. Active-runtime resolution falls back to most-recently-installed, so a
shared registry is active without touching isolated config.
- ci (e2e-gpu / MI300X only for now): pre-warm ONE runtime SERIALLY in the run
step before the suite (avoids two concurrent scenarios racing installs into the
shared dir), then point E2E_SHARED_RUNTIMES_DIR at it. Validating on MI300X
before propagating to the two Strix jobs.
Relates to task #22; unblocks #23 (same cold-install root cause).
Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
fredespi
added a commit
that referenced
this pull request
Jul 12, 2026
The first cut of the shared-runtime pre-warm (1817c5b) failed validation on MI300X (run 29205441532): the pre-warm used `cargo run --release` (a redundant release rebuild), the cold first-run still paid the full multi-GB TheRock download, the job exceeded the 35-min cap, and a scenario still ran its own `install sdk` (sharing didn't take effect) — and it wedged the runner. Revert the CI pre-warm so the branch is back to the safe pre-sharing baseline. The harness `use_shared_runtimes()` stays in place but dormant: it's a no-op unless E2E_SHARED_RUNTIMES_DIR is set, so no platform's behaviour changes. The sharing design is being reworked (WIP task #22) — key fixes: use the prebuilt debug binary for pre-warm, verify runtimes-list works through a symlinked registry, move pre-warm off the E2E clock, and persist the shared tree across runs. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
fredespi
added a commit
that referenced
this pull request
Jul 12, 2026
Priority is a complete consolidated report with all four platform columns; time optimization (shared runtime, task #22) is deferred. The MI300X collapsed run takes ~37min and was being cancelled at the 35min cap before writing platform.json. Set all three GPU jobs to 90min so they complete and produce their reports even on a cold runner. Signed-off-by: fredespi <fredrik.espinoza@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SystemTime::now().as_nanos()suffix intempdir()with anAtomicU64counterSystemTimehas ~15ms resolution; parallel tests running within the same timer tick could collide on the same directory path, causing tests to read/write each other'sresults.csvand produce wrong deserialization resultsmissing_30col_fields_default_safelygotrows[0].run == 1(from another test's ROW2) instead of 7Root cause
bench_tail::tests::tempdir()built its path asrocm-dash-test-{pid}-{nanos}. Under full parallel test load on Windows CI, the three tests in the module ran within the same 15ms clock tick → same suffix → same directory → sameresults.csv→ file contents from one test leaked into another.An atomic counter increments on every call so each invocation gets a distinct suffix regardless of clock resolution.
Test plan
cargo test -p rocm-dash-collectorspasses (54/54)Fixes flaky failure observed in https://github.com/ROCm/rocm-cli/actions/runs/27690632058/job/81900409577