Skip to content

[oximeter] really fix race in test_self_stat_collection_count#10585

Merged
sunshowers merged 3 commits into
mainfrom
sunshowers/spr/oximeter-really-fix-race-in-test_self_stat_collection_count
Jun 11, 2026
Merged

[oximeter] really fix race in test_self_stat_collection_count#10585
sunshowers merged 3 commits into
mainfrom
sunshowers/spr/oximeter-really-fix-race-in-test_self_stat_collection_count

Conversation

@sunshowers

@sunshowers sunshowers commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Following on from #10040.

Switch the test to being level-triggered, and don't rely on paused time because we do real I/O which would introduce a second time domain.

Identified and fixed by Claude based on the flake-patterns doc I wrote in #10577.

Depends on:

Fixes #7255.

Fixes #8636.

Created using spr 1.3.6-beta.1

[skip ci]
Created using spr 1.3.6-beta.1

@bnaecker bnaecker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! This looks good overall, but I have one comment about disabling collections entirely.

Comment thread oximeter/collector/src/agent.rs Outdated
// Whether collection tasks schedule collections on their producers'
// intervals. Always enabled in production; tests disable it to control
// exactly how many collections occur.
scheduled_collections: ScheduledCollections,

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm a little nervous about setting up a mechanism where no collections are ever performed on the usual timer. Leaking that into production would be pretty bad. Can we instead set up the tests we're fixing here with very long collection intervals (i.e., days)? That would provide complete control over the collections without any additional risk.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Heh that's actually what I had earlier and just didn't feel great about it! I'll switch back to that since you prefer it.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Both suck a little, but I think at least that approach can't torpedo all collections rack-wide.

@sunshowers sunshowers changed the base branch from sunshowers/spr/main.oximeter-really-fix-race-in-test_self_stat_collection_count to main June 10, 2026 21:06
Created using spr 1.3.6-beta.1
@sunshowers sunshowers marked this pull request as ready for review June 10, 2026 21:09
@sunshowers sunshowers requested a review from bnaecker June 10, 2026 21:09
@sunshowers sunshowers merged commit cfdb3c3 into main Jun 11, 2026
18 checks passed
@sunshowers sunshowers deleted the sunshowers/spr/oximeter-really-fix-race-in-test_self_stat_collection_count branch June 11, 2026 00:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test failed in CI: test_self_stat_collection_count test failed in CI: oximeter_collector::agent::tests::test_self_stat_error_counter

2 participants