Fix durable task flaky integration test#1784
Open
siri-varma wants to merge 10 commits into
Open
Conversation
Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Contributor
Author
|
@javier-aliaga durable task has a flaky test. Fixed it as part of this PR |
Contributor
Author
|
nvm. It failed again |
Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
… clock The assertion on the internal sub-timer count (counter >= 4) was flaky under CI load. The absolute timer deadline was computed on the test's wall clock at method start, but the number of sub-timers created for a long timer depends on the span from the orchestration's *creation time* to that deadline. Worker startup + scheduling latency shrinks that span, and when it exceeded ~1s one sub-timer was dropped, producing counter == 3 and failing 'expected <true> but was <false>' at line 387. Anchor the deadline to the orchestration's own replay-safe clock via ctx.getCurrentInstant() (equivalent to what createTimer(Duration) does internally), so the timer always spans exactly the intended delay regardless of startup latency, and assert the splitting invariant (counter >= 3) instead of a latency-sensitive exact count. Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Contributor
Author
|
@javier-aliaga need one review on this pr please |
Wrap the flaky integration/validation steps in nick-fields/retry@v3 (max_attempts: 2) to absorb transient CI failures: - build-durabletask: retry the mvnw integration-tests step; switch the failure gate to the action's outcome output and bump the job timeout to 45m to fit two attempts. - validate: retry each mm.py example validation step (dir folded into the command since uses: steps can't set working-directory). Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
siri-varma
changed the title
Set per-attempt timeout_minutes to 30 on all nick-fields/retry blocks (durable task + auto-validate) and revert the build-durabletask job timeout back to its original 30m. The 30m cap is applied per step; no job-level cap is added to the validate job since it runs 19 example validations sequentially. Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Collapse the 19 per-example nick-fields/retry blocks into a single retry step that runs all example validations. On failure the whole validation batch reruns once, matching the job-level retry granularity (no per-example retries). Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Wrap the sb3.5 and sb4 integration-test commands in nick-fields/retry (max_attempts: 2, timeout_minutes: 30). continue_on_error is left off so an exhausted retry still fails the step, keeping the existing failure() artifact uploads and job-failure behavior intact. Signed-off-by: Siri Varma Vegiraju <siri.varma@outlook.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #1784 +/- ##
=========================================
Coverage 77.02% 77.02%
Complexity 2324 2324
=========================================
Files 245 245
Lines 7198 7198
Branches 760 760
=========================================
Hits 5544 5544
Misses 1288 1288
Partials 366 366 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes flaky Durable Task integration tests and adds a CI retry safety net so transient infrastructure hiccups don't fail otherwise-green runs.
Test fixes (
DurableTaskClientIT)longTimeStampTimer— The timer's fire time was computed from the test's wall clock at method start, and the test then asserted an exact internal sub-timer count (counter >= 4). The number of sub-timers a long timer is split into depends on the span from the orchestration's creation time to the deadline; worker startup + scheduling latency shrank that span under CI load, occasionally dropping a sub-timer and failingexpected <true> but was <false>. Fix: anchor the timer to the orchestration's own replay-safe clock viactx.getCurrentInstant()(whatcreateTimer(Duration)does internally) so the span is always exactlydelay, and assert the splitting invariant (counter >= 3) instead of a latency-sensitive exact count.waitForInstanceStartThrowsException— Scheduling the instance on a background thread racedwaitForInstanceStart: if the wait reached the sidecar first it failed with "no such instance exists"; if the schedule won, the worker started the instance within 2s and nothing was thrown. Fix: schedule with a future start time so the instance deterministically exists but staysPending, guaranteeingwaitForInstanceStartblocks to its 2s deadline and throwsTimeoutException.CI retries (
build.yml,validate.yml)Wrap the flaky integration/validation commands in
nick-fields/retry@v3(max_attempts: 2,timeout_minutes: 30) at job granularity:mvnw ... verifystep;continue_on_errorkeeps the sidecar-log upload and the outcome-gated fail step intact.continue_on_erroris left off so an exhausted retry still fails the step, preserving the existingfailure()artifact uploads and job-failure behavior.Retries are a safety net for transient failures (sidecar/gRPC connect races, example runs), not a substitute for the root-cause test fixes above.
Issue reference
[add issue number if this closes one]
Checklist
Please make sure you've completed the relevant tasks for this PR, out of the following list: