Fix scheduler race in BackgroundJobManagerActor startup reconciliation - #1417
Merged
Aaronontheweb merged 1 commit intoJun 16, 2026
Conversation
Aaronontheweb
enabled auto-merge (squash)
June 16, 2026 15:00
Self.Tell(Reconcile.Instance) in PreStart races on slow schedulers (macOS CI): ActorOf returns before PreStart executes, allowing external messages to arrive and queue in the user mailbox ahead of the Reconcile message. This caused StartBackgroundJob commands and GetBackgroundJobManagerHealth checks to be processed before reconciliation, producing four flaky test failures exclusive to macos-26: - StartupReconciliation_MarksOrphanedJobsAsLost: health barrier resolved before reconciliation ran, so store still showed Running - StartupReconciliation_EmitsAlert_ForLegacyJobMissingTrustFields: same — sink was empty when checked, alert emitted during actor teardown - BackgroundJob_WithMissingWorkingDirectory_FailsWithHelpfulError: freshly started job was reconciled as Lost before the execution actor ran - CancelRunningJob_ViaCheckBackgroundJobTool: same reconcile-before-execute race; delivered 'was lost' instead of 'cancelled' Fix: call HandleReconcile() synchronously in PreStart. Akka.NET guarantees PreStart completes before any user message is dispatched, eliminating the race entirely. Remove the now-dead Reconcile record, its Receive<> handler, and the _startupSchemaAlertsEmitted guard (which only guarded against a re-entrant path that no longer exists).
Aaronontheweb
force-pushed
the
fix/background-job-reconcile-prestart-race
branch
from
June 16, 2026 15:00
2bc7004 to
d6242f8
Compare
Aaronontheweb
added a commit
that referenced
this pull request
Aug 1, 2026
…iation test (#1731) StartupReconciliation_DeliversLostNotificationToOwningSession was flaky on Windows CI: the manager created in ConfigureAkka runs its PreStart reconciliation asynchronously on the dispatcher, and if it landed after the orphan was persisted but before its output.log was written, it delivered a 'lost' notification with no output path (NotifyLostJob swallows FileNotFound and nulls OutputFilePath) — so Assert.Contains (logPath, delivery.Content) failed. PreStart completes before any user message dispatches, so a health barrier against the setup manager before seeding the orphan proves it reconciled an empty store. Only the freshly created manager (after the log write) reconciles the orphan, so the path is always present. Fixes the race deterministically (no timeout changes); matches the readiness-barrier idiom from #1410/#1417.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Self.Tell(Reconcile.Instance)inPreStartraced on slow schedulers (macOS CI):ActorOfreturns theActorRefbeforePreStartexecutes, so external messages could arrive in the user mailbox ahead of theReconcilemessageHandleReconcile()synchronously inPreStart— Akka.NET guaranteesPreStartcompletes before any user message is dispatched, eliminating the race entirelyReconcilerecord, itsReceive<>handler, and_startupSchemaAlertsEmittedguardRoot cause
Four tests failed exclusively on
macos-26(job 81675499970):StartupReconciliation_MarksOrphanedJobsAsLostRunningStartupReconciliation_EmitsAlert_ForLegacyJobMissingTrustFieldsBackgroundJob_WithMissingWorkingDirectory_FailsWithHelpfulErrorLostbefore the execution actor ranCancelRunningJob_ViaCheckBackgroundJobToolThe macOS CI workers have a different thread scheduler that delays actor dispatch enough for the test body to enqueue messages before
PreStartexecutes.Test plan
Netclaw.Actors.Testssuite passes (2196 tests)dotnet slopwatch analyze— no new violations