Skip to content

[CCIP-12469] fix(commit): make observations non-blocking on stuck ops - #2193

Merged
makramkd merged 2 commits into
mainfrom
mk/unblock-chainfee
Jul 15, 2026
Merged

[CCIP-12469] fix(commit): make observations non-blocking on stuck ops#2193
makramkd merged 2 commits into
mainfrom
mk/unblock-chainfee

Conversation

@makramkd

@makramkd makramkd commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Problem

The commit plugin's chainfee.Observation was hanging indefinitely in production. Root cause: asynclib.WaitForAllNoErrOperations fans its operations out into goroutines and ends with an unconditional wg.Wait(). Despite setting a timeout on the context, however, it was not being honored by the LOOPP code because other callers are calling with an unbounded context (CCIP launcher here: https://smartcontract-it.atlassian.net/browse/CCIP-12413 already fixed, but headreporter has the same issue, still not fixed https://smartcontract-it.atlassian.net/browse/CCIP-12468) which caused the plugin caller to wait on a Lock() that would never get unlocked.

Solution

Replace the free function with a stateful asynclib.Runner:

  • Time-bounded barrier. WaitForAll waits for completions or the timeout, whichever comes first, and proceeds with what finished. A stuck operation can no longer hold the round hostage.
  • In-flight guard. An operation whose previous invocation is still running is skipped, not re-spawned. Goroutines are bounded to the number of distinct operations rather than accumulating one stuck goroutine per round.
  • Empty, never stale. Results are per-round; a missing operation degrades to its zero value, and a late result from an abandoned goroutine is discarded. We prefer a visible missing price (which alerts) over silently posting a stale one.

Both commit observers (chainfee, tokenprice) now run through a reused Runner. This is defense-in-depth at the plugin boundary: the plugin can no longer be blocked by any downstream that ignores its context, independent of the chainlink-common connection fix and the launcher timeout fix in the companion PRs.

Testing

  • New unit tests: an operation that ignores its context still lets WaitForAll return within the timeout; a stuck operation is not re-spawned on the next round.
  • Existing chainfee/tokenprice unit + e2e tests pass; configs that relied on the old "ignore the timeout and wait forever" behavior now set an explicit per-round timeout (production applies this via ApplyDefaultsAndValidate).

makramkd added 2 commits July 15, 2026 11:55
- Replace the free WaitForAllNoErrOperations (which ended in an
  unconditional wg.Wait) with a stateful asynclib.Runner whose
  WaitForAll returns on timeout, so one operation that ignores its
  context can no longer hang the whole Observation / OCR round.
- Add a per-operation in-flight guard: an op whose previous invocation
  is still running is skipped rather than re-spawned, bounding
  goroutines to the number of distinct ops instead of accumulating one
  stuck goroutine per round.
- Return only operations that completed this round; a missing result
  degrades to its zero value (empty), and a late result from an
  abandoned goroutine is discarded rather than served, so consensus
  never acts on stale data.
@github-actions

Copy link
Copy Markdown
Metric mk/unblock-chainfee main
Coverage 69.7% 69.4%

@makramkd
makramkd marked this pull request as ready for review July 15, 2026 11:14
@makramkd
makramkd requested review from a team as code owners July 15, 2026 11:14
@makramkd

Copy link
Copy Markdown
Collaborator Author

Verified on staging - commit plugin no longer blocks on a misconfigured Sui plugin.

@makramkd
makramkd added this pull request to the merge queue Jul 15, 2026
@makramkd makramkd changed the title fix(commit): make observations non-blocking on stuck ops [CCIP-12469] fix(commit): make observations non-blocking on stuck ops Jul 15, 2026
Merged via the queue into main with commit 1e32cea Jul 15, 2026
58 checks passed
@makramkd
makramkd deleted the mk/unblock-chainfee branch July 15, 2026 13:35
FelixFan1992 pushed a commit that referenced this pull request Jul 15, 2026
* fix(commit): make observations non-blocking on stuck ops

- Replace the free WaitForAllNoErrOperations (which ended in an
  unconditional wg.Wait) with a stateful asynclib.Runner whose
  WaitForAll returns on timeout, so one operation that ignores its
  context can no longer hang the whole Observation / OCR round.
- Add a per-operation in-flight guard: an op whose previous invocation
  is still running is skipped rather than re-spawned, bounding
  goroutines to the number of distinct ops instead of accumulating one
  stuck goroutine per round.
- Return only operations that completed this round; a missing result
  degrades to its zero value (empty), and a late result from an
  abandoned goroutine is discarded rather than served, so consensus
  never acts on stale data.

* lint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants