fix: guard wait() against transient FAILED during saga startup

[pratyush618]

Summary

The previous fix (#201) moved _mark_run_compensating earlier in start_compensation to narrow the FAILED→COMPENSATING race window, but Python overhead between mark_workflow_node_result (Rust sets FAILED) and the Python-side transition still left a gap the 100ms poll could catch on slow Windows CI.

This PR eliminates the race entirely with two changes:

• wait() defers to the tracker event on transient terminal states. When the polled state is FAILED or COMPLETED_WITH_FAILURES and the tracker's waiter event hasn't been set yet, wait() defers to the event instead of returning immediately. The event is set only after the saga decision is made — either the saga finishes (COMPENSATED/COMPENSATION_FAILED) or no saga starts (FAILED is emitted as terminal). This closes the race regardless of how long the Python overhead takes.

• _mark_run_compensating retries on "database is locked". Transient SQLite write contention could silently leave the run in FAILED. The method now retries up to 5 times with backoff, matching the existing pattern in _dispatch_single.

Test plan

• All 23 saga tests pass
• All 98 workflow tests pass
• Full suite: 998 passed, 6 skipped
• ruff + mypy clean

Summary by CodeRabbit

• Bug Fixes
  • Improved workflow run status handling during compensation by deferring transient failure states until confirmation is available.
  • Enhanced database operation resilience during workflow compensation with automatic retry logic and exponential backoff, reducing interruptions from database locks.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 99d20ee1-3594-4f1a-8559-07ed89a0196b

📥 Commits

Reviewing files that changed from the base of the PR and between 235a8f5 and ce7826c.

📒 Files selected for processing (2)

• py_src/taskito/workflows/run.py
• py_src/taskito/workflows/saga/orchestrator.py

---

📝 Walkthrough

Walkthrough

This PR hardens the saga compensation flow by addressing two sides of the state transition. The orchestrator now retries transient database-lock errors when marking a run as Compensating, and the wait loop defers returning terminal states when a compensation transition is in flight, waiting for the tracker event instead of returning a potentially-stale status.

Changes

Saga Compensation Transition Reliability

Layer / File(s)	Summary
Orchestrator compensating transition with retry py_src/taskito/workflows/saga/orchestrator.py	_mark_run_compensating now retries up to 5 times on transient SQLite "database is locked" errors with exponential backoff, logging exceptions only after the final attempt.
Wait loop deferred terminal resolution py_src/taskito/workflows/run.py	wait() detects FAILED or COMPLETED_WITH_FAILURES when a tracker event is present but not yet set, and defers returning by waiting briefly for the event to fire before re-polling, reducing false-positive terminal statuses during compensation transitions.

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly Related PRs

• ByteVeda/taskito#201: Both PRs modify the saga compensation transition path; this PR adds retry/backoff to _mark_run_compensating, while the related PR changes when/how the run is marked Compensating during start_compensation to avoid observing FAILED during the transition.

Suggested Labels

python, workflows

Poem

🐰 A saga's path grows smooth and sure,
When locks don't block, and waits endure—
Retry, defer, and wait a beat,
Compensation flows complete! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'fix: guard wait() against transient FAILED during saga startup' accurately describes the main change—adding special handling in wait() to guard against transient FAILED states during saga initialization.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/saga-wait-race-guard

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}