Harden orchestrator lifecycle and CI recovery - #138
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR hardens orchestrator recovery behavior by expanding workspace lifecycle cleanup to all configured repos and improving CI poller persistence/accuracy during retries and escalation, reducing partial-failure gaps in recovery paths.
Changes:
- Run startup + periodic workspace lifecycle checks (orphan sweep + age GC) across every configured repo key.
- Persist a “transition pending” CI escalation state before attempting Linear state updates, so failures don’t lose escalation intent.
- Track partially successful multi-run GitHub Actions reruns and avoid re-requesting reruns for runs that already succeeded.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 4 comments.
| File | Description |
|---|---|
lib/symphony_elixir/orchestrator.ex |
Expands startup and periodic lifecycle checks to multi-repo; adjusts active workspace identifier tracking by repo. |
lib/symphony_elixir/ci_poller.ex |
Persists escalation-pending state before Linear updates; tracks rerun run-ids across partial rerun failures; clears rerun fields on green/new SHA. |
test/symphony_elixir/orchestrator_status_test.exs |
Adds regression coverage for startup lifecycle cleanup across multiple configured repos. |
test/symphony_elixir/ci_poller_test.exs |
Adds regression tests for partial rerun persistence and escalation pending persistence on final CI record update failures. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+3225
to
3232
| defp workspace_age_gc_result(repo_keys, active_identifiers_by_repo) when is_list(repo_keys) do | ||
| Enum.reduce_while(repo_keys, {:ok, []}, fn repo_key, {:ok, actions} -> | ||
| case Workspace.reclaim_stale_workspaces(repo_key, Map.get(active_identifiers_by_repo, repo_key, [])) do | ||
| {:ok, repo_actions} -> {:cont, {:ok, actions ++ repo_actions}} | ||
| {:error, reason} -> {:halt, {:error, {repo_key, reason}}} | ||
| end | ||
| end) | ||
| end |
Comment on lines
+3099
to
+3108
| repo_results = | ||
| Enum.map(repo_keys, fn repo_key -> | ||
| run_terminal_workspace_cleanup(repo_key) | ||
|
|
||
| {repo_key, | ||
| %{ | ||
| orphan_sweep_result: startup_orphan_sweep_result(repo_key), | ||
| age_gc_scan: Workspace.scan_stale_workspaces(repo_key, DateTime.utc_now()) | ||
| }} | ||
| end) |
Comment on lines
+3180
to
+3186
| Enum.reduce(repo_results, state, fn | ||
| {repo_key, %{age_gc_scan: age_gc_scan}}, acc -> | ||
| apply_startup_workspace_age_gc_scan(acc, repo_key, age_gc_scan, now_ms) | ||
|
|
||
| _result, acc -> | ||
| acc | ||
| end) |
Comment on lines
452
to
461
| defp rerun_failed_run_ids(github, run_ids, workspace_path) when is_list(run_ids) do | ||
| Enum.reduce_while(run_ids, :ok, fn run_id, :ok -> | ||
| Enum.reduce_while(run_ids, {:ok, []}, fn run_id, {:ok, attempted_run_ids} -> | ||
| case github.rerun_failed(run_id, cwd: workspace_path) do | ||
| :ok -> | ||
| {:cont, :ok} | ||
| {:cont, {:ok, attempted_run_ids ++ [run_id]}} | ||
|
|
||
| {:error, reason} -> | ||
| {:halt, {:error, {run_id, reason}}} | ||
| {:halt, {:error, {run_id, reason, attempted_run_ids}}} | ||
| end | ||
| end) |
Summary: - Run startup and periodic workspace lifecycle checks for every configured repo. - Persist CI escalation intent before transitioning Linear issue state. - Track partial multi-run CI rerun progress by workflow run id. Rationale: - Multi-repo workspaces should not be skipped by cleanup or stale GC. - External issue transitions need a durable local pending state before side effects. - Successful reruns should not repeat after a later workflow rerun fails. Tests: - Targeted ci_poller and orchestrator_status regression tests. - Full ci_poller_test and orchestrator_status_test files. - mix specs.check - HEX_HOME=/private/tmp/symphony-hex-home make check - HEX_HOME=/private/tmp/symphony-hex-home make all
chihsuan
force-pushed
the
fix/orchestration-recovery-gaps
branch
from
July 1, 2026 07:53
d171dd2 to
c07e81f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Orchestrator recovery paths still had partial-failure gaps after recent hardening.
TL;DR
Harden multi-repo workspace cleanup and CI recovery persistence.
Summary
Alternatives
Test Plan
make all