feat(scripts): scheduled factory sync-fidelity canary (cron/launchd)#382
feat(scripts): scheduled factory sync-fidelity canary (cron/launchd)#382khaliqgant wants to merge 1 commit into
Conversation
Adds factory-canary.sh + a launchd template that run `factory canary <issue>` against the live relayfile mount on a schedule and assert a known "Ready for Agent" issue is still dispatch-ready by the real triage path. This is the regression detector for Linear sync-fidelity drift (sparse records / stub primaries — cloud#2284 / factory#10): on failure it exits non-zero and optionally posts a Slack alert, catching the regression before it silently blocks every factory dispatch. - factory-canary.sh: runs the canary from the pear root (reuses the Pear broker), bounded by FACTORY_CANARY_TIMEOUT, alerts via FACTORY_CANARY_SLACK_WEBHOOK. - com.agentrelay.factory-canary.plist: every-6h launchd template. - scripts/README.md: usage + the CI-vs-cron rationale (live canary needs the operator workspace; the canary logic itself is covered by factory unit CI). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Warning Review limit reached
More reviews will be available in 16 minutes and 59 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan refill rate. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, the refill rate gradually slows as usage increases. The highest same-day bursts are limited more strictly. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a sync-fidelity canary regression detector (factory-canary.sh), its documentation, and a launchd plist template for macOS scheduling. The review feedback focuses on improving the robustness of the bash script, including ensuring the script exits if directory navigation fails, adding macOS gtimeout support while preserving stderr, handling timeouts and empty responses explicitly in failure alerts, and allowing curl errors to be logged to stderr.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | ||
| cd "$ROOT" |
There was a problem hiding this comment.
Ensure that the script exits immediately if changing directory to the repository root fails, preventing subsequent commands from running in an incorrect or unexpected directory.
| ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | |
| cd "$ROOT" | |
| ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" | |
| cd "$ROOT" || exit 1 |
| if command -v timeout >/dev/null 2>&1; then | ||
| OUT="$(timeout "$TIMEOUT" "${RUN[@]}" 2>/dev/null)" | ||
| else | ||
| OUT="$("${RUN[@]}" 2>/dev/null)" | ||
| fi |
There was a problem hiding this comment.
On macOS, timeout is not installed by default, but gtimeout (from coreutils via Homebrew) is commonly available. Adding support for gtimeout ensures timeout protection works on macOS. Additionally, removing 2>/dev/null allows stderr to flow to the log file, which is critical for diagnosing why the canary failed.
| if command -v timeout >/dev/null 2>&1; then | |
| OUT="$(timeout "$TIMEOUT" "${RUN[@]}" 2>/dev/null)" | |
| else | |
| OUT="$("${RUN[@]}" 2>/dev/null)" | |
| fi | |
| if command -v timeout >/dev/null 2>&1; then | |
| OUT="$(timeout "$TIMEOUT" "${RUN[@]}")" | |
| elif command -v gtimeout >/dev/null 2>&1; then | |
| OUT="$(gtimeout "$TIMEOUT" "${RUN[@]}")" | |
| else | |
| OUT="$("${RUN[@]}")" | |
| fi |
| # Failure: alert. | ||
| REASON="$(printf '%s' "$VERDICT" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{const v=JSON.parse(s);console.log(`${v.status||"error"}: ${v.reason||"unknown"}`)}catch{console.log("unparseable verdict")}})' 2>/dev/null)" | ||
| MSG=":rotating_light: factory canary FAILED for ${ISSUE} — ${REASON}. Sync fidelity may have regressed (issue no longer dispatch-ready). See AgentWorkforce/cloud#2284 / factory#10." | ||
| echo "[$TS] $MSG" >&2 |
There was a problem hiding this comment.
If the command times out (exit code 124) or fails without outputting JSON, the parsed REASON will be "unparseable verdict". Handling these cases explicitly provides much more actionable Slack alerts.
| # Failure: alert. | |
| REASON="$(printf '%s' "$VERDICT" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{const v=JSON.parse(s);console.log(`${v.status||"error"}: ${v.reason||"unknown"}`)}catch{console.log("unparseable verdict")}})' 2>/dev/null)" | |
| MSG=":rotating_light: factory canary FAILED for ${ISSUE} — ${REASON}. Sync fidelity may have regressed (issue no longer dispatch-ready). See AgentWorkforce/cloud#2284 / factory#10." | |
| echo "[$TS] $MSG" >&2 | |
| # Failure: alert. | |
| if [[ $CODE -eq 124 ]]; then | |
| REASON="timeout after ${TIMEOUT}s (broker/mount may be wedged)" | |
| elif [[ -z "$VERDICT" ]]; then | |
| REASON="empty response (factory command failed)" | |
| else | |
| REASON="$(printf '%s' "$VERDICT" | node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{try{const v=JSON.parse(s);console.log(`${v.status||"error"}: ${v.reason||"unknown"}`)}catch{console.log("unparseable verdict")}})' 2>/dev/null)" | |
| fi | |
| MSG=":rotating_light: factory canary FAILED for ${ISSUE} — ${REASON}. Sync fidelity may have regressed (issue no longer dispatch-ready). See AgentWorkforce/cloud#2284 / factory#10." | |
| echo "[$TS] $MSG" >&2 |
| if [[ -n "${FACTORY_CANARY_SLACK_WEBHOOK:-}" ]]; then | ||
| curl -sS -m 15 -X POST -H 'Content-type: application/json' \ | ||
| --data "$(node -e 'process.stdout.write(JSON.stringify({text:process.argv[1]}))' "$MSG")" \ | ||
| "$FACTORY_CANARY_SLACK_WEBHOOK" >/dev/null 2>&1 \ | ||
| && echo "[$TS] factory-canary: posted Slack alert" >&2 \ | ||
| || echo "[$TS] factory-canary: Slack alert post failed" >&2 | ||
| fi |
There was a problem hiding this comment.
By redirecting both stdout and stderr of curl to /dev/null, any network or webhook failures are completely silenced. Removing 2>&1 allows curl -sS to print error messages to stderr on failure, which will be captured in the log file.
| if [[ -n "${FACTORY_CANARY_SLACK_WEBHOOK:-}" ]]; then | |
| curl -sS -m 15 -X POST -H 'Content-type: application/json' \ | |
| --data "$(node -e 'process.stdout.write(JSON.stringify({text:process.argv[1]}))' "$MSG")" \ | |
| "$FACTORY_CANARY_SLACK_WEBHOOK" >/dev/null 2>&1 \ | |
| && echo "[$TS] factory-canary: posted Slack alert" >&2 \ | |
| || echo "[$TS] factory-canary: Slack alert post failed" >&2 | |
| fi | |
| if [[ -n "${FACTORY_CANARY_SLACK_WEBHOOK:-}" ]]; then | |
| curl -sS -m 15 -X POST -H 'Content-type: application/json' \ | |
| --data "$(node -e 'process.stdout.write(JSON.stringify({text:process.argv[1]}))' "$MSG")" \ | |
| "$FACTORY_CANARY_SLACK_WEBHOOK" >/dev/null \ | |
| && echo "[$TS] factory-canary: posted Slack alert" >&2 \ | |
| || echo "[$TS] factory-canary: Slack alert post failed" >&2 | |
| fi |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ca707ff5c8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # prints a JSON verdict {ok,issue,status,reason}; exit code mirrors ok. A hung | ||
| # run (e.g. broker/mount wedge) is bounded by FACTORY_CANARY_TIMEOUT. | ||
| TIMEOUT="${FACTORY_CANARY_TIMEOUT:-180}" | ||
| RUN=(node "$BIN" factory canary "$ISSUE" --config "$CONFIG" --backend internal) |
There was a problem hiding this comment.
Because package-lock.json registers bin/factory.mjs itself as the factory executable (bin.factory = "bin/factory.mjs"), invoking node "$BIN" factory canary ... passes an extra positional and runs the equivalent of factory factory canary ... instead of the documented factory canary .... With a normal subcommand CLI this makes every scheduled canary fail as an unknown command and can generate false failure alerts rather than checking the issue.
Useful? React with 👍 / 👎.
| if command -v timeout >/dev/null 2>&1; then | ||
| OUT="$(timeout "$TIMEOUT" "${RUN[@]}" 2>/dev/null)" | ||
| else | ||
| OUT="$("${RUN[@]}" 2>/dev/null)" |
There was a problem hiding this comment.
In the macOS launchd setup added with this script, stock systems do not provide GNU timeout, so this fallback runs the factory process with no bound at all even though FACTORY_CANARY_TIMEOUT is meant to catch broker/mount wedges. If the live mount or broker hangs, the launchd job can remain stuck indefinitely and stop producing the failure alert the canary is supposed to provide.
Useful? React with 👍 / 👎.
|
All references are consistent. The plist Label, filename, README reference, and launchctl commands all use I made no code edits. Everything in the diff is correct or unverifiable-but-not-fixable mechanically. No edits were left in the working tree. SummaryPR #382 adds a scheduled "factory sync-fidelity canary": a standalone bash script ( Impact on CI: None. The PR touches only What I verified in the current checkout:
No auto-edits made. Nothing was mechanically wrong (no lint/format/typo/import issues in the linted file set; the Addressed comments
Advisory Notes
The PR is documentation + an operator-only scheduled script that does not run in CI. Whether it's mergeable / all required checks are green is a post-harness determination I can't make from here, and the one open item (the |
|
ℹ️ pr-reviewer: review only — no file changes were applied to the PR (nothing to commit after review). The notes below are advisory and were not pushed. Review: PR #382 —
|
|
Superseded by AgentWorkforce/factory#11 — the canary scheduling tooling (factory-canary.sh + launchd template) now lives in the factory package, next to the |
1 participant
Adds a scheduled canary that catches Linear sync-fidelity regressions before they silently block factory dispatch — the regression-prevention control for the class of bug fixed in cloud#2284 (Nango sync dropped
state.id/team/labels) and factory#10 (factory read tolerance + thefactory canarycommand).What it does
scripts/factory-canary.shrunsfactory canary <issue>against the live relayfile mount and asserts a known "Ready for Agent" issue is still classified dispatch-ready by the real dry-run triage path. On failure it exits non-zero and (optionally) posts a Slack alert.scripts/factory-canary.sh— runs from the pear root (reuses the running Pear broker), bounded byFACTORY_CANARY_TIMEOUT, alerts viaFACTORY_CANARY_SLACK_WEBHOOK.scripts/com.agentrelay.factory-canary.plist— every-6h launchd template with install steps.scripts/README.md— usage + the CI-vs-cron rationale.Why cron, not package CI
A live canary needs the operator's workspace creds + the relayfile mount, so it belongs in a scheduled job in the operator environment — not in the factory package's CI. The canary logic itself is already covered by the factory unit suite (the stub-primary golden test runs on every PR in factory#10).
Dependency
Requires a factory build with the
canarycommand (factory#10+). Until that publishes, pointFACTORY_BINat a local build (the plist has a slot for it).🤖 Generated with Claude Code