Skip to content

fix(factory): park completed PRs in human review#362

Merged
khaliqgant merged 2 commits into
mainfrom
ar-273-human-review-terminal-state
Jun 15, 2026
Merged

fix(factory): park completed PRs in human review#362
khaliqgant merged 2 commits into
mainfrom
ar-273-human-review-terminal-state

Conversation

@khaliqgant

@khaliqgant khaliqgant commented Jun 15, 2026

Copy link
Copy Markdown
Member

Summary

  • add the AR Human Review workflow-state UUID while keeping default configs backward-compatible when stateIds.humanReview is omitted
  • make configured completions land in Human Review and update completion Slack text accordingly
  • handle merged PR metadata by advancing Agent Implementing/In Human Review issues to Done, with duplicate suppression
  • allow Human Review -> Ready for Agent re-dispatch

Review Notes

  • LINEAR_STATE_IDS is documented as the AR-team state catalog; FactoryConfig still omits humanReview from the default stateIds, so non-AR operators must opt into their own review-state UUID before terminalState: "human-review" changes behavior.
  • PR metadata events are now observed even when babysitter.enabled is false so merged PRs can advance review-state issues to Done. Legacy direct-to-Done configurations should be unaffected because the post-merge handler only advances issues still in Agent Implementing or Human Review.
  • The no-record merged-PR fallback scans the issue mirror for matching Agent Implementing/Human Review issues and prefers branch/headRef matches over title/body references; this is intentionally simple until the mirror provides a stronger PR-to-issue identity.

Tests

  • npx vitest run packages/factory-sdk/src/config/schema.test.ts packages/factory-sdk/src/orchestrator/factory.test.ts

E2E Notes

  • Unit coverage captures the four behavioral paths requested: configured Human Review, fallback without Human Review, Human Review -> Ready redispatch, and merged PR -> Done.
  • Real Linear activity-log screenshots/entries for the spec's E2E matrix still need operator verification against a live issue/PR before merging to main; this PR does not auto-merge or perform live Linear state transitions itself.

Do NOT auto-merge. Merge policy: never.

Review in cubic

@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@coderabbitai

coderabbitai Bot commented Jun 15, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@khaliqgant, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 9 minutes and 44 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: c51dcdd0-86df-4f39-a3f8-84b71d8573ad

📥 Commits

Reviewing files that changed from the base of the PR and between 14909ab and 5594a68.

📒 Files selected for processing (6)
  • packages/factory-sdk/README.md
  • packages/factory-sdk/src/config/schema.test.ts
  • packages/factory-sdk/src/config/schema.ts
  • packages/factory-sdk/src/constants/linear.ts
  • packages/factory-sdk/src/orchestrator/factory.test.ts
  • packages/factory-sdk/src/orchestrator/factory.ts
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch ar-273-human-review-terminal-state

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b64adb1588

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

this.#postMergeDoneAdvances.add(advanceKey)

try {
await this.#linear.setState(issue, this.#config.stateIds.done)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Honor dryRun before post-merge writeback

When config.dryRun is enabled and a merged PR event arrives for an existing Human Review/Agent Implementing issue that is not currently in #batch, this new post-merge path still calls setState and can also post Slack messages. Other external write paths, such as closed GitHub issue handling, explicitly skip writes under dry-run, so a dry-run live daemon can now mutate Linear just by observing a merged PR webhook.

Useful? React with 👍 / 👎.

Comment on lines +2643 to 2645
const humanReview = opts.targetState !== 'done' &&
this.#config.terminalState === 'human-review' &&
Boolean(this.#config.stateIds.humanReview)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Keep closed GitHub mirrors on the Done path

With terminalState: 'human-review' and stateIds.humanReview configured, this default now sends every #completeIssue() caller that omits targetState to Human Review. That includes #handleGithubIssueChange when a mirrored GitHub issue closes while its Linear mirror is in-flight; the source issue is already closed, so the mirror should be completed to Done, not parked for human review with agents released as issue-human-review.

Useful? React with 👍 / 👎.

@khaliqgant

Copy link
Copy Markdown
Member Author

Addressed the blocking review item: humanReview is no longer baked into LINEAR_STATE_IDS; it remains operator-configured via stateIds.humanReview / factory.config.json.

Also kept the follow-up refinements for the non-blocking comments:

  • record-path merged completion uses merged Slack wording
  • no-record PR scan now scores headRef > title > explicit body refs
  • scan logs low-noise debug timing/match info
  • README documents that humanReview is intentionally omitted from defaults

Verification rerun:

  • vitest run packages/factory-sdk/src/config/schema.test.ts packages/factory-sdk/src/orchestrator/factory.test.ts
  • tsc --noEmit -p packages/factory-sdk/tsconfig.json

@khaliqgant khaliqgant force-pushed the ar-273-human-review-terminal-state branch from 9fa7856 to 5594a68 Compare June 15, 2026 14:04
@agent-relay-code

Copy link
Copy Markdown
Contributor

ℹ️ pr-reviewer: review only — no file changes were applied to the PR (nothing to commit after review). The notes below are advisory and were not pushed.

Findings

  1. packages/factory-sdk/src/orchestrator/factory.ts:2419
    The no-record merged-PR advance path ignores config.dryRun. If a dry-run daemon observes a merged PR for a matching Agent Implementing/Human Review issue, it still calls linear.setState(...) and can post Slack writeback at 2430. This is an external mutation from dry-run mode. This should be gated before adding the duplicate-suppression key or writing state/Slack, likely mirroring the existing closed-GitHub-issue dry-run behavior.

  2. packages/factory-sdk/src/orchestrator/factory.ts:1341
    Closed GitHub issue mirror completion now calls #completeIssue(record) without forcing Done. With terminalState: 'human-review' and stateIds.humanReview configured, #completeIssue resolves humanReview true at 2658, so an already-closed source GitHub issue can park its in-flight Linear mirror in Human Review and release agents as issue-human-review. This path should force the Done terminal state.

I did not auto-edit either issue because both touch semantic completion/writeback behavior in safety-sensitive lifecycle code.

Addressed comments

  • gemini-code-assist[bot]: quota warning only; no code finding to validate.
  • coderabbitai[bot]: review was rate-limited; no code finding to validate.
  • chatgpt-codex-connector[bot]: “Honor dryRun before post-merge writeback” validated in packages/factory-sdk/src/orchestrator/factory.ts:2419; left as a required human patch because it changes writeback semantics.
  • chatgpt-codex-connector[bot]: “Keep closed GitHub mirrors on the Done path” validated at packages/factory-sdk/src/orchestrator/factory.ts:1341 and packages/factory-sdk/src/orchestrator/factory.ts:2658; left as a required human patch because it changes completion semantics.
  • khaliqgant: status comment about prior fixes and local verification; current checkout confirms the default stateIds omission behavior, but the two later Codex findings above remain valid.

Verification

Local commands passed: npm ci, npm run verify:mcp-resources-drift, npm run lint, npm run typecheck:web, npm run typecheck:node, npm test, npx vitest run, npm run build, npm run build:web, npx playwright test --config playwright.fidelity.config.ts, npx playwright test --config playwright.redraw.config.ts.

GitHub reports PR #362 is open, mergeable, not draft, and CI run 27551832167 succeeded for checks, playwright, and packaged-mcp-smoke. Note: .workforce/context.json lists head 9fa7856d..., while GitHub currently reports head 5594a689...; I reviewed the current checkout and GitHub metadata available in this run.

@agent-relay-code

Copy link
Copy Markdown
Contributor

Findings

  1. factory.ts:3358 drops existing project/keyword routing when no repo label matches. The old triage path routes by label, then project, then keyword, then default, but labelDerivedDispatchDecision now falls straight from label matches to repos.default. An issue with a non-repo label plus a repos.byProject or keywordRules match will dispatch to the default repo, or fail if no default is configured. Suggest falling back to decision.routes when there are no repo-label routes, before using repos.default.

  2. factory.ts:1057 can repeatedly post the same “invalid repo labels” Linear comment. The invalid-label path returns before recording any dispatch attempt, terminal state, backoff, or per-issue block marker, so repeated runLoop iterations or duplicate live events can re-post the same comment for the same ready issue. Given duplicate delivery is normal here, this should be made idempotent before merge.

Addressed comments

  • gemini-code-assist[bot]: quota-limit warning only; no actionable code finding to validate or fix.
  • coderabbitai[bot]: review-rate-limit warning only; no actionable code finding to validate or fix.

Advisory Notes

  • The user task says PR #362, but .workforce/context.json and the GitHub connector identify the checked-out PR as #363 (AR-274 route factory implementers by repo label). I reviewed the checked-out diff and PR #363 metadata.
  • .workforce/context.json listed head SHA ec624603..., while GitHub reports current head 33c5f815.... I based the code review on the current checkout plus .workforce/pr.diff, as requested.
  • I made no semantic edits.

Verification

Passed locally after a clean npm ci:

  • npm run verify:mcp-resources-drift
  • npm run lint (warnings only, outside this PR)
  • npm run typecheck:web
  • npm run typecheck:node
  • npm test
  • npx vitest run
  • npm run build
  • npm run build:web
  • npx playwright test --config playwright.fidelity.config.ts
  • npx playwright test --config playwright.redraw.config.ts

I did not print READY because the two review findings above need human-authored behavior changes before this PR is genuinely ready.

@agent-relay-code

Copy link
Copy Markdown
Contributor

Findings

  1. packages/factory-sdk/src/orchestrator/factory.ts: the no-record merged-PR advance path still ignores config.dryRun. A dry-run daemon observing a merged PR for a matching Agent Implementing/Human Review issue can call linear.setState(...) and post Slack writeback. This is semantic writeback behavior, so I did not auto-edit it.

  2. packages/factory-sdk/src/orchestrator/factory.ts: closed GitHub issue mirror completion calls #completeIssue(record) without forcing Done. With terminalState: 'human-review' and stateIds.humanReview, #completeIssue resolves Human Review at line 2658, so an already-closed source issue can be parked in review instead of Done. This is lifecycle/completion behavior, so I left it for a human patch.

Addressed Comments

  • gemini-code-assist[bot]: quota warning only; no code finding to validate.
  • coderabbitai[bot]: review-rate-limit warning only; no code finding to validate.
  • chatgpt-codex-connector[bot]: “Honor dryRun before post-merge writeback” validated at packages/factory-sdk/src/orchestrator/factory.ts:2419; not auto-edited because it changes safety-critical writeback semantics.
  • chatgpt-codex-connector[bot]: “Keep closed GitHub mirrors on the Done path” validated at packages/factory-sdk/src/orchestrator/factory.ts:1341 and packages/factory-sdk/src/orchestrator/factory.ts:2658; not auto-edited because it changes completion semantics.
  • khaliqgant: status comment about earlier fixes; current checkout confirms FactoryConfigSchema defaults omit stateIds.humanReview, while LINEAR_STATE_IDS still contains the AR catalog value.
  • agent-relay-code[bot] at 14:12:41Z: same two findings above remain valid in the current checkout.
  • agent-relay-code[bot] at 14:22:03Z: invalid for this PR; it discusses a different checked-out PR/routing diff and does not apply to PR fix(factory): park completed PRs in human review #362’s current .workforce/pr.diff.

Verification

No files were edited.

Passed locally: npm ci, npm run verify:mcp-resources-drift, npm run lint (warnings only), npm run typecheck:web, npm run typecheck:node, npm test, npx vitest run, npm run build, npm run build:web, Playwright fidelity, Playwright redraw. Initial parallel runs hit memory/temp cleanup noise; sequential reruns passed.

GitHub API reports PR #362 is open, non-draft, mergeable with mergeable_state: clean, and check runs checks, playwright, and packaged-mcp-smoke are completed successfully. I did not print READY because the two validated safety findings still need human-authored behavior changes.

@khaliqgant khaliqgant merged commit bdae970 into main Jun 15, 2026
4 checks passed
@khaliqgant khaliqgant deleted the ar-273-human-review-terminal-state branch June 15, 2026 17:44
khaliqgant added a commit that referenced this pull request Jun 15, 2026
Merge origin/main (#360 webhook handler, #362 park completed PRs). The
ar-366 re-dispatch and ar-243 PR-state sweep tests came from main with
pre-label-routing implementer names; under AR-274 label routing the
implementer is repo-slug suffixed (ar-366-impl-pear / ar-243-impl-pear).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
khaliqgant added a commit that referenced this pull request Jun 15, 2026
* AR-274 route implementers by repo labels

* fix(factory): fail dispatch without repo labels

* AR-274 rebind reviewer to label route

* AR-274 require repo labels for dispatch (no default-route fallback)

Remove defaultRouteAssignment/repoSlug so an unlabeled or unmapped issue
fails dispatch loudly instead of silently routing to repos.default. Cap
label implementers at MAX_LABEL_IMPLEMENTERS=4 and rebind the reviewer to
the first label route via routeReviewerSpec().

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(factory): green CI + dedupe invalid-label dispatch comments

- fleet.test.ts: implementer name is now repo-slug suffixed (ar-77-impl-pear)
  under label routing.
- Dedupe the invalid-label dispatch failure comment by reason+offending-labels
  signature so a stuck Ready issue (or the writeback's own change event) no
  longer re-posts the same notice every cycle; clear on successful dispatch.

Addresses Codex review on PR #363.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test(factory): suffix implementer names for merged-in main tests

Merge origin/main (#360 webhook handler, #362 park completed PRs). The
ar-366 re-dispatch and ar-243 PR-state sweep tests came from main with
pre-label-routing implementer names; under AR-274 label routing the
implementer is repo-slug suffixed (ar-366-impl-pear / ar-243-impl-pear).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant