feat: add corpus quality gates to GitHub Action

[Esquetta]

Summary

• run private corpus quality metrics from the composite GitHub Action
• optionally compare a retained baseline and fail on precision, recall, or false-positive regressions
• expose redacted metrics reports through Action outputs and the artifact manifest
• document the CI workflow and input requirements

Verification

• 56 test files passed
• 597 tests passed
• TypeScript build passed
• git diff --check passed

[qodo-code-review]

PR Summary by Qodo

Add corpus quality regression gates to the composite GitHub Action

✨ Enhancement 📝 Documentation 🧪 Tests 🕐 20-40 Minutes

AI Description

• Add optional corpus quality metrics evaluation and regression gating to the GitHub Action.
• Emit redacted metrics reports via Action outputs and the action artifact manifest.
• Document new inputs/outputs and validate metadata wiring with updated tests.

Diagram

graph TD
  A["GitHub Workflow"] --> B["Composite Action"] --> C["codex-plugin-doctor CLI"] --> D[("Reports dir")]
  B --> E[("Baseline JSON")]
  B --> F{"Regression gate?"} --> D
  D --> G["Action outputs + manifest"]

  subgraph Legend
    direction LR
    _p["Process"] ~~~ _d{"Decision"} ~~~ _a[("Artifact / File")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Implement gating in a JavaScript/TypeScript action wrapper

• ➕ Cleaner input validation and error messaging than bash conditionals
• ➕ Easier to unit-test branching behavior beyond metadata assertions
• ➕ More robust path/JSON handling across shells and platforms
• ➖ Bigger refactor: introduces runtime/node dependencies and packaging changes
• ➖ Less aligned with current composite-action approach
• ➖ Higher maintenance burden for what is mostly CLI orchestration

2. Split corpus metrics into a separate workflow step/job

• ➕ Keeps the action focused on validation, avoiding feature creep
• ➕ Lets teams manage private corpus access/baselines independently
• ➕ Simplifies action.yml logic
• ➖ More boilerplate for every consumer workflow
• ➖ Harder to standardize artifact naming and outputs across repos
• ➖ Loses the single-entry-point experience the action provides

Recommendation: The PR’s approach (extending the composite action to call the CLI’s existing corpus metrics and metrics diff commands) is the best fit for the current design: it keeps all CI wiring in one place, reuses the CLI for regression semantics, and limits exposure to redacted outputs. If this logic grows further (more gates, richer comparisons), consider moving orchestration to a small TS wrapper for testability and maintainability.

Files changed (4) +98 / -1

Enhancement (1) +54 / -0

action.ymlAdd corpus metrics inputs, outputs, and regression gating orchestration +54/-0

Add corpus metrics inputs, outputs, and regression gating orchestration

• Introduces new inputs for a private corpus metrics manifest, an optional baseline report, and an optional fail-on-regression gate. Generates 'corpus-metrics.json' and, when configured, 'corpus-metrics-diff.json', then exposes both paths via action outputs and the artifact manifest with guardrails for invalid input combinations.

action.yml

Tests (1) +15 / -0

action-metadata.test.tsExtend action metadata tests for corpus metrics wiring +15/-0

Extend action metadata tests for corpus metrics wiring

• Expands metadata assertions to cover the new corpus metrics inputs, outputs, and bash command invocations. Verifies the new report filenames and output path exports appear in the action metadata and usage text.

tests/action-metadata.test.ts

Documentation (2) +29 / -1

README.mdMention optional corpus quality metrics reports in Action outputs +1/-1

Mention optional corpus quality metrics reports in Action outputs

• Updates the Action overview to include optional validation corpus and quality metrics reports in the generated artifact directory. Also calls out corpus quality regression gates as a supported workflow capability.

README.md

github-action.mdDocument corpus quality gates usage and constraints +28/-0

Document corpus quality gates usage and constraints

• Adds documentation for configuring corpus metrics in CI, including examples for running metrics and failing on regressions against a retained baseline. Clarifies that reports must share a 'corpusDigest' and that 'fail-on-regression' requires both manifest and baseline inputs.

docs/guides/github-action.md

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (2) 📘 Rule violations (0) 📜 Skill insights (0)

1. Wrong manifest diff enabled 🐞 Bug ≡ Correctness

Description

In action.yml, CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF is derived solely from
corpus-metrics-baseline being non-empty, even when corpus-metrics-manifest is missing (a
configuration the script rejects and skips). This causes codex-plugin-doctor-action-manifest.json to
advertise corpusMetricsDiff.enabled=true with a path to a report that will not be generated, which
can break downstream steps that rely on the manifest to discover existing artifacts.

Code

action.yml[R331-332]

+        export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS="$([[ -n "$CORPUS_METRICS_MANIFEST_INPUT" ]] && echo true || echo false)"
+        export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF="$([[ -n "$CORPUS_METRICS_BASELINE_INPUT" ]] && echo true || echo false)"

Evidence

The script rejects baseline/gating without a manifest (record_status=2) and skips running the
metrics/diff commands, but later sets the diff-enabled env var based only on baseline presence, and
the Node heredoc writes that flag into the action manifest used for artifact discovery.

action.yml[262-270]
action.yml[331-332]
action.yml[345-376]
docs/guides/github-action.md[43-48]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF` is exported as `true` whenever `corpus-metrics-baseline` is non-empty, even in invalid configurations where the script skips running metrics/diff (e.g., baseline provided without manifest). This makes the generated `codex-plugin-doctor-action-manifest.json` inaccurate.

### Issue Context
The action explicitly treats `corpus-metrics-baseline` (and regression gating) as requiring `corpus-metrics-manifest`, but the manifest “enabled” flags are computed later from inputs alone.

### Fix Focus Areas
- action.yml[262-283]
- action.yml[331-332]
- action.yml[345-376]

### Suggested fix
Introduce explicit booleans (e.g., `corpus_metrics_enabled`, `corpus_metrics_diff_enabled`) initialized to `false` and set them to `true` only in the branches where the corresponding `run_doctor` call is actually scheduled (attempted). Export `CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS*` from these booleans rather than from raw input presence.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Baseline file not validated 🐞 Bug ☼ Reliability

Description

The action runs doctor corpus metrics diff whenever corpus-metrics-baseline is non-empty, but it
does not verify that the baseline path exists before invoking the CLI. This turns a simple
configuration/path problem into a runtime CLI failure with less targeted messaging, and it’s
avoidable in the action script.

Code

action.yml[R273-281]

+          if [[ -n "$CORPUS_METRICS_BASELINE_INPUT" && -f "$corpus_metrics_path" ]]; then
+            corpus_metrics_diff_args=(doctor corpus metrics diff --before "$CORPUS_METRICS_BASELINE_INPUT" --after "$corpus_metrics_path" --json --output "$corpus_metrics_diff_path")
+
+            if [[ "$CORPUS_METRICS_FAIL_ON_REGRESSION_INPUT" == "true" ]]; then
+              corpus_metrics_diff_args+=(--fail-on-regression)
+            fi
+
+            run_doctor "corpus metrics regression" "${corpus_metrics_diff_args[@]}"
+          fi

Evidence

The diff gate verifies only the generated after report exists (-f "$corpus_metrics_path") and
then passes the baseline path through to the CLI without checking it exists (`--before
"$CORPUS_METRICS_BASELINE_INPUT"`).

action.yml[273-281]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The regression diff block checks that the newly generated `after` report exists, but does not validate that `CORPUS_METRICS_BASELINE_INPUT` points to an existing file before calling `codex-plugin-doctor doctor corpus metrics diff`.

### Issue Context
This is inside the new corpus quality gates section in the composite action.

### Fix Focus Areas
- action.yml[273-281]

### Suggested fix
Before building `corpus_metrics_diff_args`, add a baseline existence check, e.g.:

- If `[[ ! -f "$CORPUS_METRICS_BASELINE_INPUT" ]]`, print a clear message (baseline missing/unreadable) and `record_status 2`, then skip diff.
- Otherwise proceed with the current diff invocation.

This keeps misconfiguration failures deterministic and easier to diagnose.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[qodo-code-review]

1. Wrong manifest diff enabled 🐞 Bug ≡ Correctness

In action.yml, CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF is derived solely from
corpus-metrics-baseline being non-empty, even when corpus-metrics-manifest is missing (a
configuration the script rejects and skips). This causes codex-plugin-doctor-action-manifest.json to
advertise corpusMetricsDiff.enabled=true with a path to a report that will not be generated, which
can break downstream steps that rely on the manifest to discover existing artifacts.

Agent Prompt

### Issue description
`CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF` is exported as `true` whenever `corpus-metrics-baseline` is non-empty, even in invalid configurations where the script skips running metrics/diff (e.g., baseline provided without manifest). This makes the generated `codex-plugin-doctor-action-manifest.json` inaccurate.

### Issue Context
The action explicitly treats `corpus-metrics-baseline` (and regression gating) as requiring `corpus-metrics-manifest`, but the manifest “enabled” flags are computed later from inputs alone.

### Fix Focus Areas
- action.yml[262-283]
- action.yml[331-332]
- action.yml[345-376]

### Suggested fix
Introduce explicit booleans (e.g., `corpus_metrics_enabled`, `corpus_metrics_diff_enabled`) initialized to `false` and set them to `true` only in the branches where the corresponding `run_doctor` call is actually scheduled (attempted). Export `CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS*` from these booleans rather than from raw input presence.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[qodo-code-review]

2. Baseline file not validated 🐞 Bug ☼ Reliability

The action runs doctor corpus metrics diff whenever corpus-metrics-baseline is non-empty, but it
does not verify that the baseline path exists before invoking the CLI. This turns a simple
configuration/path problem into a runtime CLI failure with less targeted messaging, and it’s
avoidable in the action script.

Agent Prompt

### Issue description
The regression diff block checks that the newly generated `after` report exists, but does not validate that `CORPUS_METRICS_BASELINE_INPUT` points to an existing file before calling `codex-plugin-doctor doctor corpus metrics diff`.

### Issue Context
This is inside the new corpus quality gates section in the composite action.

### Fix Focus Areas
- action.yml[273-281]

### Suggested fix
Before building `corpus_metrics_diff_args`, add a baseline existence check, e.g.:

- If `[[ ! -f "$CORPUS_METRICS_BASELINE_INPUT" ]]`, print a clear message (baseline missing/unreadable) and `record_status 2`, then skip diff.
- Otherwise proceed with the current diff invocation.

This keeps misconfiguration failures deterministic and easier to diagnose.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 973968527a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Guard against overwriting the baseline metrics report

When a workflow downloads the retained baseline artifact into the same output-dir and sets corpus-metrics-baseline to $report_dir/corpus-metrics.json (the default generated report name), this command overwrites the previous report before the diff is constructed, so line 274 compares the freshly written current report to itself and --fail-on-regression can never detect a regression. Please reject identical resolved before/after paths or write the current metrics report to a separate temporary path before diffing.

Useful? React with 👍 / 👎.