Skip to content

feat: add corpus quality gates to GitHub Action #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Esquetta merged 2 commits into from
Jul 21, 2026
Merged

feat: add corpus quality gates to GitHub Action #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1d8282c
feat: add corpus quality gates to GitHub Action
Esquetta Jul 21, 2026
9739685
docs: explain Action corpus regression gates
Esquetta Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ jobs:
review-bundle-verify: "true"
```

The action writes `codex-plugin-doctor-summary.md`, `codex-plugin-doctor-report.json`, `codex-plugin-doctor-action-manifest.json`, optional `codex-plugin-doctor.sarif`, optional `validation-corpus.json`, optional `output-contract.json`, and optional signed `review-bundle/` files to `codex-plugin-doctor-reports`, appends the Markdown report to the GitHub Actions step summary, uploads the report directory as an artifact, and then returns the real validation exit code. Review bundle generation requires a signing key environment variable such as `CODEX_PLUGIN_DOCTOR_SIGNING_KEY`. For runtime probing, SARIF output, corpus and contract artifacts, review bundle artifacts, installed plugin cache checks, CI policy presets, and pinned release examples, see [GitHub Action Usage](./docs/guides/github-action.md).
The action writes `codex-plugin-doctor-summary.md`, `codex-plugin-doctor-report.json`, `codex-plugin-doctor-action-manifest.json`, optional `codex-plugin-doctor.sarif`, optional validation corpus and quality metrics reports, optional `output-contract.json`, and optional signed `review-bundle/` files to `codex-plugin-doctor-reports`, appends the Markdown report to the GitHub Actions step summary, uploads the report directory as an artifact, and then returns the real validation exit code. Review bundle generation requires a signing key environment variable such as `CODEX_PLUGIN_DOCTOR_SIGNING_KEY`. For runtime probing, SARIF output, corpus quality regression gates, corpus and contract artifacts, review bundle artifacts, installed plugin cache checks, CI policy presets, and pinned release examples, see [GitHub Action Usage](./docs/guides/github-action.md).

To self-test this repository after cloning it:

Expand Down
54 changes: 54 additions & 0 deletions action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,18 @@ inputs:
description: Write bundled validation corpus JSON into the output directory.
required: false
default: "false"
corpus-metrics-manifest:
description: Optional path to a private corpus metrics manifest to evaluate.
required: false
default: ""
corpus-metrics-baseline:
description: Optional path to a previous corpus metrics JSON report for regression comparison.
required: false
default: ""
corpus-metrics-fail-on-regression:
description: Fail when corpus precision or recall decreases, or false-positive share increases.
required: false
default: "false"
contract:
description: Write public output contract JSON into the output directory.
required: false
Expand Down Expand Up @@ -118,6 +130,12 @@ outputs:
validation-corpus-path:
description: Path to the generated validation corpus JSON report when corpus is enabled.
value: ${{ steps.run-doctor.outputs.validation-corpus-path }}
corpus-metrics-path:
description: Path to the generated corpus quality metrics JSON report when configured.
value: ${{ steps.run-doctor.outputs.corpus-metrics-path }}
corpus-metrics-diff-path:
description: Path to the generated corpus quality regression JSON report when a baseline is configured.
value: ${{ steps.run-doctor.outputs.corpus-metrics-diff-path }}
output-contract-path:
description: Path to the generated output contract JSON report when contract is enabled.
value: ${{ steps.run-doctor.outputs.output-contract-path }}
Expand All @@ -141,6 +159,10 @@ runs:
- name: Run Codex Plugin Doctor
id: run-doctor
shell: bash
env:
CORPUS_METRICS_MANIFEST_INPUT: ${{ inputs['corpus-metrics-manifest'] }}
CORPUS_METRICS_BASELINE_INPUT: ${{ inputs['corpus-metrics-baseline'] }}
CORPUS_METRICS_FAIL_ON_REGRESSION_INPUT: ${{ inputs['corpus-metrics-fail-on-regression'] }}
run: |
set -euo pipefail

Expand All @@ -149,6 +171,8 @@ runs:
json_path="$report_dir/codex-plugin-doctor-report.json"
sarif_path="$report_dir/codex-plugin-doctor.sarif"
validation_corpus_path="$report_dir/validation-corpus.json"
corpus_metrics_path="$report_dir/corpus-metrics.json"
corpus_metrics_diff_path="$report_dir/corpus-metrics-diff.json"
output_contract_path="$report_dir/output-contract.json"
action_manifest_path="$report_dir/codex-plugin-doctor-action-manifest.json"
review_bundle_path="$report_dir/${{ inputs['review-bundle-dir'] }}"
Expand Down Expand Up @@ -235,6 +259,28 @@ runs:
run_doctor "validation corpus" doctor corpus --json --output "$validation_corpus_path"
fi

if [[ -z "$CORPUS_METRICS_MANIFEST_INPUT" ]]; then
if [[ -n "$CORPUS_METRICS_BASELINE_INPUT" || "$CORPUS_METRICS_FAIL_ON_REGRESSION_INPUT" == "true" ]]; then
echo "Corpus metrics baseline and regression gating require corpus-metrics-manifest."
record_status 2
fi
elif [[ -z "$CORPUS_METRICS_BASELINE_INPUT" && "$CORPUS_METRICS_FAIL_ON_REGRESSION_INPUT" == "true" ]]; then
echo "corpus-metrics-fail-on-regression requires corpus-metrics-baseline."
record_status 2
else
run_doctor "corpus metrics" doctor corpus metrics --manifest "$CORPUS_METRICS_MANIFEST_INPUT" --json --output "$corpus_metrics_path"

if [[ -n "$CORPUS_METRICS_BASELINE_INPUT" && -f "$corpus_metrics_path" ]]; then

@qodo-code-review qodo-code-review Bot Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

2. Baseline file not validated 🐞 Bug ☼ Reliability

The action runs doctor corpus metrics diff whenever corpus-metrics-baseline is non-empty, but it
does not verify that the baseline path exists before invoking the CLI. This turns a simple
configuration/path problem into a runtime CLI failure with less targeted messaging, and it’s
avoidable in the action script.
Agent Prompt 
### Issue description
The regression diff block checks that the newly generated `after` report exists, but does not validate that `CORPUS_METRICS_BASELINE_INPUT` points to an existing file before calling `codex-plugin-doctor doctor corpus metrics diff`.

### Issue Context
This is inside the new corpus quality gates section in the composite action.

### Fix Focus Areas
- action.yml[273-281]

### Suggested fix
Before building `corpus_metrics_diff_args`, add a baseline existence check, e.g.:

- If `[[ ! -f "$CORPUS_METRICS_BASELINE_INPUT" ]]`, print a clear message (baseline missing/unreadable) and `record_status 2`, then skip diff.
- Otherwise proceed with the current diff invocation.

This keeps misconfiguration failures deterministic and easier to diagnose.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

corpus_metrics_diff_args=(doctor corpus metrics diff --before "$CORPUS_METRICS_BASELINE_INPUT" --after "$corpus_metrics_path" --json --output "$corpus_metrics_diff_path")
Comment on lines +271 to +274

@chatgpt-codex-connector chatgpt-codex-connector Bot Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard against overwriting the baseline metrics report

When a workflow downloads the retained baseline artifact into the same output-dir and sets corpus-metrics-baseline to $report_dir/corpus-metrics.json (the default generated report name), this command overwrites the previous report before the diff is constructed, so line 274 compares the freshly written current report to itself and --fail-on-regression can never detect a regression. Please reject identical resolved before/after paths or write the current metrics report to a separate temporary path before diffing.

Useful? React with 👍 / 👎.


if [[ "$CORPUS_METRICS_FAIL_ON_REGRESSION_INPUT" == "true" ]]; then
corpus_metrics_diff_args+=(--fail-on-regression)
fi

run_doctor "corpus metrics regression" "${corpus_metrics_diff_args[@]}"
fi
fi

if [[ "${{ inputs.contract }}" == "true" ]]; then
run_doctor "output contract" doctor contract --json --output "$output_contract_path"
fi
Expand Down Expand Up @@ -282,13 +328,17 @@ runs:
export CODEX_PLUGIN_DOCTOR_ACTION_MARKDOWN="${{ inputs.markdown }}"
export CODEX_PLUGIN_DOCTOR_ACTION_SARIF="${{ inputs.sarif }}"
export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS="${{ inputs.corpus }}"
export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS="$([[ -n "$CORPUS_METRICS_MANIFEST_INPUT" ]] && echo true || echo false)"

@qodo-code-review qodo-code-review Bot Jul 21, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Wrong manifest diff enabled 🐞 Bug ≡ Correctness

In action.yml, CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF is derived solely from
corpus-metrics-baseline being non-empty, even when corpus-metrics-manifest is missing (a
configuration the script rejects and skips). This causes codex-plugin-doctor-action-manifest.json to
advertise corpusMetricsDiff.enabled=true with a path to a report that will not be generated, which
can break downstream steps that rely on the manifest to discover existing artifacts.
Agent Prompt 
### Issue description
`CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF` is exported as `true` whenever `corpus-metrics-baseline` is non-empty, even in invalid configurations where the script skips running metrics/diff (e.g., baseline provided without manifest). This makes the generated `codex-plugin-doctor-action-manifest.json` inaccurate.

### Issue Context
The action explicitly treats `corpus-metrics-baseline` (and regression gating) as requiring `corpus-metrics-manifest`, but the manifest “enabled” flags are computed later from inputs alone.

### Fix Focus Areas
- action.yml[262-283]
- action.yml[331-332]
- action.yml[345-376]

### Suggested fix
Introduce explicit booleans (e.g., `corpus_metrics_enabled`, `corpus_metrics_diff_enabled`) initialized to `false` and set them to `true` only in the branches where the corresponding `run_doctor` call is actually scheduled (attempted). Export `CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS*` from these booleans rather than from raw input presence.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF="$([[ -n "$CORPUS_METRICS_BASELINE_INPUT" ]] && echo true || echo false)"
export CODEX_PLUGIN_DOCTOR_ACTION_CONTRACT="${{ inputs.contract }}"
export CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE="${{ inputs['review-bundle'] }}"
export CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE_VERIFY="${{ inputs['review-bundle-verify'] }}"
export CODEX_PLUGIN_DOCTOR_ACTION_SUMMARY_PATH="$summary_path"
export CODEX_PLUGIN_DOCTOR_ACTION_JSON_PATH="$json_path"
export CODEX_PLUGIN_DOCTOR_ACTION_SARIF_PATH="$sarif_path"
export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_PATH="$validation_corpus_path"
export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_PATH="$corpus_metrics_path"
export CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF_PATH="$corpus_metrics_diff_path"
export CODEX_PLUGIN_DOCTOR_ACTION_CONTRACT_PATH="$output_contract_path"
export CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE_PATH="$review_bundle_path"
export CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE_VERIFICATION_PATH="$review_bundle_verification_path"
Expand Down Expand Up @@ -318,6 +368,8 @@ runs:
json: report("json", "CODEX_PLUGIN_DOCTOR_ACTION_JSON", "CODEX_PLUGIN_DOCTOR_ACTION_JSON_PATH"),
sarif: report("sarif", "CODEX_PLUGIN_DOCTOR_ACTION_SARIF", "CODEX_PLUGIN_DOCTOR_ACTION_SARIF_PATH"),
corpus: report("corpus", "CODEX_PLUGIN_DOCTOR_ACTION_CORPUS", "CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_PATH"),
corpusMetrics: report("corpusMetrics", "CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS", "CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_PATH"),
corpusMetricsDiff: report("corpusMetricsDiff", "CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF", "CODEX_PLUGIN_DOCTOR_ACTION_CORPUS_METRICS_DIFF_PATH"),
contract: report("contract", "CODEX_PLUGIN_DOCTOR_ACTION_CONTRACT", "CODEX_PLUGIN_DOCTOR_ACTION_CONTRACT_PATH"),
reviewBundle: report("reviewBundle", "CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE", "CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE_PATH"),
reviewBundleVerification: report("reviewBundleVerification", "CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE_VERIFY", "CODEX_PLUGIN_DOCTOR_ACTION_REVIEW_BUNDLE_VERIFICATION_PATH")
Expand All @@ -340,6 +392,8 @@ runs:
echo "json-path=$json_path"
echo "sarif-path=$sarif_path"
echo "validation-corpus-path=$validation_corpus_path"
echo "corpus-metrics-path=$corpus_metrics_path"
echo "corpus-metrics-diff-path=$corpus_metrics_diff_path"
echo "output-contract-path=$output_contract_path"
echo "action-manifest-path=$action_manifest_path"
echo "review-bundle-path=$review_bundle_path"
Expand Down
28 changes: 28 additions & 0 deletions docs/guides/github-action.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,8 @@ The action also exposes these workflow outputs for follow-up steps:
- `json-path`
- `sarif-path`
- `validation-corpus-path`
- `corpus-metrics-path`
- `corpus-metrics-diff-path`
- `output-contract-path`
- `action-manifest-path`
- `review-bundle-path`
Expand Down Expand Up @@ -130,6 +132,32 @@ The CLI can produce badge output for release notes, README automation, or a stat

`--badge-json` follows the Shields endpoint schema with `schemaVersion`, `label`, `message`, and `color`. `--badge-markdown` emits a static shields.io Markdown image link.

## Corpus Quality Gates

Use a private corpus metrics manifest to measure reviewed precision, recall, and false-positive share in CI. The action writes only the public-safe metrics report into its artifact directory; snapshots, manifest contents, local paths, and review notes are not copied.

```yaml
- uses: Esquetta/CodexPluginDoctor@v1.49.0
with:
version: "1.49.0"
path: .
corpus-metrics-manifest: ../private-corpus/metrics.json
```

This writes `corpus-metrics.json`. To compare the result with a retained report and fail the job on regression:

```yaml
- uses: Esquetta/CodexPluginDoctor@v1.49.0
with:
version: "1.49.0"
path: .
corpus-metrics-manifest: ../private-corpus/metrics.json
corpus-metrics-baseline: .doctor-baselines/corpus-metrics.json
corpus-metrics-fail-on-regression: "true"
```

The comparison writes `corpus-metrics-diff.json`. Reports must have the same `corpusDigest`; changed corpus composition is rejected as non-comparable rather than reported as a validator regression. `corpus-metrics-fail-on-regression` requires both a manifest and baseline.

## History Artifacts

Use history output when a workflow should preserve validation trend data between runs.
Expand Down
15 changes: 15 additions & 0 deletions tests/action-metadata.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ describe("GitHub Action metadata", () => {
expect(actionMetadata).toContain("json-path:");
expect(actionMetadata).toContain("sarif-path:");
expect(actionMetadata).toContain("validation-corpus-path:");
expect(actionMetadata).toContain("corpus-metrics-path:");
expect(actionMetadata).toContain("corpus-metrics-diff-path:");
expect(actionMetadata).toContain("output-contract-path:");
expect(actionMetadata).toContain("action-manifest-path:");
expect(actionMetadata).toContain("review-bundle-path:");
Expand All @@ -39,6 +41,9 @@ describe("GitHub Action metadata", () => {
expect(actionMetadata).toContain("json:");
expect(actionMetadata).toContain("markdown:");
expect(actionMetadata).toContain("corpus:");
expect(actionMetadata).toContain("corpus-metrics-manifest:");
expect(actionMetadata).toContain("corpus-metrics-baseline:");
expect(actionMetadata).toContain("corpus-metrics-fail-on-regression:");
expect(actionMetadata).toContain("contract:");
expect(actionMetadata).toContain("review-bundle:");
expect(actionMetadata).toContain("review-bundle-dir:");
Expand All @@ -58,6 +63,9 @@ describe("GitHub Action metadata", () => {
expect(actionMetadata).toContain("doctor.github.action.manifest");
expect(actionMetadata).toContain('doctor_version="$(codex-plugin-doctor --version)"');
expect(actionMetadata).toContain('run_doctor "validation corpus" doctor corpus --json --output "$validation_corpus_path"');
expect(actionMetadata).toContain('run_doctor "corpus metrics" doctor corpus metrics --manifest "$CORPUS_METRICS_MANIFEST_INPUT" --json --output "$corpus_metrics_path"');
expect(actionMetadata).toContain('corpus_metrics_diff_args=(doctor corpus metrics diff --before "$CORPUS_METRICS_BASELINE_INPUT" --after "$corpus_metrics_path" --json --output "$corpus_metrics_diff_path")');
expect(actionMetadata).toContain("corpus_metrics_diff_args+=(--fail-on-regression)");
expect(actionMetadata).toContain('run_doctor "output contract" doctor contract --json --output "$output_contract_path"');
expect(actionMetadata).toContain('review_bundle_args=(doctor review-bundle "${{ inputs.path }}" --output "$review_bundle_path" --sign-key-env "$signing_key_env")');
expect(actionMetadata).toContain('doctor review-bundle verify "$review_bundle_path" --target "${{ inputs.path }}" --sign-key-env "$signing_key_env" --json --output "$review_bundle_verification_path"');
Expand All @@ -66,6 +74,8 @@ describe("GitHub Action metadata", () => {
expect(actionMetadata).toContain('cat "$summary_path" >> "$GITHUB_STEP_SUMMARY"');
expect(actionMetadata).toContain('echo "status=$status"');
expect(actionMetadata).toContain('echo "validation-corpus-path=$validation_corpus_path"');
expect(actionMetadata).toContain('echo "corpus-metrics-path=$corpus_metrics_path"');
expect(actionMetadata).toContain('echo "corpus-metrics-diff-path=$corpus_metrics_diff_path"');
expect(actionMetadata).toContain('echo "output-contract-path=$output_contract_path"');
expect(actionMetadata).toContain('echo "action-manifest-path=$action_manifest_path"');
expect(actionMetadata).toContain('echo "review-bundle-path=$review_bundle_path"');
Expand Down Expand Up @@ -101,6 +111,11 @@ describe("GitHub Action metadata", () => {
expect(actionUsage).toContain("corpus:");
expect(actionUsage).toContain("contract:");
expect(actionUsage).toContain("validation-corpus.json");
expect(actionUsage).toContain("corpus-metrics.json");
expect(actionUsage).toContain("corpus-metrics-diff.json");
expect(actionUsage).toContain("corpus-metrics-manifest:");
expect(actionUsage).toContain("corpus-metrics-baseline:");
expect(actionUsage).toContain('corpus-metrics-fail-on-regression: "true"');
expect(actionUsage).toContain("output-contract.json");
expect(actionUsage).toContain("codex-plugin-doctor-action-manifest.json");
expect(actionUsage).toContain("action-manifest-path");
Expand Down