[P1.1] PROOF9: give the 6 unrunnable gates real runners or an explicit UNVERIFIABLE state (not perpetual fail)

[frankbria]

Severity: high | Category: unfinished | Phase: P1.1

Problem

Verified: _GATE_TO_CORE = {UNIT: pytest, CONTRACT: pytest, SEC: ruff}; every other gate returns (False, "no automated runner — cannot verify"). Since OBLIGATION_MAP assigns E2E/VISUAL/A11Y/PERF/DEMO to 5 of 7 glitch types, most captured glitches can never be SATISFIED. CONTRACT is indistinguishable from UNIT; SEC is a linter. The flagship "9-gate evidence-based" PROVE step is really pytest+ruff.

Evidence

codeframe/core/proof/runner.py:68, :83

Acceptance criteria

• Remaining gates are wired to real runners (playwright for E2E, etc.), OR unrunnable gates return an explicit UNVERIFIABLE/waivable state distinct from "ran and failed".
• The /proof UI and run results distinguish "cannot verify" from "failed"; the stale runner docstring is corrected.

Dependencies

None

---

_{Filed from the SaaS launch-readiness audit. Atomic: one developer, one session. Work order: strictly P0.1 → P3.12 (no forward dependencies).}

[coderabbitai]

Coding Plan

Summary

Introduce an explicit UNVERIFIABLE gate-result state in the PROOF9 core rather than building six real runners, since no relevant tooling is installed and the ticket permits this alternative.

Change the single chokepoint (runner.py::_run_gate) to return this distinct state for gates lacking automated tooling, correct its stale docstring, and update run_proof so unverifiable results neither auto-satisfy nor hard-fail.

Persist the state via a new optional Evidence field (schema-compatible) and thread it through the API, CLI, and TUI backend consumers.

Propagate the state through the web UI types, run hook, and proof components/page so /proof visibly distinguishes "cannot verify" from "failed".

Design Choices

Design Choice 1: Should we wire real test runners for the 6 gates, or introduce an explicit UNVERIFIABLE state?

Options Considered:

1. Wire real runners (Playwright for E2E, axe-core for A11Y, lighthouse for PERF, etc.).
2. Introduce an explicit UNVERIFIABLE state for gates lacking automated tooling, distinct from FAIL and eligible for waiver.

Chosen Option: 2

Rationale: The acceptance criteria offer both as valid ("OR"). Exploration confirmed no axe-core/lighthouse/percy/playwright-a11y tooling is configured anywhere; CONTRACT has no distinct test corpus (empty tests/contract/), and E2E Playwright suites exist but aren't structured for per-requirement gate verification. No tooling is installed and building six runners is far beyond an "atomic, one-session" ticket; the UNVERIFIABLE state directly satisfies the acceptance criteria and unblocks requirements from perpetual failure.

Design Choice 2: How should UNVERIFIABLE be represented on the persisted `Evidence` record (currently `satisfied: bool`)?

Options Considered:

1. Add a nullable status/result string field to Evidence and keep satisfied for backward compatibility (satisfied=False when unverifiable).
2. Convert satisfied to a tri-state string enum everywhere.

Chosen Option: 1

Rationale: The ledger serializes Evidence.satisfied as a boolean; a third state cannot be a boolean. Adding a new optional field is the least disruptive to the existing ledger schema and downstream boolean consumers while still carrying the distinct state.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Introduce the UNVERIFIABLE state in the PROOF9 core

Establish a canonical tri-state gate result in the core models and thread it through the runner so that gates without automated tooling produce a distinct, non-failing UNVERIFIABLE outcome that is persisted as evidence and rolled up sensibly. This phase is self-contained and testable independently of any UI.

Task 1: Define a canonical gate-result state

Introduce an explicit tri-state concept and document allowed obligation statuses.

• In codeframe/core/proof/models.py, add a small GateResult/GateOutcome string enum with PASSED, FAILED, and UNVERIFIABLE members, mirroring the existing Gate/ReqStatus enum style.
• Update the Obligation.status comment/allowed-value documentation to include unverifiable alongside pending/satisfied/failed.
• Add an optional status (or result) field to the Evidence dataclass to carry the tri-state, keeping the existing satisfied: bool field for backward compatibility (set satisfied=False when unverifiable).

Task 2: Return a tri-state from the gate executor and fix the docstring

Make _run_gate distinguish "no automated runner" from a genuine failure.

• In codeframe/core/proof/runner.py, change _run_gate (the _GATE_TO_CORE lookup at ~line 68 and the "cannot verify" return at ~line 83) so the unmapped-gate branch returns the UNVERIFIABLE outcome instead of False.
• Correct the misleading docstring (it currently claims "Returns (True, "") for gates without automated tooling" while the code returns False) to accurately describe the tri-state behavior.
• Update the return type to convey the outcome (either a tri-state enum + output string, or an enum-based tuple) rather than a plain (bool, str).

Task 3: Propagate UNVERIFIABLE through run_proof rollups

Ensure unverifiable obligations neither auto-satisfy nor hard-fail.

• In codeframe/core/proof/runner.py::run_proof, replace the binary obl.status = "satisfied" if passed else "failed" assignment (~line 196) with a three-way mapping to satisfied/failed/unverifiable.
• Persist evidence with the tri-state via the updated attach_evidence call so unverifiable results are distinguishable in the ledger.
• Update the requirement-level rollup (~lines 203-206): an unverifiable obligation must prevent the requirement from auto-transitioning to SATISFIED (leaving it OPEN/waivable), but must not be counted as a hard failure.
• Update the run-level overall_passed computation (~lines 210-219) so unverifiable outcomes are excluded from the pass/fail tally — the run should not fail solely because a gate could not be verified.
• Adjust the run_proof return type so callers can observe the tri-state (extend the list[tuple[Gate, bool]] to carry the outcome).

Task 4: Update evidence persistence for the tri-state

Carry the distinct state into the evidence layer.

• In codeframe/core/proof/evidence.py, extend attach_evidence to accept/record the tri-state outcome onto the new Evidence field.
• Update codeframe/core/proof/ledger.py serialization helpers so the new Evidence field round-trips (defaulting gracefully for existing rows so old records remain readable).
• Confirm check_obligation_satisfied continues to treat only truly satisfied evidence as satisfying (unverifiable is not satisfied).

🤖 Prompt for AI agents

Implement a tri-state gate-result concept in the PROOF9 core
(`codeframe/core/proof/`) and thread it through the runner and evidence
persistence.

1. In `models.py`: add a `GateResult`/`GateOutcome` string enum with `PASSED`,
`FAILED`, `UNVERIFIABLE` members styled like the existing `Gate`/`ReqStatus`
enums. Update the `Obligation.status` documentation/allowed values to include
`unverifiable`. Add an optional `status`/`result` field to the `Evidence`
dataclass to carry the tri-state while keeping `satisfied: bool` for backward
compatibility (set `satisfied=False` when unverifiable).

2. In `runner.py::_run_gate`: change the `_GATE_TO_CORE` lookup (~line 68) and
the "cannot verify" return (~line 83) so unmapped gates return `UNVERIFIABLE`
instead of `False`. Fix the stale docstring claiming gates without tooling
return `(True, "")`. Update the return type to convey the tri-state outcome
rather than a plain `(bool, str)`.

3. In `runner.py::run_proof`: replace the binary `obl.status = "satisfied" if
passed else "failed"` (~line 196) with a three-way mapping including
`unverifiable`. Persist evidence with the tri-state via the updated
`attach_evidence` call. Update the requirement-level rollup (~lines 203-206) so
an unverifiable obligation prevents auto-transition to `SATISFIED` (stays
OPEN/waivable) without counting as a hard failure. Update `overall_passed`
(~lines 210-219) to exclude unverifiable outcomes from the pass/fail tally.
Extend the `run_proof` return type (`list[tuple[Gate, bool]]`) to carry the
tri-state outcome.

4. In `evidence.py`: extend `attach_evidence` to accept/record the tri-state
outcome on the new `Evidence` field.

5. In `ledger.py`: update serialization helpers so the new `Evidence` field
round-trips, defaulting gracefully for existing rows so old records remain
readable. Verify `check_obligation_satisfied` still treats only truly satisfied
evidence as satisfying (unverifiable is not satisfied).

Phase 2: Surface UNVERIFIABLE across backend consumers (API, CLI, TUI)

Thread the new state through the three Python-side presentation layers so "cannot verify" is visibly distinct from "failed".

Task 1: Extend the REST API responses

Expose the tri-state to API clients.

• In codeframe/ui/routers/proof_v2.py, update the run-result serialization in run_proof_endpoint so each gate result carries the tri-state (e.g., a status string) instead of only a satisfied boolean.
• Update RunProofResponse/RunStatusResponse, ObligationOut, and EvidenceResponse/EvidenceWithContentResponse models to represent the new state.
• Ensure the overall passed aggregation logic in the endpoints mirrors the runner's rule (unverifiable neither passes nor fails), and keep _count_by_status consistent if any requirement-level count is affected.

Task 2: Add a third branch to the CLI rendering

Make cf proof run/show display "cannot verify" distinctly.

• In codeframe/cli/proof_commands.py, update the run command's PASS/FAIL ternary to add a yellow UNVERIFIABLE branch and track it separately from failures, choosing exit-code semantics so an unverifiable-only run does not exit non-zero as a failure.
• Update the show command's evidence rendering (currently a boolean satisfied ternary) and obligation display to reflect the tri-state.
• Adjust summary messaging so the user is told which gates could not be verified (and can be waived) rather than reported as failed.

Task 3: Add a TUI icon for UNVERIFIABLE

Give the dashboard a distinct glyph.

• In codeframe/tui/app.py, add an explicit "unverifiable" entry (e.g., ❓) to _OBLIGATION_ICONS so it no longer silently falls back to the pending hourglass.

🤖 Prompt for AI agents

Thread the tri-state gate result from Phase 1 through the API, CLI, and TUI
backend consumers so "cannot verify" is visibly distinct from "failed".

1. In `codeframe/ui/routers/proof_v2.py`: update the run-result serialization in
`run_proof_endpoint` so each gate result carries the tri-state (e.g., a `status`
string) instead of only a `satisfied` boolean. Update
`RunProofResponse`/`RunStatusResponse`, `ObligationOut`, and
`EvidenceResponse`/`EvidenceWithContentResponse` models to represent the new
state. Ensure the overall `passed` aggregation mirrors the runner's rule
(unverifiable neither passes nor fails), and keep `_count_by_status` consistent
if any requirement-level count is affected.

2. In `codeframe/cli/proof_commands.py`: update the `run` command's PASS/FAIL
ternary to add a yellow `UNVERIFIABLE` branch tracked separately from failures,
with exit-code semantics such that an unverifiable-only run does not exit
non-zero as a failure. Update the `show` command's evidence rendering (currently
a boolean `satisfied` ternary) and obligation display to reflect the tri-state.
Adjust summary messaging to tell the user which gates could not be verified (and
can be waived) rather than reporting them as failed.

3. In `codeframe/tui/app.py`: add an explicit `"unverifiable"` entry (e.g., `❓`)
to `_OBLIGATION_ICONS` so it no longer silently falls back to the pending
hourglass.

Phase 3: Surface UNVERIFIABLE in the web UI

Update the frontend types, run-polling hook, and proof components/page so the /proof experience clearly distinguishes "cannot verify" from "failed", satisfying the acceptance criterion for the UI.

Task 1: Extend frontend types

Model the new state in TypeScript.

• In web-ui/src/types/index.ts, add 'unverifiable' to GateRunStatus and extend the gate-result shapes on RunProofResponse.results/RunStatusResponse.results and ProofEvidence to carry the tri-state (either a status string or a nullable/optional field) rather than only satisfied: boolean.

Task 2: Map the tri-state in the run hook

Translate API results into the UI status.

• In web-ui/src/hooks/useProofRun.ts, replace the binary item.satisfied ? 'passed' : 'failed' mapping with a three-way branch that yields 'unverifiable' for the new state, and represent an overall run that only lacks verification distinctly from a failing run.

Task 3: Render the new state in proof components

Give UNVERIFIABLE a consistent visual treatment.

• Add a new badge variant (e.g., amber/warning) in web-ui/src/components/ui/badge.tsx.
• Update web-ui/src/components/proof/GateRunPanel.tsx (the Record<GateRunStatus, ...> variant map) and GateRunBanner.tsx (add a third banner branch for gates that couldn't be verified) to consume the new state.
• Update GateEvidencePanel.tsx so the per-evidence pass/fail rendering handles the tri-state.

Task 4: Update the /proof page presentation

Reflect the state in the main page and legends.

• In web-ui/src/app/proof/page.tsx, and in ProofStatusBadge.tsx/ProofStatusWidget.tsx, add handling for the new state where obligation/gate-level results are shown, including the status legend and the "Requirement Lifecycle" help section, using the new amber styling.
• Note: if the requirement-level ProofReqStatus union is intentionally left unchanged (this ticket adds a gate/obligation-level state, not a requirement lifecycle state), only the gate-result and evidence displays need the new value; keep requirement-status handling as-is unless a requirement-level state is explicitly desired.

🤖 Prompt for AI agents

Propagate the tri-state gate result from the backend into the web UI so `/proof`
visibly distinguishes "cannot verify" from "failed".

1. In `web-ui/src/types/index.ts`: add `'unverifiable'` to `GateRunStatus` and
extend the gate-result shapes on
`RunProofResponse.results`/`RunStatusResponse.results` and `ProofEvidence` to
carry the tri-state (a status string or nullable/optional field) rather than
only `satisfied: boolean`.

2. In `web-ui/src/hooks/useProofRun.ts`: replace the binary `item.satisfied ?
'passed' : 'failed'` mapping with a three-way branch yielding `'unverifiable'`
for the new state, and represent an overall run that only lacks verification
distinctly from a failing run.

3. Add a new amber/warning badge variant in
`web-ui/src/components/ui/badge.tsx`. Update
`web-ui/src/components/proof/GateRunPanel.tsx` (the `Record<GateRunStatus, ...>`
variant map) and `GateRunBanner.tsx` (add a third banner branch for gates that
couldn't be verified) to consume the new state. Update `GateEvidencePanel.tsx`
so the per-evidence pass/fail rendering handles the tri-state.

4. In `web-ui/src/app/proof/page.tsx`, `ProofStatusBadge.tsx`, and
`ProofStatusWidget.tsx`: add handling for the new state wherever
obligation/gate-level results are shown, including the status legend and the
"Requirement Lifecycle" help section, using the amber styling. Note: if the
requirement-level `ProofReqStatus` union is intentionally left unchanged (since
this ticket adds a gate/obligation-level state, not a requirement lifecycle
state), only the gate-result and evidence displays need the new value — keep
requirement-status handling as-is unless a requirement-level state is explicitly
desired.

Research

The PROOF9 subsystem lives under codeframe/core/proof/: models.py (data models/enums), runner.py (gate execution + rollup), evidence.py (evidence persistence), ledger.py (SQLite serialization), and obligations.py (glitch→gate map).

Gate results originate in runner.py::_run_gate, which collapses everything into a boolean and hard-codes False for gates missing from _GATE_TO_CORE. That boolean flows to three backend consumers — the REST API (codeframe/ui/routers/proof_v2.py), the CLI (codeframe/cli/proof_commands.py), and the TUI (codeframe/tui/app.py) — and to the web UI (web-ui/src/types/index.ts, hooks/useProofRun.ts, and components/proof/*, app/proof/page.tsx).

No axe-core/lighthouse/playwright-a11y tooling exists in the repo, so real automated runners for the 6 gates are not currently feasible; the ticket explicitly allows an UNVERIFIABLE state as the alternative.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement a tri-state gate-result concept in the PROOF9 core
(`codeframe/core/proof/`) and thread it through the runner and evidence
persistence.

1. In `models.py`: add a `GateResult`/`GateOutcome` string enum with `PASSED`,
`FAILED`, `UNVERIFIABLE` members styled like the existing `Gate`/`ReqStatus`
enums. Update the `Obligation.status` documentation/allowed values to include
`unverifiable`. Add an optional `status`/`result` field to the `Evidence`
dataclass to carry the tri-state while keeping `satisfied: bool` for backward
compatibility (set `satisfied=False` when unverifiable).

2. In `runner.py::_run_gate`: change the `_GATE_TO_CORE` lookup (~line 68) and
the "cannot verify" return (~line 83) so unmapped gates return `UNVERIFIABLE`
instead of `False`. Fix the stale docstring claiming gates without tooling
return `(True, "")`. Update the return type to convey the tri-state outcome
rather than a plain `(bool, str)`.

3. In `runner.py::run_proof`: replace the binary `obl.status = "satisfied" if
passed else "failed"` (~line 196) with a three-way mapping including
`unverifiable`. Persist evidence with the tri-state via the updated
`attach_evidence` call. Update the requirement-level rollup (~lines 203-206) so
an unverifiable obligation prevents auto-transition to `SATISFIED` (stays
OPEN/waivable) without counting as a hard failure. Update `overall_passed`
(~lines 210-219) to exclude unverifiable outcomes from the pass/fail tally.
Extend the `run_proof` return type (`list[tuple[Gate, bool]]`) to carry the
tri-state outcome.

4. In `evidence.py`: extend `attach_evidence` to accept/record the tri-state
outcome on the new `Evidence` field.

5. In `ledger.py`: update serialization helpers so the new `Evidence` field
round-trips, defaulting gracefully for existing rows so old records remain
readable. Verify `check_obligation_satisfied` still treats only truly satisfied
evidence as satisfying (unverifiable is not satisfied).
===============================================================================

Task: 2

Thread the tri-state gate result from Phase 1 through the API, CLI, and TUI
backend consumers so "cannot verify" is visibly distinct from "failed".

1. In `codeframe/ui/routers/proof_v2.py`: update the run-result serialization in
`run_proof_endpoint` so each gate result carries the tri-state (e.g., a `status`
string) instead of only a `satisfied` boolean. Update
`RunProofResponse`/`RunStatusResponse`, `ObligationOut`, and
`EvidenceResponse`/`EvidenceWithContentResponse` models to represent the new
state. Ensure the overall `passed` aggregation mirrors the runner's rule
(unverifiable neither passes nor fails), and keep `_count_by_status` consistent
if any requirement-level count is affected.

2. In `codeframe/cli/proof_commands.py`: update the `run` command's PASS/FAIL
ternary to add a yellow `UNVERIFIABLE` branch tracked separately from failures,
with exit-code semantics such that an unverifiable-only run does not exit
non-zero as a failure. Update the `show` command's evidence rendering (currently
a boolean `satisfied` ternary) and obligation display to reflect the tri-state.
Adjust summary messaging to tell the user which gates could not be verified (and
can be waived) rather than reporting them as failed.

3. In `codeframe/tui/app.py`: add an explicit `"unverifiable"` entry (e.g., `❓`)
to `_OBLIGATION_ICONS` so it no longer silently falls back to the pending
hourglass.
===============================================================================

Task: 3

Propagate the tri-state gate result from the backend into the web UI so `/proof`
visibly distinguishes "cannot verify" from "failed".

1. In `web-ui/src/types/index.ts`: add `'unverifiable'` to `GateRunStatus` and
extend the gate-result shapes on
`RunProofResponse.results`/`RunStatusResponse.results` and `ProofEvidence` to
carry the tri-state (a status string or nullable/optional field) rather than
only `satisfied: boolean`.

2. In `web-ui/src/hooks/useProofRun.ts`: replace the binary `item.satisfied ?
'passed' : 'failed'` mapping with a three-way branch yielding `'unverifiable'`
for the new state, and represent an overall run that only lacks verification
distinctly from a failing run.

3. Add a new amber/warning badge variant in
`web-ui/src/components/ui/badge.tsx`. Update
`web-ui/src/components/proof/GateRunPanel.tsx` (the `Record<GateRunStatus, ...>`
variant map) and `GateRunBanner.tsx` (add a third banner branch for gates that
couldn't be verified) to consume the new state. Update `GateEvidencePanel.tsx`
so the per-evidence pass/fail rendering handles the tri-state.

4. In `web-ui/src/app/proof/page.tsx`, `ProofStatusBadge.tsx`, and
`ProofStatusWidget.tsx`: add handling for the new state wherever
obligation/gate-level results are shown, including the status legend and the
"Requirement Lifecycle" help section, using the amber styling. Note: if the
requirement-level `ProofReqStatus` union is intentionally left unchanged (since
this ticket adds a gate/obligation-level state, not a requirement lifecycle
state), only the gate-result and evidence displays need the new value — keep
requirement-status handling as-is unless a requirement-level state is explicitly
desired.

💡 Iterate on the plan with: `@coderabbitai` <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

🔗 Related PRs

#386 - [Phase 2.5-F] Harden verification gates and add batch-level QA [merged]
#462 - test(cli): integration tests for all cf proof commands (#455) [merged]
#496 - feat(web-ui): link failed tasks to PROOF9 gate failure for root cause (#474) [merged]
#574 - feat(web-ui): Run Gates button + live gate progress for PROOF9 (#566) [merged]
#576 - feat(web-ui): Glitch Capture entry point and form for PROOF9 (#568) [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!