Couple Diagnostic.version into the regression CI coupling gate

changelog/770.feature.md

@@ -0,0 +1,7 @@
+The CI coupling gate now couples each regression baseline to its diagnostic's `Diagnostic.version`.
+A test case fails the gate when the diagnostic's in-code version no longer matches the version recorded
+when its baseline was minted, so an execution-affecting change can no longer slip through as a green replay
+against a stale committed bundle.
+The new author-declared `diagnostic_version` field lives in the test case `manifest.json` (schema version 2),
+and the new `ref test-cases migrate-manifests` command backfills it across existing baselines.
+A legitimate revert is still permitted: lower the diagnostic version and re-mint the baseline.

docs/background/regression-baselines.md

@@ -27,14 +27,22 @@ A baseline is split into two layers with very different trust and portability pr
 
 The two layers are bound by a **`manifest.json`** alongside the bundle, which records:
 
-- `schema` — integer schema version for the manifest format itself (currently `1`).
+- `schema` — integer schema version for the manifest format itself (currently `2`).
   The loader rejects manifests whose `schema` does not match the current `SCHEMA_VERSION`,
   so format migrations are detected immediately rather than silently misread.
 - `test_case_version` — a monotonic, author-bumped integer that *authorises* a new baseline.
+- `diagnostic_version` — the author-declared `Diagnostic.version` captured at mint time,
+  coupling the baseline to the diagnostic code that produced it.
 - `committed` — sha256 digests of the committed JSON artefacts, over the exact placeholder-substituted bytes on disk.
 - `native` — sha256 + size of each curated native file (empty `{}` until minted).
 - `catalog_hash` — the hash of the test case's input `catalog.yaml`, coupling the baseline to the inputs that produced it.
 
+These give the manifest **two orthogonal version axes**:
+`test_case_version` is per-case and mint-authored (it authorises a new committed baseline),
+while `diagnostic_version` is per-diagnostic and author-authored (it declares that the
+diagnostic's *results* changed). `test_case_version` remains monotonic, and `diagnostic_version`
+may move backwards only for an authorised revert. The gate consults each for a different purpose.
+
 !!! note "An empty native set is a permanent, valid state"
     Fork contributors cannot mint
     (minting needs object-store write credentials that never run on untrusted pull-request code).
@@ -67,6 +75,7 @@ flowchart LR
 | `sync` | public read | Fetch the native blobs referenced by the manifest into the local store cache. |
 | `replay` | public read | Materialise the native baseline into a slot, re-run only `build_execution_result`, and tolerantly compare to the committed bundle. |
 | `check-store` | write | Preflight the writable native store (credentials + bucket) without uploading anything. Run this before a slow `mint` to confirm credentials are correct. |
+| `migrate-manifests` | none | One-shot, idempotent maintenance command that rewrites every committed `manifest.json` at the current `SCHEMA_VERSION`, stamping each case's `diagnostic_version` from the diagnostic's in-code `Diagnostic.version`. |
 
 ### Output slots
 
@@ -172,7 +181,7 @@ because an empty native set is legitimate, a missing native baseline downgrades
 - **`skip`** — nothing relevant to this case changed, or the case is not under regression management.
 - **`replay`** — cheap, anonymous re-check against the cached native baseline (only when native blobs exist). This also verifies a `test_case_version` bump that ships native blobs.
 - **`execute`** — full end-to-end re-run with tolerant compare; required when `test_case_version` was bumped to authorise a new baseline *and* no native baseline exists to replay against.
-- **`fail`** — an unauthorised or unverifiable change (committed bundle edited without a version bump, version moved backwards, bundle drifted from its manifest, or catalog changed without regenerating the baseline).
+- **`fail`** — an unauthorised or unverifiable change: the committed bundle or catalog changed without a version bump, a version moved backwards, the bundle drifted from its manifest, or the in-code `Diagnostic.version` no longer matches the recorded `diagnostic_version`.
 
 ### Decision flow
 
@@ -190,17 +199,24 @@ flowchart TD
     committedOk -->|yes| catalogOk{catalog.yaml<br/>matches manifest<br/>catalog_hash?}
     catalogOk -->|no| failCatalog[["FAIL<br/>inputs changed without<br/>regenerating baseline"]]
 
-    catalogOk -->|yes| seeding{newly added<br/>this branch?}
+    catalogOk -->|yes| staleVersion{code Diagnostic.version<br/>vs baseline<br/>diagnostic_version}
+    staleVersion -->|"code &gt; baseline"| failStale[["FAIL<br/>stale baseline<br/>re-mint required"]]
+    staleVersion -->|"code &lt; baseline"| failDowngrade[["FAIL<br/>illegal downgrade<br/>lower code AND re-mint"]]
+
+    staleVersion -->|equal| seeding{newly added<br/>this branch?}
     seeding -->|yes| seedNative{native<br/>non-empty?}
     seedNative -->|yes| replaySeed[["REPLAY<br/>seed against committed"]]
     seedNative -->|no| skipSeed[["SKIP<br/>committed-only baseline"]]
 
-    seeding -->|no| versionCmp{version vs base}
+    seeding -->|no| versionCmp{test_case_version<br/>vs base}
     versionCmp -->|decreased| failVersion[["FAIL<br/>version not monotonic"]]
-    versionCmp -->|bumped| bumpNative{native<br/>present?}
+    versionCmp -->|"unchanged or bumped"| diagVersionCmp{diagnostic_version<br/>decreased vs base?}
+    diagVersionCmp -->|"yes, committed unchanged"| failBareDowngrade[["FAIL<br/>bare downgrade<br/>re-mint required"]]
+    diagVersionCmp -->|"no, or committed re-minted"| versionBumped{test_case_version<br/>bumped?}
+    versionBumped -->|yes| bumpNative{native<br/>present?}
     bumpNative -->|yes| replayBump[["REPLAY<br/>verify new baseline<br/>reproduces from native"]]
     bumpNative -->|no| execute[["EXECUTE<br/>full re-run<br/>(no native to replay)"]]
-    versionCmp -->|unchanged| committedChanged{committed<br/>changed?}
+    versionBumped -->|no| committedChanged{committed<br/>changed?}
 
     committedChanged -->|yes| failUnauthorised[["FAIL<br/>baseline changed without<br/>version bump"]]
     committedChanged -->|no| nativeChanged{native<br/>changed?}
@@ -224,6 +240,16 @@ and maps the changed-file list onto each case:
 - **`extraction_changed`** is coarse and errs toward `replay` (cheap, credential-free):
   any change under the diagnostic's provider package,
   or under the core regression package, counts for every case in that provider.
+  It is a *path-based* heuristic, so it cannot tell an execution change from an extraction change.
+- **`diagnostic_version`** is the *author-declared* execution-change signal: when a diagnostic's
+  results change the author bumps `Diagnostic.version`, and the gate fails any baseline whose
+  recorded `diagnostic_version` no longer matches the in-code value until it is re-minted.
+  This closes the **replay blind-spot** that `extraction_changed` alone leaves open:
+  an execution-affecting change with no extraction-path edit (the #671 ECS case) would otherwise
+  route to a green replay against a stale bundle.
+  A `diagnostic_version` decrease is permitted only as an **authorised revert**: the author lowers
+  the code version *and* re-mints, so the committed bundle changes and the gate recognises the
+  revert structurally rather than via a flag.
 - **`committed_integrity_ok`** recomputes the committed digests from the working tree
   and compares them to `manifest.committed`.
 - **`catalog_integrity_ok`** recomputes the input catalog hash

packages/climate-ref-core/src/climate_ref_core/regression/gate.py

@@ -1,22 +1,29 @@
 """
 CI coupling gate for test case regression bundles.
 
-For each test case, CI must decide *how* to verify the regression baseline in a pull request,
+For each test case, CI decides how to verify the regression baseline in a pull request,
 based on what changed relative to the base branch:
 
-- **EXECUTE** — re-run the diagnostic end-to-end and compare the regenerated committed
-    bundle to the in-repo copy (tolerant compare).
-    Required when the author has bumped ``test_case_version`` to authorise a new baseline
-    but no native baseline exists to replay against.
-- **REPLAY** — materialise the cached native baseline and re-run only
-    ``build_execution_result`` against it, comparing to the committed bundle.
-    Cheap and anonymous; used when extraction code changed but the committed bundle did
-    not, to seed a newly added manifest, and to verify a ``test_case_version`` bump that ships native blobs
-    — but only when native blobs exist to replay.
-- **FAIL** — the committed bundle (or its input catalog) changed without a
-    ``test_case_version`` bump (an unauthorised baseline change), or the version moved backwards.
-- **SKIP** — nothing relevant to this test case changed, or the case is not yet under
-    regression-baseline management.
+- **EXECUTE**: re-run the diagnostic end-to-end and compare the regenerated bundle
+    to the committed copy.
+    Used when ``test_case_version`` was bumped but no native baseline exists to replay.
+- **REPLAY**: re-run ``build_execution_result`` against the cached native baseline
+    and compare to the committed bundle.
+    Used when extraction code changed but the committed bundle did not,
+    to seed a newly added manifest,
+    and to verify a ``test_case_version`` bump that ships native blobs.
+- **FAIL**: the committed bundle or its catalog changed without a ``test_case_version`` bump,
+    a version moved backwards,
+    or the in-code ``Diagnostic.version`` no longer matches the recorded ``diagnostic_version``.
+- **SKIP**: nothing relevant changed, or the case is not under regression-baseline management.
+
+The gate tracks two orthogonal version axes:
+``test_case_version`` (per-case, mint-authored)
+and ``diagnostic_version`` (per-diagnostic, author-authored).
+``test_case_version`` is monotonic.
+``diagnostic_version`` may decrease only for an authorised revert:
+the code version is lowered and the baseline re-minted so the committed bundle changes,
+which the gate recognises structurally rather than via a flag.
 
 See [Regression Baselines](https://climate-ref.readthedocs.io/en/latest/background/regression-baselines/)
 for more information about the motivation and workflow for regression baselines.
@@ -59,13 +66,14 @@ class GateDecision:
     """A human-readable explanation, surfaced in CI logs."""
 
 
-def decide_coupling(  # noqa: PLR0911, PLR0912
+def decide_coupling(  # noqa: PLR0911, PLR0912, PLR0913
     manifest: Manifest | None,
     base_manifest: Manifest | None,
     *,
     extraction_changed: bool = False,
     committed_integrity_ok: bool = True,
     catalog_integrity_ok: bool = True,
+    code_diagnostic_version: int | None = None,
 ) -> GateDecision:
     """
     Decide how CI should verify a single test case's regression baseline.
@@ -118,6 +126,12 @@ def decide_coupling(  # noqa: PLR0911, PLR0912
         so the committed bundle no longer reflects its inputs — a hard failure.
 
         Always ``True`` when the manifest carries no ``catalog_hash``
+    code_diagnostic_version
+        The diagnostic's current in-code ``Diagnostic.version``, injected by the caller
+        (the pure function does no I/O). When provided, a code version that exceeds
+        ``manifest.diagnostic_version`` fails the case as a stale baseline that must be
+        re-minted, and a code version below it fails as an illegal downgrade with no re-mint.
+        ``None`` (the default) skips the staleness comparison entirely, preserving legacy behaviour.
 
     Returns
     -------
@@ -156,6 +170,28 @@ def decide_coupling(  # noqa: PLR0911, PLR0912
             "regenerate the baseline with `ref test-cases run` after changing the inputs",
         )
 
+    # Runs before seeding so a newly added manifest whose author bumped the code
+    # without re-minting still fails.
+    # Equality must fall through; only the > and < arms return.
+    if manifest is not None and code_diagnostic_version is not None:
+        if code_diagnostic_version > manifest.diagnostic_version:
+            return GateDecision(
+                Action.FAIL,
+                f"diagnostic Diagnostic.version is {code_diagnostic_version} but the baseline "
+                f"manifest records diagnostic_version {manifest.diagnostic_version}; the "
+                "diagnostic's results changed; re-mint this baseline (`ref test-cases mint`) "
+                "so the committed bundle reflects the new version",
+            )
+        if code_diagnostic_version < manifest.diagnostic_version:
+            return GateDecision(
+                Action.FAIL,
+                f"diagnostic Diagnostic.version ({code_diagnostic_version}) is below the "
+                f"baseline's diagnostic_version ({manifest.diagnostic_version}); "
+                "Diagnostic.version is append-only. To revert legitimately, lower the code "
+                "version AND re-mint (`ref test-cases mint`) so the bundle is re-authored at "
+                "the lower version.",
+            )
+
     if base_manifest is None:
         # Seeding a newly added manifest.
         # Replay only verifies something when native blobs exist
@@ -177,6 +213,20 @@ def decide_coupling(  # noqa: PLR0911, PLR0912
             f"{manifest.test_case_version}); version must be monotonic",
         )
 
+    # A diagnostic_version decrease only fails when the committed bundle is unchanged:
+    # a bare downgrade with no re-mint.
+    # A decrease that ships a re-minted bundle is an authorised revert and falls through,
+    # where the committed-changed logic below re-verifies it.
+    if manifest.diagnostic_version < base_manifest.diagnostic_version:
+        if manifest.committed == base_manifest.committed:
+            return GateDecision(
+                Action.FAIL,
+                f"diagnostic_version decreased ({base_manifest.diagnostic_version} -> "
+                f"{manifest.diagnostic_version}) with the committed bundle unchanged; "
+                "a bare downgrade is not permitted; re-mint to re-author the baseline at "
+                "the lower version, or restore the version",
+            )
+
     version_bumped = manifest.test_case_version > base_manifest.test_case_version
     committed_changed = manifest.committed != base_manifest.committed
     native_changed = manifest.native != base_manifest.native

packages/climate-ref-core/src/climate_ref_core/regression/manifest.py

@@ -6,6 +6,8 @@
 - ``schema``: the manifest schema version (``SCHEMA_VERSION``).
 - ``test_case_version``: a monotonic, author-bumped integer coupling the committed
   bundle to its native outputs.
+- ``diagnostic_version``: the author-declared ``Diagnostic.version`` captured at mint
+  time, coupling the baseline to the diagnostic code that produced it.
 - ``committed``: sha256 digests of the committed regression JSON artefacts,
   computed over the exact placeholder-substituted bytes as they sit on disk,
   so that a CI recompute is deterministic.
@@ -26,7 +28,7 @@
 
 from climate_ref_core.paths import safe_path
 
-SCHEMA_VERSION: int = 1
+SCHEMA_VERSION: int = 2
 """Current manifest schema version."""
 
 _DIGEST_RE = re.compile(r"^[0-9a-f]{64}$")
@@ -133,6 +135,14 @@ class Manifest:
     test_case_version: int
     """Monotonic, author-bumped version coupling the bundle to its native outputs."""
 
+    diagnostic_version: int
+    """Author-declared ``Diagnostic.version`` at mint time.
+
+    Monotonic and append-only (with an authorised-revert carve-out in the CI gate).
+    The gate fails a case whose in-code ``Diagnostic.version`` exceeds this value
+    (a stale baseline that must be re-minted).
+    """
+
     committed: dict[str, str]
     """Digests of committed regression JSON artefacts: ``{relpath: sha256}``."""
 
@@ -195,7 +205,11 @@ def loads(cls, text: str, *, source: str = "<string>") -> Manifest:
             (e.g. hand-edited or written by an incompatible version).
         """
         data = json.loads(text)
-        missing = [key for key in ("schema", "test_case_version", "committed", "native") if key not in data]
+        missing = [
+            key
+            for key in ("schema", "test_case_version", "diagnostic_version", "committed", "native")
+            if key not in data
+        ]
         if missing:
             raise ValueError(
                 f"Invalid manifest {source}: missing required keys {missing}. "
@@ -209,6 +223,16 @@ def loads(cls, text: str, *, source: str = "<string>") -> Manifest:
                 f"expected {SCHEMA_VERSION}. The manifest was written by an incompatible "
                 "version; regenerate it with `ref test-cases run --force-regen`."
             )
+        diagnostic_version = data["diagnostic_version"]
+        if (
+            isinstance(diagnostic_version, bool)
+            or not isinstance(diagnostic_version, int)
+            or diagnostic_version < 1
+        ):
+            raise ValueError(
+                f"Invalid manifest {source}: diagnostic_version {diagnostic_version!r} "
+                "is invalid; expected an integer >= 1."
+            )
         try:
             native = {
                 relpath: NativeEntry(sha256=entry["sha256"], size=entry["size"])
@@ -235,6 +259,7 @@ def loads(cls, text: str, *, source: str = "<string>") -> Manifest:
         return cls(
             schema=data["schema"],
             test_case_version=data["test_case_version"],
+            diagnostic_version=diagnostic_version,
             committed=dict(data["committed"]),
             native=native,
             # TODO: remove optonality when all manifests have this field.
@@ -256,6 +281,7 @@ def dump(self, path: Path) -> None:
         payload = {
             "schema": self.schema,
             "test_case_version": self.test_case_version,
+            "diagnostic_version": self.diagnostic_version,
             "catalog_hash": self.catalog_hash,
             "committed": self.committed,
             "native": {relpath: asdict(entry) for relpath, entry in self.native.items()},
@@ -264,7 +290,12 @@ def dump(self, path: Path) -> None:
         path.write_text(text, encoding="utf-8")
 
     @classmethod
-    def seed_v1(cls, committed_digests: dict[str, str], catalog_hash: str | None = None) -> Manifest:
+    def seed_v1(
+        cls,
+        committed_digests: dict[str, str],
+        catalog_hash: str | None = None,
+        diagnostic_version: int = 1,
+    ) -> Manifest:
         """
         Create an initial manifest at ``test_case_version == 1`` with no native outputs.
 
@@ -274,6 +305,9 @@ def seed_v1(cls, committed_digests: dict[str, str], catalog_hash: str | None = N
             Digests of the committed regression JSON artefacts.
         catalog_hash
             Hash of the input ``catalog.yaml`` that produced the baseline, if known.
+        diagnostic_version
+            Author-declared ``Diagnostic.version`` at seed time. Defaults to ``1``,
+            the version of a brand-new baseline.
 
         Returns
         -------
@@ -283,6 +317,7 @@ def seed_v1(cls, committed_digests: dict[str, str], catalog_hash: str | None = N
         return cls(
             schema=SCHEMA_VERSION,
             test_case_version=1,
+            diagnostic_version=diagnostic_version,
             committed=dict(committed_digests),
             catalog_hash=catalog_hash,
             native={},