diff --git a/README.md b/README.md index 2efe296b..132a036e 100644 --- a/README.md +++ b/README.md @@ -203,7 +203,7 @@ python3 -m http.server 8000 # then open http://localhost:8000/ | `forecasts/` | the forecast track record, plus the data store (`#data`) and the release notes index (`#notes`) | | `forecasts/us/` | why there is no US track record yet | | `papers//` | the four working-paper pages: `obr-macro`, `boe-svar`, `frb-us`, `psl-og` (`papers/us-hank/` holds figures only, and `papers/*.pdf` are served directly) | -| `reports/` | replication reports — `define-uk-replication/` and `us-hank-open-source.html` | +| `reports/` | replication reports — `define-uk-replication/` and `us-hank-open-source/` | | `notes/releases/` | index of the generated per-vintage release notes, plus `notes/-/` note pages | | `connect/` | connect it or code it — MCP / CLI setup and the Python API | | `contact/` | who to contact | diff --git a/define/code/index.html b/define/code/index.html index 4042971b..a661dffd 100644 --- a/define/code/index.html +++ b/define/code/index.html @@ -72,7 +72,7 @@

Methodology Validation Code - Paper + Paper diff --git a/define/index.html b/define/index.html index 91b5f270..d30d2a17 100644 --- a/define/index.html +++ b/define/index.html @@ -91,7 +91,7 @@

Connected to the microsimulation — experimentally, deltas in, incidence ou Methodology Validation Code - Paper + Paper @@ -135,7 +135,7 @@

Validated to the ceiling the publications allow. Deltas only.

Two things do not pass. The baseline emissions path runs below the - published table — −3.4% in 2025, widening to −23% by 2040 + published table — −3.5% in 2025, widening to −23.3% by 2040 — pinned at the observed ratios and raised with the authors. And the baseline sits far from official outturns: 2025 growth 4.66% against an ONS 1.31%, 2024 emissions 401.5 MtCO2e against a DESNZ 371 — a gap @@ -146,10 +146,11 @@

Validated to the ceiling the publications allow. Deltas only.

Two more things to know before quoting a number

The multiplier runs high. Our cumulative - green-public-investment multiplier is 1.78, against IMF - green-spending estimates of 1.1–1.5 and an OBR capital - multiplier of ≈1.0 — the demand-led closure, labelled wherever - a delta is served. The upstream table's own multiplier columns are + green-public-investment multiplier is 1.78. It is not comparable to the + short-horizon published multipliers it used to sit beside: ours is + cumulative to 2040, its denominator is endogenous and biases the + ratio up, and a demand-led closure implies above 1 anyway. Labelled + wherever a delta is served. The upstream table's own multiplier columns are unusable, and never quoted.

diff --git a/define/methodology/index.html b/define/methodology/index.html index 5774294c..b5647d5a 100644 --- a/define/methodology/index.html +++ b/define/methodology/index.html @@ -80,7 +80,7 @@

Methodology Validation Code - Paper + Paper @@ -232,7 +232,7 @@

What version 1.1 added

DEFINE-UK 1.78 (our computation) - IMF green spending 1.1–1.5 + short-horizon published multipliers (not comparable) OBR capital spending ≈ 1.0 @@ -243,12 +243,12 @@

What version 1.1 added

The closure

DEFINE-UK is closed demand-led: output adjusts to aggregate demand rather than being pinned to a supply-side potential path. That is the standard post-Keynesian SFC closure, and it has a direct, observable consequence — fiscal expansions are not crowded out by an assumed return to potential, so fiscal multipliers come out larger than in supply-constrained models like the OBR emulator.

Measured, not asserted

-

We measure this rather than assert it: our own computation of the model's cumulative green-public-investment multiplier is 1.78, against IMF green-spending estimates of 1.1–1.5 and an OBR capital-spending impact multiplier of ≈1.0. Above the official range, but explicably so given the closure — and every surface that quotes a DEFINE-UK delta carries the demand-led caveat. The full comparison is on the validation page.

+

We measure this rather than assert it: our own computation of the model's cumulative green-public-investment multiplier is 1.78. It is not comparable to the short-horizon published multipliers it used to be set against: ours is cumulative to 2040, its denominator is endogenous and biases the ratio up, and a demand-led closure implies above 1 anyway. Every surface that quotes a DEFINE-UK delta carries the caveat. The full comparison is on the validation page.

closure and multipliers
closure
demand-led: \(Y\) adjusts to aggregate demand, not a potential path
our computation
cumulative green-public-investment multiplier 1.78
-
comparators
IMF green spending 1.1–1.5 · OBR capital-spending impact ≈ 1.0
+
comparators
not comparable to short-horizon published multipliers: ours is cumulative to 2040, its denominator is endogenous and biases the ratio up, and a demand-led closure implies above 1 anyway
@@ -308,7 +308,7 @@

Verifying the switches

numerical oracle only clean-room Python reimplementation: the manual is the specification - upstream source never read to write equations · 5 milestones · milestone 1 passed with 27 identity tests + upstream source never read to write equations · 5 milestones · milestone 1 passed with 44 identity tests of 62 collected
click to expand
@@ -317,7 +317,7 @@

The constraint

Deltas only, local only

pe-macro define-scenarios and pe-macro define-scenario return annualised scenario-minus-baseline delta paths only, framed with mandatory caveats — never levels, because the baseline's calibration gap to ONS/DESNZ/OBR actuals is computed and material. The hosted MCP tools return run instructions instead of results, because the unlicensed upstream is never hosted. score_reform will not accept this model.

The clean-room path to hosting

-

The path to hosting is a clean-room Python reimplementation of the published Model Manual — the same adapted-model approach as the OBR emulator. The protocol: the manual is the specification, and the upstream run is a numerical oracle only — its outputs are compared against ours, but its source code is never read to write equations, and every equation must carry a reference to the manual section and equation number it implements. Five milestones run from the accounting core to the full policy scenarios; milestone 1 — the §2.2 transactions and balance-sheet matrices on the §5 initial values — passed with 27 identity tests, including one documented inconsistency in the manual itself (Table 6 omits a dividend term of Eq. 383 from the interbank-lending row, which the tests pin exactly rather than paper over).

+

The path to hosting is a clean-room Python reimplementation of the published Model Manual — the same adapted-model approach as the OBR emulator. The protocol: the manual is the specification, and the upstream run is a numerical oracle only — its outputs are compared against ours, but its source code is never read to write equations, and every equation must carry a reference to the manual section and equation number it implements. Five milestones run from the accounting core to the full policy scenarios; milestone 1 — the §2.2 transactions and balance-sheet matrices on the §5 initial values — passed with 44 identity tests of 62 collected, including one documented inconsistency in the manual itself (Table 6 omits a dividend term of Eq. 383 from the interbank-lending row, which the tests pin exactly rather than paper over).

adapter architecture
pinned commit
runtime fetch at 846081a; executed unmodified, in R; nothing vendored or hosted
@@ -357,7 +357,7 @@

The clean-room path to hosting

closure and multipliers
closure
demand-led: \(Y\) adjusts to aggregate demand, not a potential path
our computation
cumulative green-public-investment multiplier 1.78
-
comparators
IMF green spending 1.1–1.5 · OBR capital-spending impact ≈ 1.0
+
comparators
not comparable to short-horizon published multipliers: ours is cumulative to 2040, its denominator is endogenous and biases the ratio up, and a demand-led closure implies above 1 anyway
@@ -82,19 +82,19 @@

The replication gates, check by check.

- Everything below is recorded in the adapter repository's - VALIDATION.md - and enforced by tests against the cached pinned run (upstream commit - 846081a), so drift fails loudly rather than silently. + Every row below is enforced by tests against the cached pinned run + (upstream commit 846081a) and recorded in the adapter's + VALIDATION.md, + so drift fails loudly.

- + - + @@ -102,14 +102,14 @@

The replication gates, check by check.

- + - + @@ -117,22 +117,21 @@

The replication gates, check by check.

- + - + - +
DEFINE-UK replication gates against the manual and the papers
what is checkedtolerancestatus
Manual Table 4, macro block — real GDP growth, unemployment, population 16+, labour force at 2025/2030/2040, S1 baselineManual Table 4, macro block — growth, unemployment, population 16+, labour force, S1 baseline population and labour force ±0.01m; growth ±0.31pp; unemployment ±0.2ppPASS (2026-08-04) — growth exact at 2030/2040, −0.30pp at 2025 against a ±0.31pp gate: passing by 0.01pp is not the same as matchingPASS (2026-08-04) — but 2025 growth misses by −0.30pp, clearing by 0.01pp on a gate set after the run
Manual Table 4, total emissions, S1 baseline DIVERGENCE — pinned; see below
Scenario design — each cached scenario toggles exactly the policy switches its published description claimsScenario design — each cached scenario toggles exactly its published policy switches exact flag set per scenario; HOUSING_SUB_RATE=0.4 pinned PASS (2026-08-05, 10 tests)
FMM-2023 anchors — green public investment GDP peak; baseline 2030 emissions wide (pre-1.0 vintage): GDP peak 0.7–1.3%; emissions 330–352 MtCO2ePASS — peak +0.92% vs published ≈+1%; 342.8 vs “just under 350”. The tests assert membership of the wide bands, not these two figuresPASS — peak +0.92% vs ≈+1%; 342.8 vs “just under 350”. Regression pins, not agreement tests: a band reaching 352 cannot test “just under 350”
Published numeric scenario results for v1.1 CLOSED at the achievable ceiling — none exist; see below
Clean-room reimplementation, milestone 1 — §2.2 transactions and balance-sheet matrices on the §5 initial valuesClean-room reimplementation, milestone 1 — §2.2 matrices on the §5 initial values manual's 4-significant-figure printing precisionPASS (2026-08-04, 62 tests, 44 of them Table 1/2 row and column identities)PASS (2026-08-04) — 62 tests, 44 of them Table 1/2 row and column identities; 13 defects in the manual pinned as machine-readable MANUAL_GAPS records
Calibration vs external observations (ONS/DESNZ/OBR) computed and regression-tested, not tolerance-gatedCOMPUTED — gaps are material; see the table belowCOMPUTED — gaps are material; see below
This adapter runs the authors' unmodified code at a pinned commit. - DEFINE-UK's design belongs to George, Dafermos, Nikolaidi and - co-authors; nothing here is produced or endorsed by them, and no - result should be presented as theirs. + DEFINE-UK belongs to George, Dafermos, Nikolaidi and co-authors; + nothing here is produced or endorsed by them, and no result is theirs.
@@ -144,15 +143,12 @@

The baseline against official numbers, computed not prose.

- Since 2026-08-05 this comparison is an artifact, not a paragraph: - define_uk.validation.baseline_calibration() recomputes - every row from the cached pinned run against pinned external - observations, the result is committed as - validation/baseline_vs_external.csv, - and the test suite fails on any drift between the committed table and - a recomputation. Values use the annual-mean convention, so 2025 - growth reads 4.66% where Table 4 prints 4.96% — the same - −0.30pp anchoring gap recorded under the Table 4 gate. + Every row is recomputed from the cached pinned run against pinned + external observations and committed as + baseline_vs_external.csv; + the suite fails on any drift. The annual-mean convention puts 2025 + growth at 4.66% where Table 4 prints 4.96% — the same + −0.30pp anchoring gap as the Table 4 gate above.

@@ -203,24 +199,18 @@

The baseline against official numbers, computed not prose.

DEFINE-UK baseline (cached pinned run) against external observations, from validation/baseline_vs_external.csv

- external sources, row by row: ONS quarterly national accounts - (uk_gdp_cvm vintage 2026-07-26; 2025 calendar-year mean over 2024) - and OBR March 2026 EFO near-term 1–2%; OBR long-run growth - assumption ≈1.5–1.8%; ONS MGSX (vintage 2026-07-26; 2025 - annual mean, matching the model column's annual-mean convention); - ONS 16+ population estimate; ONS labour force - ≈34–35m; DESNZ UK territorial greenhouse gas emissions, - 2024 provisional (2023 final: 384); UK NDC of 68% below 1990 by 2030 - — the manual itself notes the current-policies baseline falls short - by design. + external sources: ONS national accounts and MGSX (2026-07-26 + vintages), ONS population and labour-force estimates, DESNZ 2024 + provisional territorial emissions, OBR March 2026 EFO, and the UK + NDC — attributed row by row in the committed CSV.

- The implication drives the product: near-term baseline levels are not - competitive with the OBR emulator or the - BoE SVAR and must never be presented as - forecasts. The manual says the baseline “should not be seen as - a prediction”; we take that literally. The defensible outputs - are scenario deltas, with the demand-led caveat stated. + So the levels are not forecasts and are never served as such: the + manual says the baseline “should not be seen as a + prediction”, and near-term levels are not competitive with the + OBR emulator or the BoE SVAR. + Only scenario deltas are defensible, carrying the demand-led + caveat.

@@ -232,34 +222,25 @@

The fiscal multiplier, stated honestly.

- Our own computation of the model's cumulative - green-public-investment multiplier — cumulative ΔGDP over cumulative - ΔSPEND_GVT, nominal, over the full simulation — is - 1.78. It is not comparable to the published - green-spending multipliers it sat beside here — an IMF range of - 1.1–1.5 and the OBR's ≈1.0 capital-spending impact - multiplier. Three things all push the same way. Ours is a - cumulative ratio to 2040 set against short-horizon - multipliers. Its denominator SPEND_GVT is - endogenous: government investment is only 12.8% of it, and - social benefits fall when unemployment does, so a scenario that cuts - unemployment shrinks the denominator and biases the ratio up. And the - closure is demand-led with Kaldor–Verdoorn productivity and real - GDP not held to a supply ceiling, so a ratio above 1 over fifteen - years is what the model implies rather than a finding. The clean - comparison — cumulative ΔGDP over cumulative ΔGCF_GVTG — has not been - computed; until it is, treat 1.78 as upper-leaning. The caveats travel - with every served delta (GPI_MULTIPLIER_CAVEATS, gated by - test). + The model's cumulative green-public-investment multiplier, on our own + computation — nominal cumulative ΔGDP over ΔSPEND_GVT to 2040 — is + 1.78. It is not comparable to the IMF's + 1.1–1.5 or the OBR's ≈1.0, and three things push the same + way: those are short-horizon multipliers where ours is cumulative to + 2040; the denominator is endogenous (government investment is 12.8% of + SPEND_GVT, and social benefits fall as unemployment does, + shrinking it); and a demand-led closure with Kaldor–Verdoorn + productivity implies a ratio above 1 anyway. The clean impulse-based + version has not been computed, so treat 1.78 as upper-leaning. The + caveats travel with every served delta + (GPI_MULTIPLIER_CAVEATS, gated by test).

- The upstream repository's own Multiplier_Summary.csv is - documented unusable: its multiplier columns are - identical across all eight scenarios, and its ΔG figure matches the - cumulative delta of no variable in the scenario file. It is never - quoted; a previously circulated ≈2.4 derived from it should not - be either. Our computation supersedes it and is pinned in the test - suite. + Upstream's own Multiplier_Summary.csv is + unusable and never quoted: its multiplier columns are + identical across all eight scenarios and its ΔG matches the cumulative + delta of no variable in the scenario file. The ≈2.4 derived from + it should not be quoted either.

@@ -272,22 +253,13 @@

The emissions path diverges from the published table.

The one open divergence: the pinned upstream code runs - below the manual's published Table 4 emissions, and the - gap widens with horizon. + below the manual's published Table 4 emissions by + −3.5% in 2025, −10.2% in 2030 and −23.3% in 2040.

-
- - - - - - - -
S1 baseline total emissions: pinned run vs manual Table 4, MtCO2e/yr
yearpinned runpublished tablegap
2025393407−3.4%
2030343382−10%
2040249324−23%
define-uk: S1 baseline total emissions — pinned run vs manual Table 4 (MtCO2e/yr) -Grouped bar chart of S1 baseline total UK emissions in MtCO2e per year, the cached pinned run of upstream commit 846081a against the manual's published Table 4. 2025: pinned run 393, published table 407, a gap of -3.4 per cent; 2030: pinned run 343, published table 382, a gap of -10.2 per cent; 2040: pinned run 249, published table 324, a gap of -23.1 per cent. The pinned code runs below the published table and the gap widens with horizon; the divergence is gated at the observed ratios so any further drift fails loudly. Data from validation/figures/data/define_emissions_divergence.csv. +Grouped bar chart of S1 baseline total UK emissions in MtCO2e per year, the cached pinned run of upstream commit 846081a against the manual's published Table 4. 2025: pinned run 393, published table 407, a gated ratio of 0.965, a gap of -3.5 per cent; 2030: pinned run 343, published table 382, a gated ratio of 0.898, a gap of -10.2 per cent; 2040: pinned run 249, published table 324, a gated ratio of 0.767, a gap of -23.3 per cent. The pinned code runs below the published table and the gap widens with horizon; what is gated is the ratio of the two, pinned to plus or minus 0.02, so any further drift fails loudly. Bar heights are the levels in validation/figures/data/define_emissions_divergence.csv, rounded to whole MtCO2e; the percentages are the gated ratios themselves, which is why they differ slightly from the ratio of the rounded bars. 0 @@ -304,33 +276,31 @@

The emissions path diverges from the published table.

407 published -2025 (-3%) +2025 (-3.5%) 343 pinned run 382 published -2030 (-10%) +2030 (-10.2%) 249 pinned run 324 published -2040 (-23%) +2040 (-23.3%) -
S1 baseline total emissions at the manual's benchmark years: the cached pinned run (upstream commit 846081a) against the published Table 4. The gap widens from −3.4% in 2025 to −23% in 2040 and is gated at the observed ratios. Source: validation/figures/data/define_emissions_divergence.csv, from the adapter repository's VALIDATION.md.
+
S1 baseline total emissions: the cached pinned run (upstream 846081a) against the published Table 4. What is gated is the ratio of the two — 0.965 / 0.898 / 0.767, pinned to ±0.02 in the adapter's validation/reference_outputs.json, which is where the percentages above come from; the bar labels round the published levels. Source: validation/figures/data/define_emissions_divergence.csv.

- This is not an extraction error: the components + Not an extraction error: the components (EMIS_NELEC + EMIS_ELEC) sum - exactly to the total, and the baseline is identical across scenario - folders — so it is a vintage or calibration gap between the published - table and commit 846081a. The divergence is gated at the - observed ratios so any further drift fails loudly, and it has been - raised with the authors on the upstream issue tracker - (DEFINE-model/DEFINE_UK_1.1#1). + exactly to the total and the baseline is identical across scenario + folders, so it is a vintage or calibration gap between the published + table and commit 846081a — raised with the authors as + DEFINE_UK_1.1#1.

@@ -342,35 +312,20 @@

No numeric v1.1 scenario results are published. That is the ceiling.

- The explicit statement. An exhaustive search - established that no machine-readable numeric scenario results are - published for DEFINE-UK v1.1: the manual's results stop at the - baseline (Table 4) plus scenario design parameters, and - the accompanying papers are paywalled with figure-only results. A - full published-figure replication of the scenarios is therefore not - possible today, and every scenario delta this adapter serves carries - that status. + No machine-readable numeric scenario results are published for + DEFINE-UK v1.1: the manual stops at the baseline plus scenario + design parameters, and the accompanying papers are paywalled + with figure-only results. A published-figure replication of the + scenarios is not possible today, and every delta this adapter serves + carries that status.

- What is verifiable is verified, at three levels. The design gate - pins that each scenario toggles exactly its published policy - switches. The oracle gates pin the delta paths against the cached - pinned run — deltas finite, baselines identical across scenario - blocks, sign sanity, eleven pinned delta points. And the two figures - the paper's open 2023 conference vintage states in text both anchor, - within deliberately wide vintage tolerances: green public investment - peaks at +0.92% of GDP against a published - ≈+1%, and the 2030 current-policies baseline emits - 342.8 MtCO2e/yr against “just under - 350”. -

-

- The upgrade path is equally explicit: the gate reopens if the - authors publish scenario tables, or if the paywalled papers become - accessible. Until then the status label stands — experimental; - partial replication; scenario deltas only, never levels — and the - score a reform workflow does not accept - this model. + So what is gated is design and self-consistency — the switch sets, and + the delta paths against the cached pinned run — plus the two coarse + FMM-2023 anchors above. The gate reopens if the authors publish + scenario tables or the papers become accessible. Until then: + experimental, partial replication, deltas only, and + score a reform does not accept this model.

diff --git a/economy/topics/growth/index.html b/economy/topics/growth/index.html index 49c82b37..b596f0d4 100644 --- a/economy/topics/growth/index.html +++ b/economy/topics/growth/index.html @@ -148,7 +148,7 @@

The model view, and the same breath its limits.

Predictive validation: weak. - No demonstrated skill against a random walk WITH DRIFT on any variable at any horizon once the 64 variable-by-horizon tests are adjusted together: minimum Benjamini-Hochberg q = 0.36, and 0.40 under the published specification. A driftless walk on a trending log level is too weak a benchmark, so the strong-looking CPI result (0.63 at h=1) becomes 0.83 at h=1 and 1.03 at h=8 against drift. Bank Rate comes closest (0.79 at h=1, unadjusted p=0.018 -- the smallest of the 64) but does not survive the adjustment. UK GDP is not distinguishable from drift at any horizon (p=0.33-0.43). Two caveats on figures quoted elsewhere: the ex-Covid ratio of 0.77 and the p=0.38-0.67 range are against the weaker no-change benchmark, not drift. Separately, the rolling evaluation had estimated without the six Covid dummies that every published forecast carries; under the published specification UK GDP goes 1.06 to 0.99 at h=1 and 1.12 to 0.95 at h=8 -- level with naive rather than worse, still not better. The frozen-edge run gives 0.32pp RMSE from a single origin, on the 2024Q2 real-time vintage. + No demonstrated skill against a random walk WITH DRIFT on any variable at any horizon. The grid runs 64 variable-by-horizon tests; adjusted together, the smallest Benjamini-Hochberg q is 0.36. Bank Rate comes closest (0.79 at h=1) and does not survive. The model looks strong against a driftless walk, but that benchmark forfeits the trend on a trending log level and is too weak to learn from. Under the published specification UK GDP is level with naive rather than worse -- an improvement, not a win. Benchmarks, horizons and the multiplicity table are on the validation page. Read the validation page →

diff --git a/economy/topics/inflation/index.html b/economy/topics/inflation/index.html index ede1b525..2fd47119 100644 --- a/economy/topics/inflation/index.html +++ b/economy/topics/inflation/index.html @@ -137,7 +137,7 @@

The model view, and the same breath its limits.

Predictive validation: weak. - No demonstrated skill against a random walk WITH DRIFT on any variable at any horizon once the 64 variable-by-horizon tests are adjusted together: minimum Benjamini-Hochberg q = 0.36, and 0.40 under the published specification. A driftless walk on a trending log level is too weak a benchmark, so the strong-looking CPI result (0.63 at h=1) becomes 0.83 at h=1 and 1.03 at h=8 against drift. Bank Rate comes closest (0.79 at h=1, unadjusted p=0.018 -- the smallest of the 64) but does not survive the adjustment. UK GDP is not distinguishable from drift at any horizon (p=0.33-0.43). Two caveats on figures quoted elsewhere: the ex-Covid ratio of 0.77 and the p=0.38-0.67 range are against the weaker no-change benchmark, not drift. Separately, the rolling evaluation had estimated without the six Covid dummies that every published forecast carries; under the published specification UK GDP goes 1.06 to 0.99 at h=1 and 1.12 to 0.95 at h=8 -- level with naive rather than worse, still not better. The frozen-edge run gives 0.32pp RMSE from a single origin, on the 2024Q2 real-time vintage. + No demonstrated skill against a random walk WITH DRIFT on any variable at any horizon. The grid runs 64 variable-by-horizon tests; adjusted together, the smallest Benjamini-Hochberg q is 0.36. Bank Rate comes closest (0.79 at h=1) and does not survive. The model looks strong against a driftless walk, but that benchmark forfeits the trend on a trending log level and is too weak to learn from. Under the published specification UK GDP is level with naive rather than worse -- an improvement, not a win. Benchmarks, horizons and the multiplicity table are on the validation page. Read the validation page →

diff --git a/frb-us/validation/index.html b/frb-us/validation/index.html index f6d8a226..9902d654 100644 --- a/frb-us/validation/index.html +++ b/frb-us/validation/index.html @@ -81,38 +81,33 @@

The baseline agreement is an identity; the shocks are the evidence.

- This is a replication with a published anchor — the - sharpest-tested member of the suite, because for once the ground truth - is exact rather than statistical. Three invariants carry the weight, - all enforced in continuous integration, including four like-for-like - scenarios gated against vendor-generated references. One of the three - carries no weight of its own: the tracking invariant is an identity - init_trac constructs, so the evidence is the agreement - with the Board's shocked paths, where the add-factors are - held fixed and the solver has real work to do. + Shocked paths match the Fed's own pyfrbus to + 6.0×10−9, inside the gap between the Board's own + two releases. The baseline tracking residual is not evidence: + init_trac makes that solve an algebraic identity + (table below).

- Machine precision is a claim about fidelity to the Fed's - baseline, not about the baseline being right: against realized - 2026 outturns, the April 2026 LONGBASE tracks GDP growth, - unemployment, and the funds rate within ~0.3pp but misses - core PCE inflation by −1.7pp (2.7% baseline vs 4.4% - annualized outturn, 2026Q1) — recorded by a weekly + Fidelity to the Fed's baseline is not the baseline being right. Against + 2026 outturns the April 2026 LONGBASE tracks GDP growth, unemployment + and the funds rate within ~0.3pp but misses core PCE inflation + by −1.7pp (2.7% vs a 4.4% annualized outturn, 2026Q1) — + a weekly outturn - backtest that reports rather than gates, since the baseline is - the Board's, not ours. + backtest that reports rather than gates, because the baseline is + the Board's.

tracking invariant 5.6e−17 - vs LONGBASE, all 284 endos × 20 quarters; CI gate 1e−8. An identity init_trac guarantees — round-off, not evidence + an identity, not evidence — see the table
shock vs pyfrbus 1.0.0 6.0e−9 - the committed reference; CI gate 1e−6 + 100bp shock, 284 endos × 20 quarters; the pinned reference, CI gate 1e−6
shock vs pyfrbus 1.1.1 @@ -127,115 +122,97 @@

The baseline agreement is an identity; the shocks are the evidence.

- The last figure frames the other three. The Board's two releases ship - a byte-identical model.xml and LONGBASE and - differ only in how the Newton routine reuses its LU factorisation — - yet they disagree with each other by - 1.3×10−8, as much as this - implementation disagrees with either. Agreement therefore sits at the - reference implementation's own numerical noise, not at a tolerance - anyone chose. All three residuals concentrate in the same near-zero - expectational-gap series (wpsn, zgap05). + The last tile frames the other three. The Board's two releases ship a + byte-identical model.xml and LONGBASE, differing + only in how Newton reuses its LU factorisation, yet disagree by + 1.3×10−8 — as much as this implementation + disagrees with either. Agreement sits at the reference's own numerical + noise, not at a chosen tolerance; all three residuals concentrate in + the same near-zero expectational-gap series (wpsn, + zgap05).

- + frb-us: residuals against the Fed’s pyfrbus, log scale -Horizontal bar chart on a base-10 logarithmic axis of maximum absolute residuals; shorter is closer. ours vs LONGBASE (identity — not evidence): 5.6e-17; pyfrbus 1.1.1 vs LONGBASE: 1.1e-08; ours vs pyfrbus 1.0.0 (shock): 6.0e-09; ours vs pyfrbus 1.1.1 (shock): 1.4e-08; pyfrbus 1.1.1 vs 1.0.0 — the Fed’s own two releases: 1.3e-08. The framing comparison is the last row: the Federal Reserve's own two pyfrbus releases disagree with each other by as much as this implementation disagrees with either, so our agreement sits at the scale of the reference implementation's own numerical noise rather than at a chosen tolerance. +Horizontal bar chart on a base-10 logarithmic axis of maximum absolute residuals; shorter is closer. The rows fall into three kinds. ours vs LONGBASE — tracking identity, not evidence: 5.6e-17 (tracking identity); pyfrbus 1.1.1 vs LONGBASE — tracking identity: 1.1e-08 (tracking identity); ours vs pyfrbus 1.0.0 (shock): 6.0e-09 (ours vs the Fed); ours vs pyfrbus 1.1.1 (shock): 1.4e-08 (ours vs the Fed); pyfrbus 1.1.1 vs 1.0.0 — the Fed’s own releases: 1.3e-08 (the Fed vs itself). The two tracking rows are an identity — init_trac defines the add-factors as minus the residuals at the input data, so re-solving reproduces the input for any input at all — and are shown for completeness, not as evidence. The framing comparison is the last row: the Federal Reserve's own two pyfrbus releases disagree with each other by as much as this implementation disagrees with either, so our agreement sits at the scale of the reference implementation's own numerical noise rather than at a chosen tolerance. 1e-18 - -1e-16 - -1e-14 - -1e-12 - -1e-10 - -1e-8 -ours vs LONGBASE (identity — not evidence) - -5.6e-17 -pyfrbus 1.1.1 vs LONGBASE - -1.1e-08 + +1e-16 + +1e-14 + +1e-12 + +1e-10 + +1e-8 +ours vs LONGBASE — tracking identity, not evidence + +5.6e-17 +pyfrbus 1.1.1 vs LONGBASE — tracking identity + +1.1e-08 ours vs pyfrbus 1.0.0 (shock) - -6.0e-09 + +6.0e-09 ours vs pyfrbus 1.1.1 (shock) - -1.4e-08 -pyfrbus 1.1.1 vs 1.0.0 — the Fed’s own two releases - -1.3e-08 + +1.4e-08 +pyfrbus 1.1.1 vs 1.0.0 — the Fed’s own releases + +1.3e-08 +ours vs the Fed’s pyfrbus, under shock +the Fed’s two releases vs each other +tracking identity — round-off, not fidelity -
Maximum absolute residuals, log scale — lower is closer. The bottom bar is the framing one: the Fed's own two releases disagree by as much as we disagree with either, so the three shock rows sit at the reference implementation's own noise floor. The top two bars are the tracking identity and are
not a ranking: after init_trac the solve reproduces its input by construction, so 5.6×10−17 against pyfrbus 1.1.1's 1.1×10−8 compares two implementations' round-off, not their fidelity to the Fed's economics. Source: papers/frb-us validation tables. +
Maximum absolute residuals, log scale — lower is closer. The top two bars are the tracking identity and are not a ranking: they compare two implementations' round-off, not their fidelity to the Fed's economics. Source: papers/frb-us validation tables.
-
+
- - + + - - - - - - - + +
Validation results against the Federal Reserve Board's own artefacts
testwhat it checksresult
Tracking invariantAfter init_trac, solving the baseline must reproduce LONGBASE for all 284 endogenous variables over 2026Q1–2030Q4. This is an identity, not a comparison: init_trac sets each equation's add-factor to minus that equation's residual at the input data, so the solve is algebraically the identity on whatever it was tracked to, and Newton is warm-started from the row it is about to reproduce.Max absolute error 5.6e−17 — floating-point round-off, against a 1e−8 gate. It measures re-evaluation noise, not agreement with the Fed's data or economics: the model repository's test_tracking_invariant_holds_for_arbitrary_data shows the identical gate passes on a scrambled baseline with every accounting identity destroyed. Read the cross-validation row below, not this one, as evidence.Re-solving the baseline reproduces LONGBASE, 284 endogenous variables over 2026Q1–2030Q4.5.6e−17 against a 1e−8 gate — but an identity, not a comparison. init_trac sets each add-factor to minus that equation's residual at the input data, so the solve reproduces whatever it was tracked to, for any input: the model repository's test_tracking_invariant_holds_for_arbitrary_data passes at 6.7e−9 on a scrambled baseline with every accounting identity destroyed. Read the row below, not this one, as evidence.
Cross-validationAn identical 100bp rffintay_aerr shock run in both implementations, compared across all 284 variables and all 20 quarters.6.0e−9 against pyfrbus 1.0.0 and 1.4e−8 against 1.1.1 — where the two Board releases differ from each other by 1.3e−8.
Simulation propertiesA 100bp tightening must move the right things in the right direction by the right order of magnitude.rff +1.000pp on impact, GDP trough −0.55%, unemployment peak +0.26pp, core inflation trough −0.034pp — consistent with the Board's documented VAR-expectations properties.The same 100bp rffintay_aerr shock run in both implementations, all 284 variables × 20 quarters.6.0e−9 against pyfrbus 1.0.0 and 1.4e−8 against 1.1.1, where the two Board releases differ from each other by 1.3e−8.

- Multipliers. Government-purchases and personal-tax - shocks of 1% of GDP land inside the published ranges they are compared - against, with one exception recorded below: the year-two - purchases multiplier under a pegged funds rate is - 0.90, below the 1.1–1.2 cross-model accommodation - range, and matches only the Board's looser “roughly - one” characterisation. The comparisons are also not horizon-matched - throughout — the year-one tax figure (0.22) is set beside a benchmark - the source describes as a two-year multiplier, and Ramey's survey range - pools empirical estimates across horizons and monetary regimes rather - than fixing one. Read the last three rows as plausibility bands, not as - a test the model passed. + Multipliers. Read these as plausibility bands, not as + a test passed: the comparisons are not horizon-matched throughout, and + Ramey's survey range pools estimates across horizons and monetary + regimes.

- + - - - - - - + + +
frb-us against the Fed's pyfrbus and published multiplier rangesfrb-us fiscal multipliers (1% of GDP, sustained) against published ranges
QuantityOursPublishedReading
Tracking invariant, max abs. error5.6×10−17CI gate <10−8; pyfrbus 1.1.1 gives 1.1×10−8an identity, not accuracy — passes on scrambled data
Shock responses vs pyfrbus 1.0.06.0×10−9—at the reference's noise floor
Shock responses vs pyfrbus 1.1.11.4×10−81.1.1 vs 1.0.0: 1.3×10−8the Fed's releases differ as much as we do
Gov. purchases multiplier, yr 1, inertial Taylor rule0.720.7–1.0 (Coenen et al. 2012); Ramey (JEP 2019): "a surprisingly narrow range of 0.6 to 1"inside range
Gov. purchases multiplier, yr 2, fixed funds rate0.90"roughly one" pegged; 1.1–1.2 with accommodationmatches the Board's characterisation
Personal tax cut multiplier, yrs 1–20.22 → 0.320.2–0.4; CBO central ≈0.3inside both ranges
Gov. purchases, yr 1, inertial Taylor rule0.720.7–1.0 (Coenen et al. 2012); Ramey (JEP 2019): "a surprisingly narrow range of 0.6 to 1"inside range
Gov. purchases, yr 2, pegged funds rate0.90"roughly one" pegged; 1.1–1.2 with accommodationbelow the accommodation range
Personal tax cut, yrs 1–20.22 → 0.320.2–0.4; CBO central ≈0.3; yr-1 benchmark is a two-year figureinside range, horizons unmatched

- Uncertainty engine. The implementation now supports - seeded stochastic simulations that jointly resample the official - stochastic-equation residual vector, preserving contemporaneous - dependence and reporting failed replications. Model and LONGBASE - packages are also verified against separate SHA-256 provenance gates. - Published probability results still require a reviewed simulation - design; model-consistent expectations remain unavailable. + Uncertainty engine. Seeded stochastic simulation + jointly resamples the official residual vector, preserving + contemporaneous dependence; model and LONGBASE packages carry SHA-256 + provenance gates. Published probability results still need a reviewed + simulation design, and model-consistent expectations remain + unavailable.

- The committed reference stays pinned to pyfrbus 1.0.0 so the gate has a - fixed anchor; the 1.1.1 comparison is a recorded cross-check. CI goes - beyond re-checking a committed CSV: a dedicated job re-runs the Board's - pyfrbus from source in a throwaway environment and gates against that - freshly generated solution, and a weekly scheduled run repeats the suite - across Python 3.10–3.12 so dependency drift surfaces without a pull - request. Full tables in + CI does not just re-check a committed CSV: a dedicated job re-runs the + Board's pyfrbus from source and gates against that fresh solution, and a + weekly run repeats the suite across Python 3.10–3.12. The multiplier + comparisons are refreshed manually. Full tables in VALIDATION.md and the working paper.

@@ -245,15 +222,6 @@

The baseline agreement is an identity; the shocks are the evidence.

maintained, or endorsed by the Federal Reserve, and its results should not be presented as official Federal Reserve estimates.

-

- How to read the evidence on this page. The tracking - invariant, the pyfrbus cross-validation, and the monetary-shock - sign/magnitude bands are gated — CI on the model - repository fails if they break. The fiscal-multiplier comparisons - against published ranges were computed for the paper - and are refreshed manually. Both are real evidence; only the first - kind is continuously enforced. -

@@ -264,28 +232,26 @@

Run the same 100bp shock, read it against the Board.

- One command reproduces the validation experiment — a 100 basis-point - funds-rate tightening under the inertial Taylor rule: + One command reproduces a 100bp funds-rate tightening under the + inertial Taylor rule: 

pe-macro frbus-shock --var rffintay_aerr --shock 1.0
- + - - - - - + + + + +
ResponseOursPublished Fed referenceReading
ResponseOursPublished Fed reference
Funds rate, on impact+1.000pp+1.00pp by designexact
Real GDP, trough−0.55%consistent with the Board's documented VAR-expectations propertiesright sign, size and timing
Output gap, trough−0.50pp~−0.4pp (2014 FEDS Note, read off charts)approximate — vintage and shock-design differences
Unemployment, peak+0.26ppconsistent with the documented propertiesright sign and order
Core inflation, trough−0.034ppconsistent with the documented propertiesright sign and order
Funds rate, on impact+1.000pp+1.00pp by design — exact
Real GDP, trough−0.55%right sign, size and timing vs the Board's documented VAR-expectations properties
Unemployment, peak+0.26ppright sign and order, same source
Core inflation, trough−0.034ppright sign and order, same source
Output gap, trough−0.50pp~−0.4pp, read off 2014 FEDS Note charts — approximate by nature (limits)

The exact cross-check is numerical, not chart-reading: the same shock - run in the Board's own pyfrbus matches to - 6.0×10−9 (see the invariants above). The FEDS - Note comparison is approximate by nature — its values are read off - 2014-vintage charts. + in the Board's own pyfrbus matches to + 6.0×10−9.

@@ -301,27 +267,27 @@

Known limitations.

VAR expectations only - The model-consistent-expectations (MCE) equation variants are not implemented — Frbus(path, mce=...) raises NotImplementedError — so anything that turns on agents anticipating announced future policy, forward guidance most obviously, is out of scope. + Model-consistent expectations are not implemented (mce= raises NotImplementedError), so forward guidance is out of scope. No mcontrol - The reference implementation's trajectory-targeting routine is out of scope. Stochastic simulation is implemented — stochsim_bands() produces seeded residual-bootstrap percentile bands — but the hosted frbus_shock path returns a single deterministic projection, so bands are not available over MCP. + Trajectory targeting is out of scope. stochsim_bands() does produce seeded bootstrap bands, but the hosted frbus_shock returns a single deterministic path. Hosted, but only as raw shocks - The hosted tools and their pe-macro mirrors expose one lever, one shock size, a policy rule and a headline set of responses — not the full 284-variable model. Anything beyond that (the Board's demo scripts, exogenize, arbitrary variable sets) means installing the package and using its Python API. + One lever, one shock size, a policy rule and a headline set of responses — not the full 284-variable model. Anything more means installing the package. No PolicyEngine reform bridge - Reforms enter as shocks to model variables, not as statute. There is no equivalent of the OBR emulator's static-costing bridge from a PolicyEngine reform dict — score_reform refuses model="frbus" outright and points at frbus_shock — so US reform scoring stops at pe-microsim's static answer. Hosting did not change this. + Reforms enter as shocks, not statute: score_reform refuses model="frbus", so US reform scoring stops at pe-microsim's static answer. Approximate published comparisons - The 100bp funds-rate responses in the Board's 2014 FEDS Note are read off charts, so those comparisons are approximate by nature: our output-gap trough of −0.50pp against the Note's roughly −0.4pp reflects vintage and shock-design differences, not a claimed match. + The 2014 FEDS Note responses are read off charts. Our −0.50pp output-gap trough against its ~−0.4pp reflects vintage and shock-design differences, not a claimed match. Vintage - The April 2026 model.xml and LONGBASE as published by the Board. Later Board vintages have not been adopted. + The Board's April 2026 model.xml and LONGBASE. Later vintages not adopted. diff --git a/integration/src/policyengine_macro/capabilities.py b/integration/src/policyengine_macro/capabilities.py index 31ec7b8d..74f3a0cd 100644 --- a/integration/src/policyengine_macro/capabilities.py +++ b/integration/src/policyengine_macro/capabilities.py @@ -247,23 +247,15 @@ def _quality(level: str, evidence: str, next_gate: str) -> dict: "predictive_validation": _quality( "weak", "No demonstrated skill against a random walk WITH DRIFT on any " - "variable at any horizon once the 64 variable-by-horizon tests " - "are adjusted together: minimum Benjamini-Hochberg q = 0.36, and " - "0.40 under the published specification. A driftless walk on a " - "trending log level is too weak a benchmark, so the strong-looking " - "CPI result (0.63 at h=1) becomes 0.83 at h=1 and 1.03 at h=8 " - "against drift. Bank Rate comes closest (0.79 at h=1, unadjusted " - "p=0.018 -- the smallest of the 64) but does not survive the " - "adjustment. UK GDP is not distinguishable from drift at any " - "horizon (p=0.33-0.43). Two caveats on figures quoted elsewhere: " - "the ex-Covid ratio of 0.77 and the p=0.38-0.67 range are against " - "the weaker no-change benchmark, not drift. Separately, the " - "rolling evaluation had estimated without the six Covid dummies " - "that every published forecast carries; under the published " - "specification UK GDP goes 1.06 to 0.99 at h=1 and 1.12 to 0.95 " - "at h=8 -- level with naive rather than worse, still not better. " - "The frozen-edge run gives 0.32pp RMSE from a single origin, on the " - "2024Q2 real-time vintage.", + "variable at any horizon. The grid runs 64 variable-by-horizon " + "tests; adjusted together, the smallest Benjamini-Hochberg q is " + "0.36. Bank Rate comes closest (0.79 at h=1) and does not " + "survive. The model looks strong against a driftless walk, but " + "that benchmark forfeits the trend on a trending log level and " + "is too weak to learn from. Under the published specification " + "UK GDP is level with naive rather than worse -- an " + "improvement, not a win. Benchmarks, horizons and the " + "multiplicity table are on the validation page.", "Score the predictive densities rather than point forecasts, " "report rolling interval coverage, and re-run once the estimation " "sample extends past the Covid dummies.", @@ -271,14 +263,11 @@ def _quality(level: str, evidence: str, next_gate: str) -> dict: "identification_robustness": _quality( "moderate", "On the paper's own definition -- the posterior mean of the " - "per-draw group share of TOTAL forecast-error variance, four " - "quarters ahead -- UK GDP replicates (37.4% against ~40%) and UK " - "CPI falls about 8pp short (42.3% against ~50%). The earlier " - "match on both came from summing per-shock medians and " - "renormalising them to 100%, which inflated the identified shares " - "by about a third. The 68% posterior band is roughly +/-14pp, " - "wider than the shortfall, and proxy world data and undisclosed " - "source settings matter.", + "per-draw group share of TOTAL four-quarter-ahead variance -- UK " + "GDP replicates (37.4% against ~40%) and UK CPI falls about 8pp " + "short (42.3% against ~50%). The 68% band is roughly +/-14pp, " + "wider than the shortfall, so the data do not resolve it. Proxy " + "world aggregates remain a candidate explanation.", "Show conclusions across lag, prior, proxy-data and weighting grids " "with effective-sample-size diagnostics.", ), @@ -422,12 +411,15 @@ def _quality(level: str, evidence: str, next_gate: str) -> dict: ), "policy_counterfactual_validity": _quality( "moderate", - "The static costing of a 1p basic-rate rise gives £6.46bn in 2026 " - "rising to £7.38bn by 2030, against HMRC's published ready " - "reckoner at £6.9bn and £8.2bn — within range, and the gap is " - "reported rather than tuned away. That is one reform, one " - "country, one tax, against another estimate on a different data " - "basis, so it is a benchmark and not a validation.", + "The static costing of a 1p basic-rate rise gives £6.46bn in " + "2026-27, which is 6.4% BELOW HMRC's published ready reckoner at " + "£6.9bn -- not within its range, and the gap is reported rather " + "than tuned away. The reckoner stops at 2028-29, so the £7.38bn " + "this suite scores for 2030 has no year-matched counterpart. " + "HMRC's income-tax reckoner is post-behavioural on administrative " + "data and ours is static and survey-based, so this is one reform, " + "one country, one tax, on a different basis: a benchmark and not " + "a validation.", "Benchmark a US reform and a distributional result against an " "independent published costing.", ), @@ -561,11 +553,9 @@ def _quality(level: str, evidence: str, next_gate: str) -> dict: "§2.2 accounting and §3.2, §3.3.1 and §3.3.2 — 118 equations, " "Eqs. (21)-(138) contiguous — and surfaced defects in the manual " "itself, thirteen of them pinned as machine-readable gap records " - "rather than absorbed into a tolerance. An audit of the wider " - "write-up re-characterised four findings as first-period jumps " - "(Table 5 describes those parameters as historical means, never " - "claimed to fit the manual's own initial period) and withdrew one " - "claim outright.", + "rather than absorbed into a tolerance. Four apparent " + "contradictions proved to be historical-mean calibration and are " + "recorded as such in the replication report.", "Complete §3.3 and the oracle comparison, closing milestone 2.", ), "predictive_validation": _quality( diff --git a/integration/tests/test_site_contract.py b/integration/tests/test_site_contract.py index a1eae02d..88e669d6 100644 --- a/integration/tests/test_site_contract.py +++ b/integration/tests/test_site_contract.py @@ -135,6 +135,12 @@ def test_paper_landings_publish_the_page_count_of_the_pdf_they_link(): at all, obr-macro 36 -> 40 (the refreshed scorecard, the lever scorecard and the anchored-fit dependency) and boe-svar 30 -> 33 (the corrected FEVD statistic and the multiplicity result). + + define-uk joined at 28 pages, filling the one gap in the suite: it was the + only model without a working paper. Its /models card has been repointed + from /reports/define-uk-replication to /papers/define-uk, so every paper + is linked from the hub again and the exemption that recorded the gap is + gone. """ expected = { "obr-macro": 40, @@ -142,6 +148,7 @@ def test_paper_landings_publish_the_page_count_of_the_pdf_they_link(): "frb-us": 38, "psl-og": 34, "pe-microsim": 25, + "define-uk": 28, } # The /papers listing was retired; each landing carries its own # "slug · … · N pages" line, and /models links every paper landing. diff --git a/models/index.html b/models/index.html index f6d1248a..fcc8ec83 100644 --- a/models/index.html +++ b/models/index.html @@ -71,7 +71,7 @@

Choose the model that fits the quest
→Worked example: 1p on the basic rate — £6.46bn (2026) → £7.38bn (2030), GDP −0.02% on impact building to −0.06%
-
→Static costing sits within HMRC's published ready-reckoner range
+
→Static costing runs below HMRC's ready reckoner at every matched year — −6.4% in 2026–27; a benchmark, not a validation
→Hosted — MCP tools or one CLI command

The score_reform MCP tool and the pe-macro score CLI take the same flat reform dict and a scoring model: microsim (static costing), obr (with macro feedback), og (long-run steady state; local-only), or og+microsim (dynamic).

@@ -350,7 +350,7 @@

Read the source literature and replication notes.

frb-usworking paper

Parsing and solving the Fed's FRB/US model, and matching pyfrbus on shocked paths at its own noise floor.

read →
psl-ogworking paper

The OG-UK calibration, tax-function estimation, and the limits of a model with no ground truth.

read →
us-hankfeasibility note

The open US HANK landscape, and why the Auclert et al. (2021) toolkit was the foundation to build on. Not a replication write-up — that gate is on the us-hank validation page.

read → -
define-ukreplication report

The authors' unmodified code at a pinned commit, the Table 4 gate, thirteen pinned manual defects, and why only deltas are served.

read → +
define-ukworking paper

The authors' unmodified code at a pinned commit, the Table 4 gate, thirteen pinned manual defects, and why only deltas are served.

read → diff --git a/obr/index.html b/obr/index.html index 6061c1f7..69d11226 100644 --- a/obr/index.html +++ b/obr/index.html @@ -81,7 +81,7 @@

Microsimulation supplies the direct cost; this model returns the feedback.
→Connection. Static-costing bridge, both hosted: PolicyEngine computes the direct Exchequer effect of a statutory reform; score_reform injects it through the declared HHDI_ADDFACTOR point and this model returns the economy-wide second round.
-
→Worked example. 1p on the basic rate from April 2026: direct yield £6.46bn in 2026; GDP −0.020% on impact, deepening to −0.058% by 2027Q4 (full walkthrough → score a reform).
+
→Worked example. 1p on the basic rate from April 2026: direct yield £6.46bn in 2026; GDP −0.020% on impact, deepening to −0.057% by 2027Q4 (full walkthrough → score a reform).
→Validation. Anchored to the March 2026 EFO (0.15% GDP MAPE) — but anchored accuracy is a by-construction invariant, not a forecast claim: free-running, the same equations miss GDP by 4.48%. Checked against HMRC’s ready reckoner (£6.46bn vs £6.9bn).
→Results. Run it from any connected client; every score carries its assumptions and data vintage.
@@ -168,7 +168,7 @@

Second-round effects on a static costing.

Worked example — 1p on the basic rate of income tax from April 2026: direct yield £6.46bn in 2026, £7.38bn by 2030; GDP falls 0.020% (−£0.14bn) on impact in - 2026Q1, deepening to 0.058% (−£0.40bn) by 2027Q4 on the + 2026Q1, deepening to 0.057% (−£0.40bn) by 2027Q4 on the March 2026 baseline. Full walkthrough → /score.

diff --git a/obr/validation/index.html b/obr/validation/index.html index 104f106b..db3e13b3 100644 --- a/obr/validation/index.html +++ b/obr/validation/index.html @@ -82,20 +82,20 @@

Excellent anchored. Weak free-running. Both published.

- Three configurations, three very different levels of trust — errors - are MAPE (mean absolute percentage error) against the published EFO path. + Three configurations, three levels of trust. Errors are MAPE (mean + absolute percentage error) against the published EFO path.

anchored — real GDP 0.15% MAPE - vs published EFO; CI fails at 1% + by construction, not skill; CI fails at 1%
anchored — consumption 0.25% MAPE - vs published EFO; CI fails at 1% + not a second test — same £m error as GDP
held-add-factor forecast @@ -105,7 +105,7 @@

Excellent anchored. Weak free-running. Both published.

free-running 4.48% MAPE - real GDP; 4 of 11 computed in band — report-only + real GDP; 6 of 11 computed in band — report-only
@@ -115,17 +115,17 @@

Excellent anchored. Weak free-running. Both published.

Anchored - Add-factors on, matched to the March 2026 EFO. The configuration every reform score uses. - GDP 0.15% MAPE, consumption 0.25% over 2025Q1–2027Q4; horizon extends to 2031Q1, where GDP is reproduced to 0.29% (anchored unemployment unreliable beyond 2027Q4). Hard-gated in CI, with finiteness, a 0.5%-of-GDP expenditure-identity check, and structural sign invariants. + Add-factors matched to the March 2026 EFO. Every reform score uses it. + GDP 0.15% MAPE, consumption 0.25%, 2025Q1–2027Q4. GDP holds to 0.29% out to 2031Q1; anchored unemployment is unreliable beyond 2027Q4. CI hard-gates the tolerance, finiteness, the expenditure identity to 0.5% of GDP, and sign invariants. Held add-factors - Add-factors fitted 2024Q1–2025Q4, held flat, projected to 2027Q4. Scored over 2026Q1–2027Q4 on the March 2026 baseline (caveats in the notes below). - GDP 0.37%, consumption 0.33%; 6 of 8 computed variables within band. (The working paper reports GDP 2.2% and consumption 3.6% for the same 2026Q1–2027Q4 window — the difference is the EFO vintage it was anchored to, November 2025 against March 2026 here, not the window.) + Fitted 2024Q1–2025Q4, held flat to 2027Q4, scored over 2026Q1–2027Q4. Initialised at the EFO values it is scored against, and its fitting window contains OBR-forecast quarters — so it partly encodes “agree with the OBR”. + GDP 0.37%, consumption 0.33%; 6 of 8 computed in band. The working paper’s 2.2% / 3.6% is the same window on the November 2025 vintage. Free-running - De-seeded, add-factors off. The raw structural dynamics with no OBR judgement. + De-seeded, add-factors off — raw structural dynamics, no OBR judgement. Weak, and reported as such. GDP 4.48%, consumption 7.49%, household income 6.27%, business investment 15.73%, company profits 63.29%. The model contracts while the EFO grows. @@ -133,27 +133,13 @@

Excellent anchored. Weak free-running. Both published.

in band rates within 1.0pp · net balances within 1.5% of GDP · levels within 10% MAPE.

-

note anchored accuracy is a by-construction invariant, not a forecast claim: add-factors absorb the tracking error; CI hard-fails if it slips.

-

caveat the held-add-factor forecast is initialised at the EFO values it is scored against.

-

caveat its add-factor base window includes OBR-forecast (not outturn) quarters, so held add-factors partly encode "agree with the OBR".

-

more full scorecards in the working paper; the charts below regenerate from its committed data.

+

gated CI enforces the anchored tolerance and the multiplier band; the HMRC comparison and the outturn backtest are computed for the working paper and refreshed by hand.

An independent implementation built from the OBR's published model - code and forecast data — not produced, maintained, or endorsed by the - Office for Budget Responsibility. Its results should not be presented - as official OBR estimates. -

-

- How to read the evidence below. Two kinds of check - appear on this page: gated — enforced by CI on the - model repository, so a regression cannot merge without failing a - build (the anchored-vs-EFO tolerance, the multiplier plausibility - band) — and computed for the paper — run once for - the working paper's figures and refreshed manually (the HMRC - ready-reckoner comparison, the ONS outturn backtest). Both are real - evidence; only the first is continuously enforced. + code and forecast data. Not produced, maintained or endorsed by the + Office for Budget Responsibility, and not official OBR estimates.

@@ -165,10 +151,10 @@

Against the OBR's own forecast, and HMRC's reckoner.

- The chart below shows the anchored tracking error quarter by quarter - (headline MAPEs are in the strip and table above); because anchoring - is by construction, the independent HMRC costing further down is the - stronger test. + Anchored GDP and consumption are one test, not two: + the deviation is identical in £m in all twelve quarters, because + every other term in the demand identity is exogenous or pinned. The + independent test is the HMRC costing below.

@@ -219,15 +205,17 @@

Against the OBR's own forecast, and HMRC's reckoner.

-
Current March 2026 EFO baseline. CI hard-fails at ±1.00% — off the top and bottom of this frame. MAPE: 0.15% for GDP and 0.25% for consumption over 2025Q1–2027Q4. Computed from papers/obr-macro/figures/fig_anchored_data.csv, regenerated from the March 2026 detailed forecast tables on 21 July 2026.
+
Current March 2026 EFO baseline. Quarterly deviation from the published EFO; CI hard-fails at ±1.00%, off the top and bottom of this frame. Computed from papers/obr-macro/figures/fig_anchored_data.csv, regenerated from the March 2026 detailed forecast tables on 21 July 2026.

- The independent check: a 1pp basic-rate rise from April 2026, costed - through the PolicyEngine static-costing bridge and set against HMRC's - ready reckoner. Ours sits inside the £6–8bn range of recent - vintages; the gap widens by 2028–29, where HMRC's figures embed - administrative-data fiscal drag that survey microdata capture less - fully. + A 1pp basic-rate rise from April 2026, costed through the PolicyEngine + static-costing bridge: £6.46bn against HMRC's £6.9bn, + −6.4%. Every PolicyEngine year is below every HMRC + year, and the gap widens where HMRC's administrative data carry fiscal + drag that survey microdata capture less fully. HMRC's reckoner stops + at 2028–29, so the 2030 row is set against that same figure + carried forward. A benchmark, not a validation — + caveats in full.

@@ -260,7 +248,7 @@

Against the OBR's own forecast, and HMRC's reckoner.

HMRC basic rate +1pp, 2028–29 -
£bn/yr. 2028–29 emulator figure interpolated between scored endpoints (£6.46bn 2026, £7.38bn 2030). PolicyEngine's static costing of a 1pp basic-rate rise against HMRC's Direct effects of illustrative tax changes (June 2025 vintage). Source: obr-macro working paper, comparison table panel B.
+
£bn/yr, against HMRC's Direct effects of illustrative tax changes (June 2025 vintage). The 2028–29 emulator bar is interpolated between the scored endpoints £6.46bn (2026) and £7.38bn (2030). Source: obr-macro working paper, comparison table panel B.
@@ -280,87 +268,65 @@

Against the OBR's own forecast, and HMRC's reckoner.

- + How much of the OBR emulator scorecard the model actually computes -Two stacked bars. Of 21 headline variables in the OBR emulator calibration scorecard, 11 are actually computed by the model and 10 are passthrough, held at the OBR published value and therefore scoring zero error trivially. Of the 11 computed, 5 are fair, 1 is an identity, 3 are poor, 2 are off. 6 of the 11, or 55 per cent, land within band, and one of those is a trivial accounting identity, so only 5 non-trivial computed variables are in band. The worst are company profits 63.29 per cent and the current account 3.60 per cent of GDP. +Two stacked bars. Of 21 headline variables in the OBR emulator calibration scorecard, 11 are actually computed by the model and 10 are passthrough, held at the OBR published value and therefore scoring zero error trivially. Of the 11 computed, 5 are fair, 1 is an identity, 3 are poor, 2 are off. 6 of the 11, or 55 per cent, land within band, but one of those is an accounting identity that closes over passthrough inputs, so 5 of the 10 non-trivial computed variables are in band. The worst are company profits 63.29 per cent and the current account 3.60 per cent of GDP. 21 headline scorecard variables - - -11 computed -10 passthrough — held at the OBR value - - -of which, the 11 the model computes - -fair 5 - -identity 1 - -poor 3 - -off 2 -Only 5 of 11 non-trivial variables are in band. -A fourth pass is an identity over passthrough inputs. -bands: rates ±1.0pp · net balances ±1.5% of GDP · levels ≤10% MAPE + + +11 computed +10 passthrough — held at the OBR value + + +of which, the 11 the model computes + +fair 5 + + +identity 1 + +poor 3 + + +off 2 +Only 5 of the 10 non-trivial computed variables are in band. +One of the 6 in-band passes is an accounting identity over passthrough inputs. +bands: rates ±1.0pp · net balances ±1.5% of GDP · levels ≤10% MAPE -
Raw calibration against the March 2026 EFO. Two “off” variables depend on unpublished OBR constants and are regression-gated rather than tuned. Source:
docs/calibration_scorecard.md in the obr-macroeconomic-model repository. +
Raw calibration against the March 2026 EFO. The two “off” variables depend on unpublished OBR constants and are regression-gated, not tuned. Source: docs/calibration_scorecard.md in the obr-macroeconomic-model repository.

- The spending multiplier is ~1.0 by construction, against the + The spending multiplier is 1.0 by construction, against the OBR's own published 0.6 for day-to-day public services and welfare spending. - Under the demand closure a spending - shock lands directly in the GDP identity and the behavioural second - round is largely inactive, so a £5bn injection returns almost - exactly £5bn of GDP — roughly a - two-thirds overstatement of the impact multiplier, - applying to every spending-side figure this page reports. It is the - single most important number for anyone reading a policy score off - this model. -

-

- Why, and what it biases → obr-macro overview. + Under the demand closure a spending shock lands straight in the GDP + identity with the behavioural second round inactive, so £5bn in + returns £5bn of GDP — a two-thirds overstatement + of every spending-side figure on this page.

The honest scorecard, outturns, and the March 2026 re-anchoring

- The honest scorecard. The free-running row is the one - that matters for anyone tempted to read the 0.15% as forecasting skill. - The same equations that track the EFO to 0.15% when anchored miss it - by 4.48% free-running — precisely the gap the OBR's - own add-factor judgement closes in the official process, and the - reason reform deltas are always scored against the anchored baseline - rather than the raw one. The free-running score is de-seeded, with - passthrough variables excluded, and is published report-only. - Across the full scorecard 6 of the 11 computed headline - variables land within band (real GDP, consumption, both - household-income lines, the trade balance, and the trivial employment - identity); of the full 21-line scorecard, 10 lines are passthroughs - held at the OBR value. The - worst line is company profits at 63.29% MAPE on the - March baseline, which - traces to a single unpublished constant in households' operating - surplus OSHH — the paper documents and regression-gates - it rather than re-tuning it, since tuning it would be fitting to the - answer. Other lines that moved on re-anchoring, reported rather than - smoothed: the free-running current account widened from 2.76 to - 3.60% of GDP and is now over band; RPI improved from - 2.03pp to 1.71pp; business investment stands at - 15.73%; the two household-income lines stand at - 6.27% and 6.03%. Every figure in this paragraph moved when the - OSHH ONS anchor landed upstream, and this page had gone - on publishing the pre-anchor ones — which understated the model's - accuracy on every line. + The honest scorecard. The 4.48% free-running miss is + the gap the OBR's own add-factor judgement closes — which is why + reform deltas are scored against the anchored baseline and never the + raw one. Company profits, the worst line, trace to a single + unpublished constant in households' operating surplus + OSHH: documented and regression-gated rather than + re-tuned, since tuning it would be fitting to the answer. Two lines + not in the chart above — real household income 6.03%, RPI 1.71pp — + and every figure here is the March 2026 vintage after the upstream + OSHH ONS anchor.

obr-macro: real GDP level, anchored vs free-running vs the March 2026 EFO (£bn/qtr) Line chart of quarterly real GDP levels in billions of pounds, 2025Q1 to 2027Q4. The published March 2026 EFO path rises from 703.4 to 728.6. The anchored emulator is visually indistinguishable from it, running from 703.8 to 730.6 (mean absolute deviation 0.15 per cent, recomputed here from the plotted series). The free-running emulator, de-seeded and with no add-factors, contracts from 694.4 to 675.8 — a gap that widens to 53 billion pounds, 4.48 per cent mean absolute deviation over the horizon. Free-running and EFO paths from papers/obr-macro/figures/fig_free_running_data.csv; anchored path from papers/obr-macro/figures/fig_anchored_data.csv. Coordinates: value v in billions maps to y = 292 - (v - 660) * 2.95 on a 660 to 740 axis; quarter i of 12 maps to x = 58 + i * 60.545. - + anchored (0.15% MAD) - + free-running (4.48% MAD) -EFO Mar 2026 +EFO Mar 2026 660 @@ -379,26 +345,22 @@

Against the OBR's own forecast, and HMRC's reckoner.

2027Q1 2027Q4 -
Current March 2026 EFO baseline. Real GDP, £bn/qtr, 2025Q1–2027Q4. The anchored path (0.15% MAPE) sits on top of the EFO; the same equations free-running — de-seeded, no add-factors — contract away from it (4.48% MAPE). Computed from papers/obr-macro/figures/fig_free_running_data.csv and fig_anchored_data.csv, regenerated on 12 August 2026.
+
Current March 2026 EFO baseline. Real GDP, £bn/qtr. The anchored path sits on top of the EFO; the same equations free-running — de-seeded, no add-factors — contract away from it. Computed from papers/obr-macro/figures/fig_free_running_data.csv and fig_anchored_data.csv, regenerated on 12 August 2026.

Forecast versus outturn. Comparing one forecast - vintage with another tests agreement, not accuracy. Against ONS - outturns published since anchoring, quarter-on-quarter real GDP - growth ran 0.1% in 2025Q2, 0.2% in Q3, 0.2% in Q4 and 0.6% in - 2026Q1. The emulator's path (0.15, 0.14, 0.25, 0.37) tracks the three - 2025 quarters to within 0.06 percentage points, but — like the - November EFO it inherits (0.28, 0.20, 0.27, 0.39) — misses the strong - 2026Q1 outturn by roughly a quarter of a point. Two caveats govern - the reading: this is primarily a test of the OBR's November vintage, - the emulator's own contribution being the 0.02–0.13 point gap - between the two model rows; and ONS quarterly estimates are - themselves revised, so the outturn is a moving target. + vintage with another tests agreement, not accuracy. The emulator + tracks the three 2025 outturns to within 0.06pp and — like the + November EFO it inherits — misses the strong 2026Q1 outturn by + roughly a quarter point. It is mostly a test of that November + vintage: the emulator's own contribution is the 0.02–0.13pp gap + between the two model rows. ONS quarterly estimates are themselves + revised.

obr-macro: quarterly real GDP growth — emulator vs EFO Nov 2025 vs ONS outturn (% q/q) -Grouped bar chart, percentage quarter-on-quarter real GDP growth for the four quarters with ONS outturns since anchoring. 2025Q2: emulator 0.15, EFO 0.28, ONS 0.1; 2025Q3: emulator 0.14, EFO 0.20, ONS 0.2; 2025Q4: emulator 0.25, EFO 0.27, ONS 0.2; 2026Q1: emulator 0.37, EFO 0.39, ONS 0.6. The emulator tracks the three 2025 outturns to within 0.06 points; both the emulator and the November EFO it inherits miss the strong 0.6 per cent 2026Q1 outturn by roughly a quarter of a point. Data from papers/obr-macro/figures/fig_outturn_data.csv. Coordinates: value v maps to y = 258 - v * 331.4 on a 0 to 0.7 axis. +Grouped bar chart, percentage quarter-on-quarter real GDP growth for the four quarters with ONS outturns since anchoring. 2025Q2: emulator 0.15, EFO 0.28, ONS 0.10; 2025Q3: emulator 0.14, EFO 0.20, ONS 0.20; 2025Q4: emulator 0.25, EFO 0.27, ONS 0.20; 2026Q1: emulator 0.37, EFO 0.39, ONS 0.60. The emulator tracks the three 2025 outturns to within 0.06 points; both the emulator and the November EFO it inherits miss the strong 0.6 per cent 2026Q1 outturn by roughly a quarter of a point. Data from papers/obr-macro/figures/fig_outturn_data.csv. Coordinates: value v maps to y = 258 - v * 331.4 on a 0 to 0.7 axis. emulator EFO Nov 2025 ONS outturn @@ -412,64 +374,56 @@

Against the OBR's own forecast, and HMRC's reckoner.

0.6 0.15 +emul. 0.28 +EFO 0.10 -2025Q2 +ONS +2025Q2 0.14 +emul. 0.20 +EFO 0.20 -2025Q3 +ONS +2025Q3 0.25 +emul. 0.27 +EFO 0.20 -2025Q4 +ONS +2025Q4 0.37 +emul. 0.39 +EFO 0.60 -2026Q1 +ONS +2026Q1 -
November 2025 EFO vintage — the working paper's study; the live baseline is anchored to the March 2026 EFO. % q/q real GDP growth. The emulator tracks the three 2025 outturns to within 0.06pp; both model rows miss the strong 2026Q1 outturn (0.6%) by roughly a quarter point. Computed from papers/obr-macro/figures/fig_outturn_data.csv; the table below carries the exact values.
+
November 2025 EFO vintage — the working paper's study; the live baseline is anchored to the March 2026 EFO. % q/q real GDP growth, each bar labelled. Computed from papers/obr-macro/figures/fig_outturn_data.csv.
-
- - - - - - - - - -
Quarterly real GDP growth: emulator, EFO vintage, and ONS outturn
Real GDP, % q/qEmulatorEFO Nov 2025ONS outturn
2025Q20.150.280.1
2025Q30.140.200.2
2025Q40.250.270.2
2026Q10.370.390.6
-

- Vintage: re-anchored to the March 2026 EFO. The - hosted emulator has been re-anchored from the November 2025 EFO to - the OBR's March 2026 forecast, and the headline numbers on this page - are computed on that baseline: anchored GDP 0.15% MAPE, consumption - 0.25% over 2025Q1–2027Q4, with the anchored horizon extended to - 2031Q1 (GDP reproduced to 0.29% at 2031Q1; anchored unemployment is - unreliable beyond 2027Q4, drifting to 0.9% against the EFO's 4.1% by - 2031Q1). The headline and free-running charts use March 2026. Only - the outturn backtest above retains November 2025, because changing its - forecast vintage would erase the historical forecast being tested. The - working paper opens with a dated current-vintage note. Reform - effects are differences between structurally - identical runs and are insensitive to modest baseline drift, which is - why the re-anchoring leaves the £6.46bn/£7.38bn static costing - untouched and moves the second-round GDP effect only from - −0.057% to −0.058% by 2027Q4. + Vintage. Re-anchored from the November 2025 EFO to + March 2026, and every headline on this page is computed on that + baseline. Only the outturn backtest above retains November 2025, + because changing its forecast vintage would erase the historical + forecast being tested. Reform effects are differences between + structurally identical runs, so the re-anchoring leaves the + £6.46bn/£7.38bn static costing untouched and moves the 2027Q4 + second-round GDP effect by 0.001pp, to −0.057%.

@@ -491,15 +445,19 @@

Known limitations.

Two household-income equations never fire - The listing's only bare log() left-hand sides — log(HHTFA) and log(NDIVHH) — never execute: inputs MAJGDP and CORP are absent from the databank. The profits → dividends → household-income channel is inert: a 5pp corporation-tax rise moves FYCPR by −£1,780m; ΔNDIVHH is exactly zero. A parser fix landed, numerically inert; reviving the channel needs a CORP series — a calibration decision not yet made. All published figures have these channels inert. + The listing's only bare log() left-hand sides — log(HHTFA) and log(NDIVHH) — never execute: their inputs MAJGDP and CORP are absent from the databank. The profits → dividends → household-income channel is inert: a 5pp corporation-tax rise moves FYCPR by −£1,780m and NDIVHH by exactly zero. Reviving it needs a CORP series. Every published figure has the channel inert. + + + Impact multiplier 1.0 by construction + 1.0 vs the OBR's published 0.6 — the warning above explains why and what it biases. - Impact multiplier ~1 by construction - ~1.0 vs the OBR's published 0.6 — the warning above explains why and what it biases. + Government investment does not transmit + CGIPS moves business investment by exactly zero in all twelve quarters and leaves a wrong-signed GDP residue, against the OBR's published 1.0. Capital spending should not be scored on this model. - Corporation-tax closure is specialised - The investment closure runs the published TCPRO → cost-of-capital → investment chain via obr_shock: business-investment equation reconstructed from the OBR's commented, truncated line; level anchored with held add-factors; MSGVA, PIF, PIRHH frozen to a shared tracking path against uncalibrated feedback. CI hard-gates the expected sign and a £50bn-per-quarter bound. A controlled scenario closure, not a calibrated supply block. + Corporation-tax closure never converges + The published TCPRO → cost-of-capital → investment chain, with the business-investment equation reconstructed from the OBR's truncated line and MSGVA, PIF, PIRHH frozen against uncalibrated feedback. The deviation compounds ~25% a quarter with no steady state: the 12-quarter horizon is where the magnitude still looks plausible, not where it settles. CI gates the sign and a £50bn-per-quarter bound — a controlled scenario closure, not a calibrated supply block. Passthrough channels @@ -511,7 +469,7 @@

Known limitations.

Vintage - October 2025 equation listing, March 2026 EFO, current-vintage ONS series. Where the ONS has revised history the identities don't close exactly; that slack lands in the add-factors. The working paper remains a November 2025 EFO vintage study. + October 2025 equation listing, March 2026 EFO, current-vintage ONS series. Where the ONS has revised history the identities don't close exactly; that slack lands in the add-factors. No behavioural micro diff --git a/olg/validation/index.html b/olg/validation/index.html index 61fb8e50..225f1861 100644 --- a/olg/validation/index.html +++ b/olg/validation/index.html @@ -81,12 +81,12 @@

Two UK overlapping-generations models, side by side.

- This page compares OG-UK — the UK calibration of + OG-UK — the UK calibration of OG-Core, maintained by the - Policy Simulation Library — with the - OBR UK OLG model described in + Policy Simulation Library — against the + OBR UK OLG model of OBR Working Paper No. 22 - (Brzezinski, Hantzsche & Watson, April 2025) — the closest available + (Brzezinski, Hantzsche & Watson, April 2025), the closest available external check for a model of this kind.

@@ -110,9 +110,6 @@

Two UK overlapping-generations models, side by side.

Production technology.

-

- How each model represents firms. -

@@ -146,9 +143,9 @@

Household heterogeneity.

- + - +
OG-UKOBR UK OLG
Precautionary savingsNot a primary channelYes — driven by earnings uncertainty
Utility functionCRRA on consumption; elliptical labour disutilityCobb-Douglas over consumption and leisure (non-separable)
Labour disutilityElliptical function — always interior solutionsImplicit in Cobb-Douglas composite — corner solutions possible
Risk aversion parameterσ on consumption onlyσ₂ = 2 on consumption-leisure composite
Risk aversion parameterσ = 1.5 on consumption onlyσ₂ = 2 on consumption-leisure composite
Consumption aggregationStone-Geary Cobb-Douglas (subsistence levels, I goods)Single consumption good
Discount factorβj varies by ability typeβ = 0.99 (uniform)
Discount factorβ = 0.965, fixed and uniform (per-type βj supported, not used here)β = 0.99, calibrated to the WAS asset profile
@@ -160,9 +157,6 @@

Household heterogeneity.

Bequest motive.

-

- How each model handles wealth transfer across generations at death. -

@@ -180,7 +174,6 @@

Bequest motive.

Tax system.

-

How each model represents the UK tax system.

OG-UK — estimated tax functions
@@ -229,9 +222,6 @@

Government & fiscal policy.

Open economy.

-

- How foreign capital enters each model, and where the interest rate comes from. -

OG-UKOBR UK OLG
@@ -283,9 +273,8 @@

Solution method & calibration.

- Source: OBR Working Paper No. 22, "A new UK - overlapping generations model" (April 2025); OG-Core documentation and OG-UK - calibration code. + Sources: OBR Working Paper + No. 22; OG-Core documentation and OG-UK calibration code.

@@ -297,10 +286,9 @@

No ground truth. Calibration targets instead.

- No published replication target exists. The table - compares calibration targets with official UK aggregates. Imposed - matches are anchors; structural counterfactuals remain conditional on - the model's auditable assumptions and code. + No published replication target exists for this model + class. The table compares calibration inputs with official UK + aggregates. Nothing in it is a model output.
OG-UKOBR UK OLG
@@ -308,7 +296,7 @@

No ground truth. Calibration targets instead.

- + @@ -317,38 +305,37 @@

No ground truth. Calibration targets instead.

QuantityOfficial UK valueSourceModel treatmentDeviation
Government debt-to-GDP94.9% (June 2026)ONS · HF6X, vintage 2026-07-26imposed via closure target+0.1pp vs latest ONS (target 95.0%)
Household saving ratio8.9% (2026Q1, down from 9.6%)ONS quarterly sector accountsnot targeted — β is fixed, not estimatedno model-produced value to compare
Household saving ratio8.9% (2026Q1, down from 9.6%)ONS quarterly sector accountsnot targeted — β = 0.965 fixed, not estimatedno model-produced value to compare
Potential growth≈1.1% / yrOBR EFOimposed, gy = 0.011≈0
Depreciation / capital≈6–7% (CFC / net stock)ONS capital stocks bulletinimposed, δ = 0.065within range
Labour share of income≈0.59–0.60, rising in 2024ONS labour-share series / Blue Book 2025imposed, γ = 0.35 (OG-Core default)≈5pp — the capital share is not yet re-anchored to ONS factor shares

- A zero deviation on an imposed quantity is not independent evidence. + An imposed target cannot disagree, so a zero + deviation above is an anchor, not evidence. Only the emergent row is + a check.

The nearest external benchmark is OBR - Working Paper 22, Table 5.1 — the long-run effect of a 1pp - basic-rate rise: GDP/person −0.1%, labour supply −0.2%, - productivity +0.1%. The OG-UK side of that comparison is open - validation work: WP 22's figures are the target a future run will - be judged against, not a match claimed today. + Working Paper 22, Table 5.1 — a 1pp basic-rate rise in the long + run: GDP/person −0.1%, labour supply −0.2%, productivity + +0.1%. Those are the figures a future OG-UK run will be judged + against, not a match claimed today.

What would count as over-identifying evidence, and the dating and structural caveats

- The genuinely over-identifying checks are the ones the - steady state must deliver: the capital–output ratio and - the interest rate consistent with the calibrated (γ, δ, β) triple. - These emerge in an economically sensible range in the deployed - configuration, but no systematic published reconciliation of - achieved-versus-target moments exists for OG-UK 0.3.2, and - the working paper declines to manufacture one. Producing that - reconciliation from logged solves is the single most valuable next - step for this member. + The genuinely over-identifying checks are the ones the steady state + must deliver: the capital–output ratio and the interest + rate implied by the calibrated (γ, δ, β) triple. Both emerge in an + economically sensible range, but no systematic published + reconciliation of achieved-versus-target moments exists for + OG-UK 0.3.2. Producing one from logged solves is the most valuable + next step for this member.

- Two dating caveats: the official column reports figures current at - mid-2026 while the deployed calibration was frozen against slightly - earlier vintages, and the saving ratio is volatile quarter to - quarter, so β is matched to a smoothed level rather than the latest - print. Two structural caveats carried over from OG-Core: the ability - types and lifetime earnings profiles are still the US tax-microdata - estimates, and UK re-estimation is an open task. + Two dating caveats: the official column reports mid-2026 figures + while the deployed calibration was frozen against slightly earlier + vintages, and the saving ratio is volatile quarter to quarter, so the + fixed β was set against a smoothed level rather than the latest + print. Two structural caveats from OG-Core: the ability types and + lifetime earnings profiles are still the US tax-microdata estimates, + and UK re-estimation is open.

@@ -370,7 +357,7 @@

Experimental limitations.

Version pin - The integration installs oguk from GitHub (0.3.2), which pins policyengine-uk==2.88.0. Versions ≥ 2.89 renamed the microdata dataset keys (enhanced_frs_2023_24_<year> → populace_uk_*), so mixing them fails with a KeyError at calibration time; the adapter traps this and raises an actionable error. Lifting the pin is roadmap work — until it lands the OLG member scores against a slightly older statute vintage than the microsimulation member. A declared inconsistency, not a hidden one. + oguk 0.3.2 pins policyengine-uk==2.88.0. Versions ≥ 2.89 renamed the microdata dataset keys, so mixing them fails with a KeyError at calibration time; the adapter traps it and raises an actionable error. Until the pin lifts, the OLG member scores against an older statute vintage than the microsimulation member — a declared inconsistency, not a hidden one. Gated data @@ -382,7 +369,7 @@

Experimental limitations.

Steady-state-only in the CLI - Because the CLI exposes steady-state solves only (Code, step 7), the model cannot yet speak to fiscal-year profiles through the suite — the long-run destination is available through it, the path there is not. + The CLI exposes steady-state solves only (Code, step 7): the long-run destination is reachable through the suite, the path there is not. Ability profiles not UK-estimated diff --git a/papers/boe-svar/index.html b/papers/boe-svar/index.html index 070007d7..85e46579 100644 --- a/papers/boe-svar/index.html +++ b/papers/boe-svar/index.html @@ -22,6 +22,7 @@ @@ -72,7 +73,7 @@

-
+
boe-svar · UK · hosted · 33 pages @@ -84,15 +85,13 @@

An open replication of the Bank of England structural VA

-

Current forecast: data through 2026Q1, outlook from 2026Q2–2029Q2. The PDF keeps the frozen 2024Q2 evaluation separately: 0.32pp RMSE for GDP and CPI, with 14/14 outturns inside the 68% bands — though the 49-origin rolling evaluation finds the bands under-cover on average, so this single-origin result overstates calibration.

- -

Corrected August 2026, and the PDF now reflects it. The headline replication statistic was the wrong quantity: computed the way Brignone & Piffer (2025) define it — posterior mean of the per-draw group share of total four-quarter-ahead variance — the one-year global share is 37.4% for UK GDP against the paper's ~40% (a match) and 42.3% for UK CPI against ~50% (about 8pp short). The previously published 42.1% / 49.5% came from summing per-shock posterior medians and renormalising them to 100%. On forecasting: across 49 rolling origins, nothing beats a random walk with drift on any variable at any horizon once the 64 tests are adjusted together (Benjamini–Hochberg minimum q = 0.36).

+

On the statistic Brignone & Piffer (2025) actually define — posterior mean of the per-draw group share of total four-quarter-ahead variance — the identified global shocks explain 37.4% of UK GDP forecast-error variance against the paper's ~40%, and 42.3% of UK CPI against ~50%, a gap narrower than the 68% band of about ±14pp. Forecasting is the weaker half: from the frozen 2024Q2 edge the RMSE is 0.32pp for both GDP and CPI, but 14 of 14 outturns inside the 68% band is over-dispersion at that origin rather than a pass, and across 49 rolling origins nothing beats a random walk with drift once the 64 tests are adjusted together (Benjamini–Hochberg minimum q = 0.36). Current forecast: data through 2026Q1, outlook 2026Q2–2029Q2. Validation →

- +

Your browser can’t display PDFs inline. The paper is 33 @@ -119,7 +118,7 @@

An open replication of the Bank of England structural VA  ·  All papers →

-

Cite this work

+

Cite this work

Ahmadi, Vahid (2026). “An open replication of the Bank of England structural VAR.” PolicyEngine Macro working paper. https://policyengine-macro.vercel.app/papers/boe-svar

@techreport{policyengine2026boesvar,
   title       = {An open replication of the Bank of England structural VAR},
diff --git a/papers/define-uk/index.html b/papers/define-uk/index.html
new file mode 100644
index 00000000..1cdfd107
--- /dev/null
+++ b/papers/define-uk/index.html
@@ -0,0 +1,148 @@
+
+
+
+
+
+DEFINE-UK ecological SFC replication — PolicyEngine Macro
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+

+  
+    PolicyEngine Macro
+  
+  
+

+
+
+
+

+  

+    

+      
model 07 — ecological stock-flow consistent · define-uk · UK · experimental

+      

+        Climate policy meets the UK financial system.
+      

+      

+        Who bears a climate-policy scenario — through balance sheets, not an
+        output gap. The replication record, and the ceiling the publications
+        impose on it.
+      

+    

+  

+
+  
+
+  

+    

+      

+        define-uk · UK · experimental · 28 pages
+        
DEFINE-UK: running and gating an unlicensed ecological stock-flow consistent model

+      

+      

+        Download PDF
+        All papers
+      

+    

+
+    
One gate passes, one diverges, two close at a ceiling. The manual's Table 4 macro block replicates — population and labour force exact to 0.01m, growth matching to 0.06pp at 2030 and 2040 — but the 2025 growth miss of −0.30pp clears a ±0.31pp gate set after the run, so that year is a pass of the tolerance and not a match, and the paper labels it that way at the point of claim. Baseline emissions diverge and are pinned rather than tolerated: 393 vs 407 MtCO2e/yr in 2025, 343 vs 382 in 2030, 249 vs 324 by 2040 (−3.5% / −10.2% / −23.3%, computed from the pinned ratios).

+
+    
No numeric v1.1 scenario results are published, so the two scenario targets are closed at the achievable ceiling, not passed: what is gated is scenario design plus two coarse anchors from a pre-1.0 conference version, and the emissions anchor admits 330–352 for a published claim of “just under 350”. The upstream is unlicensed — fetched at pinned commit 846081a at runtime, never vendored or hosted — and the baseline sits far from outturns (2025 growth 4.66% against an ONS 1.31%), so the adapter serves deltas, never levels. The 1.78 green-investment multiplier is ours and is not comparable to any published multiplier. In parallel, a clean-room Python reimplementation is 118 contiguous equations into its second milestone, with thirteen manual defects pinned as machine-readable gap records.

+
+    
+    

+      
+        

+          

+            Your browser can’t display PDFs inline. The paper is 28
+            pages — open or download it instead.
+          

+          Open the PDF
+        

+      
+    

+
+    
+    

+      28 pages · pdf
+      

+        Inline PDF viewing is unreliable on mobile browsers, so it’s disabled
+        here. Open the paper in your device’s PDF reader instead.
+      

+      Open the PDF
+    

+
+    

+      Source & reproduction: papers/define-uk/ on GitHub
+       ·  Replication report →
+       ·  All papers →
+    

+
+    
Cite this work

+    
Ahmadi, Vahid (2026). “DEFINE-UK: running and gating an unlicensed ecological stock-flow consistent model.” PolicyEngine Macro working paper. https://policyengine-macro.vercel.app/papers/define-uk

+    
@techreport{policyengine2026defineuk,
+  title       = {DEFINE-UK: running and gating an unlicensed ecological stock-flow consistent model},
+  author      = {Ahmadi, Vahid},
+  year        = {2026},
+  institution = {PolicyEngine},
+  url         = {https://policyengine-macro.vercel.app/papers/define-uk},
+  note        = {PolicyEngine Macro working paper}
+}

+    
No DOI is assigned. For an exact version stamp, cite the git commit of the artifact you used from github.com/PolicyEngine/macro. The model design belongs to its authors (George and Dafermos, SOAS); the replication and the clean-room implementation are ours, and are not endorsed by them.

+  

+

+
+
+
+
+
diff --git a/papers/define-uk/main.pdf b/papers/define-uk/main.pdf
new file mode 100644
index 00000000..952215af
Binary files /dev/null and b/papers/define-uk/main.pdf differ
diff --git a/papers/define-uk/main.tex b/papers/define-uk/main.tex
new file mode 100644
index 00000000..98eb96f2
--- /dev/null
+++ b/papers/define-uk/main.tex
@@ -0,0 +1,52 @@
+\documentclass[11pt]{article}
+\usepackage[margin=1.1in]{geometry}
+\usepackage{booktabs}
+\usepackage{tabularx}
+\usepackage{longtable}
+\usepackage{amsmath}
+\usepackage{amssymb}
+\usepackage{graphicx}
+\usepackage{xcolor}
+% Paper chrome uses the firm-microsim template blue (YGTemplate MyBlue);
+% figures use the PolicyEngine palette separately.
+\definecolor{paperblue}{rgb}{0,0,0.7}   % MyBlue, matching the sister papers
+\definecolor{peteal}{HTML}{39C6C0}      % PolicyEngine teal accent
+\usepackage{sectsty}
+\allsectionsfont{\color{paperblue}}
+\usepackage[colorlinks=true, citecolor=paperblue, linkcolor=paperblue, urlcolor=paperblue]{hyperref}
+\usepackage{xurl}
+\usepackage[hang,flushmargin]{footmisc}
+\usepackage{natbib}
+\emergencystretch=1.2em
+\bibliographystyle{apalike}
+
+\title{\bfseries DEFINE-UK: running and gating an unlicensed ecological stock-flow consistent model\thanks{\textbf{Acknowledgements:} I thank Max Ghenis for his great suggestions and helpful discussions. DEFINE-UK is the work of Adam George and Yannis Dafermos at SOAS University of London, building on the DEFINE framework of Yannis Dafermos, Maria Nikolaidi and Giorgos Galanis; this paper documents an independent replication and reimplementation of their published model, is neither produced by nor endorsed by them, and any errors in the description of their work are mine. This paper and the accompanying codebase were prepared with the assistance of Claude (Anthropic); all remaining errors are my own.}\\[0.7cm]}
+\author{Vahid Ahmadi\thanks{Research Associate at PolicyEngine. Email: \href{mailto:vahid@policyengine.org}{vahid@policyengine.org}.}}
+\date{\small \ifcase\month\or January\or February\or March\or April\or May\or June\or July\or August\or September\or October\or November\or December\fi\ \the\year}
+
+\begin{document}
+\maketitle
+
+\begin{abstract}
+\noindent DEFINE-UK is the ecological stock-flow consistent (E-SFC) model of the United Kingdom built at SOAS by \citet{georgedafermos2026manual}: a closed monetary accounting system for seven institutional sectors, coupled to an energy and emissions block, with UK-specific housing and power-generation structures through which climate policy reaches the financial system. It is the only member of the PolicyEngine Macro suite that answers the question \emph{who bears a climate-policy scenario}. This paper is its replication record, and it is deliberately a record of a ceiling as much as of a result. The upstream repository is public but carries \textbf{no licence}, so nothing is vendored or hosted: the adapter fetches it at pinned commit \texttt{846081a} at runtime and runs the authors' unmodified R. Four published-target gates were attempted. The manual's Table~4 baseline macro block \textbf{passes} --- population and labour force exact to 0.01\,m, growth matching to 0.06\,pp at 2030 and 2040 --- but the 2025 growth miss of $-0.30$\,pp clears a $\pm0.31$\,pp gate \emph{that was set after the run}, so that year's pass is a statement about the tolerance, not about agreement, and it is labelled as such at every point of claim. The baseline emissions path \textbf{diverges} and is pinned rather than tolerated: 393 vs 407 MtCO$_2$e/yr in 2025, 343 vs 382 in 2030, 249 vs 324 by 2040 ($-3.5$, $-10.2$, $-23.3$ per cent, computed from the pinned ratios rather than the rounded levels). The two scenario targets are \textbf{closed at the achievable ceiling, not passed}: no machine-readable numeric scenario results are published for version 1.1, so what is gated is scenario \emph{design} --- the exact policy-switch set each cached scenario toggles --- plus two coarse anchors from a pre-1.0 conference version, which are regression pins and not agreement tests; the emissions anchor admits 330--352 for a published claim of ``just under 350'', so a run at 351 would pass a gate named for a statement it violates. Because baseline levels sit far from outturns (2025 growth of 4.66 per cent against an ONS 1.31) and the manual itself says the baseline ``should also not be seen as a prediction or forecast'', the adapter serves annualised scenario-minus-baseline \emph{deltas, never levels}. A cumulative green-public-investment multiplier of \textbf{1.78} is reported as ours and as \emph{not comparable} to any published multiplier, for three reasons that all push the same way. In parallel, a clean-room Python reimplementation of the published equations --- the only route to a hostable model given the licence --- has passed milestone~1 (62 collected accounting tests, 44 of them Table~1/2 row and column identities) and stands at 118 contiguous equations, Eqs.~(21)--(138), on milestone~2, which is \textbf{in progress and not passed}. Thirteen defects in the manual are carried as machine-readable gap records rather than absorbed into tolerances; an August 2026 audit re-characterised four of them and withdrew one claim outright.
+\end{abstract}
+
+\medskip
+\noindent\textbf{Keywords:} ecological macroeconomics, stock-flow consistent modelling, climate policy, replication, clean-room reimplementation, open source
+
+\newpage
+\input{sections/intro}
+\input{sections/literature}
+\input{sections/model}
+\input{sections/source}
+\input{sections/implementation}
+\input{sections/validation}
+\input{sections/calibration}
+\input{sections/reimplementation}
+\input{sections/limitations}
+
+\input{sections/references}
+
+\input{sections/appendix}
+
+\end{document}
diff --git a/papers/define-uk/references.bib b/papers/define-uk/references.bib
new file mode 100644
index 00000000..374c07b8
--- /dev/null
+++ b/papers/define-uk/references.bib
@@ -0,0 +1,184 @@
+@misc{georgedafermos2026manual,
+  author       = {George, Adam and Dafermos, Yannis},
+  title        = {{DEFINE-UK} Model Manual, Version 1.1},
+  howpublished = {SOAS University of London},
+  year         = {2026},
+  month        = {April},
+  url          = {https://define-model.org/wp-content/uploads/2026/04/define_uk_1.1.pdf}
+}
+
+@techreport{georgedafermos2026scenario,
+  author      = {George, Adam and Dafermos, Yannis},
+  title       = {Green Fiscal Policy in the {UK}: A Scenario Analysis},
+  institution = {SSRN},
+  year        = {2026},
+  type        = {Working paper},
+  note        = {SSRN abstract 6541398; {DEFINE-UK} 1.0. Access-restricted},
+  url         = {https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6541398}
+}
+
+@techreport{georgedafermos2026mixes,
+  author      = {George, Adam and Dafermos, Yannis},
+  title       = {Evaluating Climate Policy Mixes in the {UK}: An {E-SFC} Approach},
+  institution = {SSRN},
+  year        = {2026},
+  type        = {Working paper},
+  note        = {SSRN abstract 6588778; {DEFINE-UK} 1.1. Access-restricted},
+  url         = {https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6588778}
+}
+
+@techreport{georgedafermos2023fmm,
+  author      = {George, Adam and Dafermos, Yannis},
+  title       = {Green Fiscal Policy in an Empirical {UK} {E-SFC} Model},
+  institution = {Forum for Macroeconomics and Macroeconomic Policies (FMM) Conference},
+  year        = {2023},
+  month       = {October},
+  note        = {Conference version, marked ``preliminary and incomplete''},
+  url         = {https://www.imk-boeckler.de/data/downloads/IMK/FMM\%20Konferenz\%202023/v_2023_10_20_george.pdf}
+}
+
+@article{dafermos2017,
+  author  = {Dafermos, Yannis and Nikolaidi, Maria and Galanis, Giorgos},
+  title   = {A Stock-Flow-Fund Ecological Macroeconomic Model},
+  journal = {Ecological Economics},
+  volume  = {131},
+  pages   = {191--207},
+  year    = {2017}
+}
+
+@article{dafermos2018,
+  author  = {Dafermos, Yannis and Nikolaidi, Maria and Galanis, Giorgos},
+  title   = {Climate Change, Financial Stability and Monetary Policy},
+  journal = {Ecological Economics},
+  volume  = {152},
+  pages   = {219--234},
+  year    = {2018}
+}
+
+@book{godley2007,
+  author    = {Godley, Wynne and Lavoie, Marc},
+  title     = {Monetary Economics: An Integrated Approach to Credit, Money, Income, Production and Wealth},
+  publisher = {Palgrave Macmillan},
+  address   = {Basingstoke},
+  year      = {2007}
+}
+
+@article{caverzasi2015,
+  author  = {Caverzasi, Eugenio and Godin, Antoine},
+  title   = {Post-{K}eynesian Stock-Flow-Consistent Modelling: A Survey},
+  journal = {Cambridge Journal of Economics},
+  volume  = {39},
+  number  = {1},
+  pages   = {157--187},
+  year    = {2015}
+}
+
+@article{nikiforos2017,
+  author  = {Nikiforos, Michalis and Zezza, Gennaro},
+  title   = {Stock-Flow Consistent Macroeconomic Models: A Survey},
+  journal = {Journal of Economic Surveys},
+  volume  = {31},
+  number  = {5},
+  pages   = {1204--1239},
+  year    = {2017}
+}
+
+@article{battiston2017,
+  author  = {Battiston, Stefano and Mandel, Antoine and Monasterolo, Irene and Schütze, Franziska and Visentin, Gabriele},
+  title   = {A Climate Stress-Test of the Financial System},
+  journal = {Nature Climate Change},
+  volume  = {7},
+  pages   = {283--288},
+  year    = {2017}
+}
+
+@misc{carney2015,
+  author       = {Carney, Mark},
+  title        = {Breaking the Tragedy of the Horizon --- Climate Change and Financial Stability},
+  howpublished = {Speech at Lloyd's of London, 29 September},
+  year         = {2015},
+  url          = {https://www.bankofengland.co.uk/speech/2015/breaking-the-tragedy-of-the-horizon-climate-change-and-financial-stability}
+}
+
+@book{kaldor1966,
+  author    = {Kaldor, Nicholas},
+  title     = {Causes of the Slow Rate of Economic Growth of the United Kingdom: An Inaugural Lecture},
+  publisher = {Cambridge University Press},
+  address   = {Cambridge},
+  year      = {1966}
+}
+
+@techreport{batini2014,
+  author      = {Batini, Nicoletta and Eyraud, Luc and Forni, Lorenzo and Weber, Anke},
+  title       = {Fiscal Multipliers: Size, Determinants, and Use in Macroeconomic Projections},
+  institution = {International Monetary Fund},
+  year        = {2014},
+  type        = {Technical Notes and Manuals},
+  number      = {14/04}
+}
+
+@article{dewald1986,
+  author  = {Dewald, William G. and Thursby, Jerry G. and Anderson, Richard G.},
+  title   = {Replication in Empirical Economics: The {Journal of Money, Credit and Banking} Project},
+  journal = {American Economic Review},
+  volume  = {76},
+  number  = {4},
+  pages   = {587--603},
+  year    = {1986}
+}
+
+@misc{policyengine,
+  author       = {Ghenis, Max and {PolicyEngine contributors}},
+  title        = {{PolicyEngine}: Open-Source Tax--Benefit Microsimulation and Macroeconomic Models},
+  howpublished = {\url{https://policyengine.org}},
+  year         = {2025}
+}
+
+@techreport{ahmadi2026boesvar,
+  author      = {Ahmadi, Vahid},
+  title       = {An Open Replication of the {Bank of England} Structural {VAR}},
+  institution = {PolicyEngine},
+  year        = {2026},
+  type        = {PolicyEngine Macro working paper},
+  url         = {https://policyengine-macro.vercel.app/papers/boe-svar}
+}
+
+@techreport{ahmadi2026obr,
+  author      = {Ahmadi, Vahid},
+  title       = {An Open {OBR} Macro Model for {UK} Fiscal Scoring},
+  institution = {PolicyEngine},
+  year        = {2026},
+  type        = {PolicyEngine Macro working paper},
+  url         = {https://policyengine-macro.vercel.app/papers/obr-macro}
+}
+
+@misc{ons2026gdp,
+  author       = {{Office for National Statistics}},
+  title        = {Gross Domestic Product: Chained Volume Measures},
+  howpublished = {Quarterly national accounts, series {ABMI}; vintage 26 July 2026},
+  year         = {2026},
+  url          = {https://www.ons.gov.uk/economy/grossdomesticproductgdp}
+}
+
+@misc{ons2026lms,
+  author       = {{Office for National Statistics}},
+  title        = {Unemployment Rate (Aged 16 and Over, Seasonally Adjusted), Series {MGSX}},
+  howpublished = {Labour market statistics; vintage 26 July 2026},
+  year         = {2026},
+  url          = {https://www.ons.gov.uk/employmentandlabourmarket/peoplenotinwork/unemployment}
+}
+
+@misc{desnz2025ghg,
+  author       = {{Department for Energy Security and Net Zero}},
+  title        = {{UK} Territorial Greenhouse Gas Emissions National Statistics},
+  howpublished = {2024 provisional figures; 2023 final figures},
+  year         = {2025},
+  url          = {https://www.gov.uk/government/collections/uk-greenhouse-gas-emissions-statistics}
+}
+
+@techreport{obr2026efo,
+  author      = {{Office for Budget Responsibility}},
+  title       = {Economic and Fiscal Outlook, March 2026},
+  institution = {Office for Budget Responsibility},
+  year        = {2026}
+}
diff --git a/papers/define-uk/sections/appendix.tex b/papers/define-uk/sections/appendix.tex
new file mode 100644
index 00000000..a8162cd1
--- /dev/null
+++ b/papers/define-uk/sections/appendix.tex
@@ -0,0 +1,104 @@
+\appendix
+\section*{Appendices}
+\addcontentsline{toc}{section}{Appendices}
+
+\section{The published baseline table}
+\label{app:table4}
+
+Table~\ref{tab:table4} reproduces the manual's Table~4 \citep[p.~50]{georgedafermos2026manual} in full, since it is the only published numeric result this replication has to work with and only its first four rows plus emissions are gated. Its note reads: ``All quarterly values are annualised and the mean and standard deviation are calculated from 2025-2040.'' It does not state \emph{which} annualisation, which is the unresolved convention behind the 2025 growth gap of Section~\ref{sec:target1a}.
+
+\begin{table}[h]
+\centering
+\caption{DEFINE-UK Model Manual v1.1, Table 4: key characteristics of the baseline scenario}
+\label{tab:table4}
+\footnotesize
+\begin{tabular}{@{}lrrrrr@{}}
+\toprule
+Variable & 2025 & 2030 & 2040 & Mean & St.\ dev. \\
+\midrule
+Real GDP growth (\%) & 4.96 & 2.35 & 2.01 & 2.35 & 0.71 \\
+Unemployment (\%) & 4.31 & 4.73 & 4.72 & 4.71 & 0.14 \\
+Population (millions) & 55.66 & 57.74 & 61.61 & 58.68 & 1.87 \\
+Labour force (millions) & 35.97 & 37.12 & 39.13 & 37.62 & 1.00 \\
+Proportion of non-fossil electricity generation (\%) & 57.93 & 61.77 & 77.14 & 65.92 & 6.35 \\
+Total emissions (MtCO$_2$e/year) & 407.32 & 381.76 & 324.37 & 367.10 & 25.46 \\
+Emission price (\pounds/tCO$_2$e) & 8.01 & 10.58 & 15.71 & 11.86 & 2.44 \\
+Green investment (\% of GDP) & 1.00 & 1.07 & 1.30 & 1.13 & 0.10 \\
+NFC default rate (\%) & 0.28 & 0.33 & 0.45 & 0.36 & 0.05 \\
+NFC credit rationing (\%) & 3.81 & 2.56 & 4.79 & 3.49 & 0.74 \\
+Power sector default rate (\%) & 0.16 & 0.11 & 0.13 & 0.12 & 0.01 \\
+Power sector credit rationing (\%) & 47.22 & 35.44 & 21.09 & 30.64 & 9.04 \\
+Imports (\% of GDP) & 39.99 & 39.97 & 41.18 & 40.31 & 0.51 \\
+Exports (\% of GDP) & 38.58 & 40.19 & 44.15 & 41.21 & 1.80 \\
+Real effective exchange rate (index, 2022 $=$ 1) & 1.00 & 1.00 & 0.98 & 0.99 & 0.01 \\
+\bottomrule
+\end{tabular}
+\par\smallskip
+\parbox{0.95\textwidth}{\footnotesize \emph{Notes:} ``Population'' is the population aged 16 and over, not total population. Rows 1--4 are the macro block of target 1a (Section~\ref{sec:target1a}); row 6 is target 1b (Section~\ref{sec:target1b}). The remaining nine rows are published but not gated: no tolerance has been set for them and no verdict is claimed on them here.}
+\end{table}
+
+\section{The thirteen gap records}
+\label{app:gaps}
+
+Table~\ref{tab:gaps} lists every machine-readable gap record carried by the clean-room reimplementation as of 12 August 2026. Each is pinned by a test that fails if the situation changes --- including if the DEFINE team publishes a value for a defaulted parameter.
+
+\begin{table}[h]
+\centering
+\caption{Machine-readable gap records in the clean-room reimplementation}
+\label{tab:gaps}
+\scriptsize
+\begin{tabularx}{\textwidth}{@{}llX@{}}
+\toprule
+Section & Record & What it records \\
+\midrule
+\S3.2 & intercept of Eq.~(31) & Table~5 never tabulates the Kaldor--Verdoorn intercept; the $\alpha_0$ block ends one entry earlier. Defaulted to 0.0. \\
+\addlinespace
+\S3.3.1 & direct-energy symbols & Eqs.~(59)--(63) use a ``direct energy'' subscript family that appears nowhere in Tables~5--6, while the tables carry a non-electric-energy family that appears nowhere in the body. They are the same variables: Eqs.~(60) and (62) reproduce the tabulated values to $1.1\times10^{-5}$ and $5.8\times10^{-5}$ under that identification. The tabulated names are used. \\
+\S3.3.1 & split green capital stocks & Eq.~(68) sums the NFC and government real green capital stocks; Table~6 tabulates only their sum. Nothing defaulted --- both are exogenous to the section --- and the identity is checked against the tabulated sum instead. \\
+\addlinespace
+\S3.3.2 & power-sector depreciation rate & Seven equations use one rate; Table~6 tabulates only the two split rates, which are equal, so the identification is unambiguous and the rate is seeded from them. If a revision ever splits them, two equations need rewriting rather than reseeding. \\
+\S3.3.2 & intercept of Eq.~(96) & Not in Table~5. The table carries the two intercepts of a \emph{split} investment rule that v1.1's body replaced, and neither can be substituted without guessing which. Defaulted to 0.0; a steady-state reading of Table~6 implies about 0.0210, recorded and deliberately not used. \\
+\S3.3.2 & Eq.~(99) split parameters and profit rates & Two logistic parameters untabulated and two \emph{variables} --- the fossil and non-fossil capital profit rates --- never defined anywhere in the manual. All four defaulted to 0.0, which forces a 50:50 green/fossil investment split against the 69:31 Table~6 implies. The largest hole in the section, since this split is what steers investment green. \\
+\S3.3.2 & Eq.~(138) credit-rationing slopes & Three of four slopes untabulated. The NFC equation has identical structure and its slopes \emph{are} tabulated, and Table~5 reuses NFC values for the power sector in at least nine other places --- but reusing them here would be our inference, not the manual's instruction, and it does not reproduce Table~6 anyway. Defaulted to 0.0. \\
+\S3.3.2 & government non-fossil capital & Eq.~(86) puts it in a denominator; Table~6 does not tabulate it. Measured to be zero in the baseline twice over, via Eq.~(86) and via \S3.1's Eq.~(5). It is a scenario instrument. \\
+\S3.3.2 & forward-looking switches & Eqs.~(88)--(89) introduce four quantities \S5 never tabulates (ban period, credibility, planning horizon, hyperbolic discount); the manual gives only bounds. Treated as scenario switches with an inert baseline. \\
+\S3.3.2 & expected capital path & \emph{Our choice, not a manual defect.} Eq.~(89) averages expected utilisation over the planning horizon, but the manual never says what capital stock the sector expects to hold, and the model carries no forecast of it. Real capital and electricity demand are held at current values; inert whenever credibility is zero. \\
+\S3.3.2 & Eq.~(88) inequality direction & The printed condition and the prose above it point opposite ways. The printed inequality is implemented --- transcription beats interpretation --- with the baseline switch set so the no-ban case returns current utilisation, which is what the manual's own annotation asserts. \\
+\S3.3.2 & unprinted electricity-price rule & \S5 calibrates a long-run electricity-price formation rule --- a switch quarter and a long-run price --- that \S3 never prints. Nothing is implemented, because nothing is printed to implement. The most economical account of Eq.~(84)'s factor of 3.04. \\
+\S3.3.2 & five untabulated endogenous variables & Table~6 omits five variables the section determines. Three are recoverable from what Table~6 does print and are checked that way; one recovery is confirmed by Eq.~(106) reproducing a tabulated lending figure to $2.7\times10^{-5}$. \\
+\bottomrule
+\end{tabularx}
+\end{table}
+
+\section{Run record}
+\label{app:runrecord}
+
+Every entry below is dated and held in the adapter's \texttt{VALIDATION.md}. Changing any pinned reference number requires editing the committed reference artifact deliberately and adding an entry here --- never editing a test.
+
+\footnotesize
+\begin{longtable}{@{}l>{\raggedright\arraybackslash}p{0.16\textwidth}>{\raggedright\arraybackslash}p{0.53\textwidth}@{}}
+\caption{Dated run record} \label{tab:runrecord} \\
+\toprule
+Date & Basis & Result \\
+\midrule
+\endfirsthead
+\toprule
+Date & Basis & Result \\
+\midrule
+\endhead
+\bottomrule
+\endfoot
+2026-08-01 & \texttt{846081a}, R 4.3.0 & Full upstream notebook renders end to end, unmodified, with an \texttt{rstudioapi} shim the only accommodation. 151 output files. \textbf{Execution is not validation}: no comparison against published figures yet, so every target remains pending. \\
+\addlinespace
+2026-08-04 & \texttt{846081a}, cached run & Table~4 replication run. Macro block passes (population and labour force exact to 0.01\,m; growth exact at 2030/2040, $-0.30$\,pp at 2025; unemployment within 0.2\,pp; 2025--40 means and standard deviations match). Emissions diverge as target 1b records. \\
+\addlinespace
+2026-08-04 & clean-room, manual only & \textbf{Milestone 1 PASS.} \S2.2 matrices and \S5 Tables~5--6 transcribed in full; every Table~1/2 row and column identity holds on the \S5 initial values at the manual's four-significant-figure printing precision. The rest-of-world lending inconsistency pinned exactly. \\
+\addlinespace
+2026-08-04 & \texttt{846081a}, cached run & Scenario surface gated against the pinned oracle: all 22 scenario files across four blocks registered, annualised delta paths only with mandatory caveats, 11 pinned delta points, sign and ordering structure pinned. \\
+\addlinespace
+2026-08-05 & \texttt{846081a}, cached run & \textbf{Targets 2 and 3 closed at the achievable ceiling}; scenario-design gate added with the two FMM 2023 anchors; external comparator table computed and committed; the 1.78 multiplier computed, replacing the unusable upstream multiplier table; published sources catalogued. \\
+\addlinespace
+2026-08-11 & clean-room, manual only & \S3.2, \S3.3.1 and \S3.3.2 landed: 118 contiguous equations, Eqs.~(21)--(138). \S3.3.3 and the oracle gate outstanding, so \textbf{milestone 2 is not passed}. \\
+\addlinespace
+2026-08-12 & audit, manual + committed artifacts & All 661 transcribed \S5 values re-checked against the manual page each source comment names; every pinned discrepancy recomputed independently of the sector modules. All arithmetic reproduced. Four findings re-characterised as first-period jumps, one claim withdrawn outright, five discrepancies unified into one deflator finding, one new gap found. Suite: 236 passed, 59 skipped. \\
+\end{longtable}
diff --git a/papers/define-uk/sections/calibration.tex b/papers/define-uk/sections/calibration.tex
new file mode 100644
index 00000000..248883e3
--- /dev/null
+++ b/papers/define-uk/sections/calibration.tex
@@ -0,0 +1,51 @@
+\section{External calibration and the multiplier}
+\label{sec:calibration}
+
+Replication against a publication and calibration against the world are different questions, and a model can pass the first while failing the second badly. DEFINE-UK does. This section reports the second, and then reports the one number from the model that has circulated most widely --- a cumulative green-investment multiplier of 1.78 --- with the reasons it cannot be read against anybody else's multiplier.
+
+\subsection{The baseline against outturns}
+
+Since 5 August 2026 this comparison is an artifact rather than a paragraph. A function in the adapter recomputes every row from the cached pinned run against vintage-exact external observations; the result is committed as \texttt{validation/baseline\_vs\_external.csv}; and a test fails on any drift between the committed table and a recomputation. Table~\ref{tab:calibration} reproduces it.
+
+\begin{table}[t]
+\centering
+\caption{DEFINE-UK baseline against external observations}
+\label{tab:calibration}
+\footnotesize
+\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}Xrrl>{\raggedright\arraybackslash}X@{}}
+\toprule
+Quantity & Model & External & Units & Verdict \\
+\midrule
+Real GDP growth, 2025 & 4.66 & 1.31 & \% y/y & far above outturn; never a forecast \\
+Real GDP growth, mean 2025--40 & 2.4 & 1.65 & \% y/y & high vs official long-run \\
+Unemployment, 2025 & 4.39 & 4.88 & \% & below outturn \\
+Population 16+, 2025 & 55.66 & 56.0 & millions & consistent \\
+Labour force, 2025 & 35.97 & 34.5 & millions & slightly high \\
+Emissions, 2024 & 401.5 & 371.0 & MtCO$_2$e/yr & above the actuals it should start from \\
+Emissions, 2030 vs NDC & 342.8 & 260.0 & MtCO$_2$e/yr & short of the NDC by design \\
+\bottomrule
+\end{tabularx}
+\par\smallskip
+\parbox{0.97\textwidth}{\footnotesize \emph{Notes:} Model column from the cached pinned run at commit \texttt{846081a}, annual-mean convention --- which is why 2025 growth reads 4.66 where the manual's Table~4 prints 4.96, the same anchoring gap recorded under target 1a (Section~\ref{sec:target1a}). Sources: ONS quarterly national accounts, vintage 26 July 2026 \citep{ons2026gdp}; ONS series MGSX, same vintage, 2025 quarterly path 4.6/4.7/5.0/5.2 giving an annual mean of 4.88 \citep{ons2026lms}; ONS 16+ population and labour force; DESNZ UK territorial greenhouse gas emissions, 2024 provisional, with 2023 final at 384 \citep{desnz2025ghg}; OBR March 2026 long-run growth assumption \citep{obr2026efo}; and the UK NDC of 68 per cent below 1990 by 2030. The final row is not a failure: the manual states that its current-policies baseline falls short of the NDC by design.}
+\end{table}
+
+The 2025 growth gap is the one to keep in view: 4.66 per cent against an ONS outturn of 1.31, a gap of well over three percentage points on the single most familiar macroeconomic quantity there is. Unemployment sits about half a point below the outturn, the labour force about a million and a half high, and 2024 emissions some 30 MtCO$_2$e above the DESNZ actuals the model should be starting from. None of this is presented as a defect of the model as its authors intend it to be used --- the manual is explicit that the baseline is not a forecast --- but it is decisive for how the adapter may serve it. It is the measured basis for Section~\ref{sec:deltas}'s deltas-only rule.
+
+\subsection{The 1.78 multiplier, and why it is not comparable to any published one}
+\label{sec:multiplier}
+
+The model's cumulative green-public-investment multiplier is \textbf{1.78} by \emph{our own} computation: cumulative $\Delta$GDP of 209.9 over cumulative $\Delta$SPEND\_GVT of 118.2, nominal, over the full simulation, on the green-public-investment-plus-green-bonds scenario against its baseline. The arithmetic reproduces ($209.9/118.2 = 1.7758$) and the number is pinned in the test suite to $\pm0.01$ with its cumulants pinned to $\pm0.5$.
+
+Two labels must travel with it. The first is that \textbf{the number is ours, not the authors'}: no published DEFINE-UK multiplier corresponds to it, and it is a quantity this replication defined and computed, not one it reproduced. The second is that \textbf{it is not like-for-like against any published multiplier}, on three separate grounds, all of which push in the same direction.
+
+\paragraph{Horizon.} Ours is a cumulative ratio over the whole simulation to 2040. Published green-spending multipliers, and the OBR's capital-spending impact multiplier of roughly 1.0, are short-horizon figures --- impact or first-few-year. A long-horizon cumulative ratio is \emph{expected} to sit above them, so setting 1.78 beside a short-horizon number and reading a disagreement is a units error dressed as a finding.
+
+\paragraph{Denominator.} SPEND\_GVT is total government spending excluding wages. The manual's Table~6 components imply it is other government consumption plus social benefits plus government gross capital formation ($58.03 + 82.8 + 20.76 = 161.59$ against a tabulated 161.6), of which government investment --- the actual policy lever --- is only \textbf{12.8 per cent}. The denominator is therefore an endogenous aggregate rather than the policy impulse, and it moves the wrong way for the ratio: the scenario cuts unemployment, social benefits fall, the denominator shrinks, and the measured multiplier is \textbf{biased up}. The clean comparison --- cumulative $\Delta$GDP over cumulative $\Delta\mathrm{GCF\_GVTG}$, the actual impulse --- \textbf{has never been computed}. Until it is, 1.78 should be read as upper-leaning, and that is stated wherever the number is served.
+
+\paragraph{Closure.} As Section~\ref{sec:model} sets out, the model is demand-led with Kaldor--Verdoorn productivity (Eq.~(31)) and real GDP not held to a supply ceiling (Eq.~(38)). A multiplier above one over fifteen years is what this closure \emph{implies}. It is not a symptom of an implementation error, and it is not a finding about the world; it is a property of the model one has chosen to run. That is why the number is labelled rather than corrected.
+
+These three caveats are not editorial: they are carried in code as a constant and gated hermetically by a test that fails if the number is served without them.
+
+\paragraph{A withdrawn comparator.} Earlier versions of this project's surfaces set 1.78 beside ``IMF green-spending multipliers of 1.1--1.5'', attributed to a 2021 IMF publication. That publication could not be located in either repository or anywhere else, and the attribution has been \textbf{withdrawn rather than repaired}, on the principle that an unidentifiable citation is worse than none. The only IMF reference this project actually holds is \citet{batini2014}, which is a different paper reporting a different quantity: first-year multipliers of 0--1 for advanced economies. No published multiplier is quoted alongside 1.78 in this paper, because none of them measures the same object.
+
+\paragraph{The upstream multiplier table is unusable.} The upstream run also produces its own \texttt{tables/Multiplier\_Summary.csv}, and it is documented unusable rather than quietly ignored. Its impact, four-quarter and eight-quarter multiplier columns are \emph{identical across all eight scenarios} ($-32.22$, $-0.88$, $2.6$), which no correct scenario-specific multiplier can be; and its cumulative government-spending figure of 104.22 matches the cumulative delta of \emph{no} variable in the scenario file. One column does reconcile --- its cumulative GDP delta of 250.76 is the cumulative quarterly real-GDP delta of the green-public-investment scenario, 250.6 by our own sum --- which is what makes the failure of the others diagnostic rather than ambiguous. Neither the table nor the multiplier of roughly 2.4 once derived from it is quoted anywhere.
diff --git a/papers/define-uk/sections/implementation.tex b/papers/define-uk/sections/implementation.tex
new file mode 100644
index 00000000..64932003
--- /dev/null
+++ b/papers/define-uk/sections/implementation.tex
@@ -0,0 +1,37 @@
+\section{The adapter: how the model is run, and what it serves}
+\label{sec:implementation}
+
+\subsection{Code architecture}
+
+The adapter is a small Python package, \texttt{define\_uk} (src layout), at \url{https://github.com/PolicyEngine/define-uk-model}. It has two halves that never touch: a harness that runs the authors' R code, and a clean-room reimplementation that must not read it.
+
+\begin{itemize}
+\item \texttt{upstream.py} --- clones the upstream repository into a local cache at run time at the pinned commit \texttt{846081a580a6033159d5c421632ad8f0b30d0ded}, and nothing else. Bumping the pin is an explicit, reviewed change, because the validation targets are tied to that revision.
+\item \texttt{runner.py} --- renders the upstream R Markdown notebook in place, unmodified.
+\item \texttt{scenarios.py} --- the curated scenario surface over the \emph{cached} outputs of that run. It never runs R; if no cached run exists it raises with instructions to produce one.
+\item \texttt{validation.py} --- the computed comparators: the baseline-versus-external calibration table, the green-public-investment multiplier, and the caveats that travel with it.
+\item \texttt{model/} --- the clean-room reimplementation (Section~\ref{sec:reimplementation}): an equation registry that requires a manual reference on every equation, a Gauss--Seidel per-period solver, one module per manual \S3 section, the \S5 parameter and initial-value tables, and the \S2.2 accounting matrices.
+\end{itemize}
+
+\subsection{The scenario surface}
+
+The pinned run produces 22 scenario files across four blocks: government investment, housing regulation, power-sector regulation and mixed packages. The power-sector block is run under five expectation regimes --- mutual trust, poor credibility, false confidence, no forward guidance, and unexpected enforcement --- of which mutual trust is treated as canonical and the others carry an explicit variant suffix. Each of the 22 is registered, and each is checked against the policy switches its published description claims (Section~\ref{sec:targets23}).
+
+The surface returns \emph{annualised scenario-minus-baseline delta paths only}. The convention is fixed and tested: scenario minus baseline, annual mean of quarterly levels, calendar-anchored 1987Q1--2040Q4. Eleven delta points across four scenarios and three variables are pinned to relative tolerance $10^{-6}$ against the cached run, and the sign and ordering structure --- deltas finite, baselines identical across block folders, emissions falling in every scenario by 2035, the mixed package dominating --- is pinned exactly.
+
+\subsection{Deltas, never levels}
+\label{sec:deltas}
+
+The decision to serve deltas and never levels is the single most consequential design choice in the adapter, and it is a direct reading of two facts established in Sections~\ref{sec:model} and \ref{sec:calibration}: the manual's own statement that its baseline ``should also not be seen as a prediction or forecast'', and the measured distance between the baseline and UK outturns --- 2025 real GDP growth of 4.66 per cent against an ONS 1.31, unemployment below the outturn, emissions above the DESNZ actuals they should start from.
+
+Enforcing that is not left to documentation. Every scenario result carries a result type of ``scenario deltas'' and a mandatory caveat list, held in code and gated by tests: that the exercise is experimental and gated at the ceiling described in Section~\ref{sec:targets23}; that baseline levels are not validated against outturns and must never be presented as forecasts; that the closure is demand-led, so multipliers sit above mainstream estimates; and that the pinned commit's baseline emissions path diverges from the manual's published table. Near-term DEFINE-UK levels are not competitive with the suite's OBR emulator \citep{ahmadi2026obr} or its Bank of England SVAR \citep{ahmadi2026boesvar}, and are never presented as forecasts.
+
+\subsection{Integration}
+
+Within PolicyEngine Macro the model is exposed through the \texttt{pe-macro} command-line interface --- \texttt{define-scenarios} to list the registry, \texttt{define-scenario} to produce delta paths --- marked local-only and experimental, for the licence reason of Section~\ref{sec:licence}. A further command bridges a scenario's household income deltas into PolicyEngine's UK microsimulation as a pre-tax earnings overlay, the same economic-assumptions pattern the suite uses for its other macro members, so that a climate-policy scenario can be read household by household through the tax and benefit system. Only numbers travel across that bridge; the unlicensed upstream code does not. The suite's reform-scoring entry point continues to refuse this model.
+
+\subsection{What continuous integration can and cannot check}
+
+Because the upstream is unlicensed, continuous integration never fetches or runs it, so the oracle comparisons skip there and run only where a cached pinned run exists. This would be a serious weakness if the gate were left at that, so the gate is split.
+
+Every pinned reference number the oracle tests compare against lives in one committed artifact, \texttt{validation/reference\_outputs.json}, rather than in test code --- so that a local run and a CI run gate identical numbers, and no tolerance or reference can drift silently inside a test. The \emph{hermetic} half of the suite runs everywhere, on every pull request, without R or any upstream fetch: it checks that the reference artifact is well-formed and internally consistent (for instance, that the committed multiplier reproduces from its committed cumulants), that it is pinned to the same commit the code is, that the committed calibration table keeps its exact schema, pinned external observations, gap arithmetic and headline divergences, that the scenario registry matches, and that the repository's own documentation still names the pinned commit. Changing any reference number requires editing the artifact deliberately, with a dated run-record entry --- never editing a test.
diff --git a/papers/define-uk/sections/intro.tex b/papers/define-uk/sections/intro.tex
new file mode 100644
index 00000000..bd9a6d60
--- /dev/null
+++ b/papers/define-uk/sections/intro.tex
@@ -0,0 +1,22 @@
+\section{Introduction}
+\label{sec:intro}
+
+Climate policy is a macrofinancial event. A fossil-fuel ban writes down capital that somebody owns and somebody else has lent against; a retrofit subsidy is a fiscal flow that lands on households with a particular distribution of housing stock; a decarbonisation path that misses its target does so through investment decisions taken under credit rationing. Most macroeconomic models cannot say any of this, because their accounting stops short of balance sheets or their environmental block is a reduced-form elasticity bolted onto a supply-side core. Ecological stock-flow consistent (E-SFC) models are built for exactly this joint question, and DEFINE-UK is the United Kingdom's.
+
+DEFINE-UK is documented in the \emph{DEFINE-UK Model Manual, Version 1.1} \citep{georgedafermos2026manual}, released April 2026 alongside version~1.0. It is a closed monetary accounting system covering households, non-financial corporations, the power-generation sector, monetary and non-monetary financial institutions, government and the rest of the world, with a non-sectoral production module, coupled to an energy and emissions block, and it carries two UK-specific structures --- a housing stock tracked by energy efficiency and a power sector with separate fossil and non-fossil capital --- through which climate policy reaches the financial system rather than merely the output gap. Version~1.1 added regulatory policies that directly depreciate affected capital (stranded assets as an explicit balance-sheet event), limited forward-looking behaviour towards policy announcements, direct government ownership of non-fossil power capital, and a green sovereign bond.
+
+Within PolicyEngine Macro \citep{policyengine}, an open suite of macroeconomic models for the UK and US, DEFINE-UK occupies a role no other member can fill: it is the only model in the suite that answers \emph{who bears a climate-policy scenario}. It is also the suite's most constrained member, and this paper is as much a record of those constraints as of what was reproduced.
+
+\paragraph{What this paper is.} The suite's standard is that a model earns its place by replication: run the published code or reimplement the published equations, then compare against the published results at recorded tolerances, and report the comparison with numbers rather than adjectives --- the same bar applied to the Bank of England SVAR \citep{ahmadi2026boesvar} and the OBR macroeconometric emulator \citep{ahmadi2026obr}. This paper applies that standard to DEFINE-UK and records, target by target, where it passes, where it diverges, and where the publications themselves impose a ceiling on what can be checked at all.
+
+\paragraph{Three constraints shape everything that follows.} The first is legal. The upstream repository is public but carries \textbf{no licence}, which means its code cannot be vendored, redistributed or hosted. The adapter therefore fetches it at pinned commit \texttt{846081a} at runtime and executes it as published, in R, unmodified --- and DEFINE-UK is the only member of the suite that is local-only, never hosted. The second is evidential. No machine-readable numeric scenario results are published for version~1.1: the manual's own results stop at the baseline table plus scenario \emph{design} parameters, and both companion papers are access-restricted. There is, in short, nothing to replicate on the scenario side, and Section~\ref{sec:validation} says so rather than manufacturing a comparison. The third is calibrational. The published baseline sits far from UK outturns --- 2025 real GDP growth of 4.66 per cent against an ONS 1.31 --- and the manual states plainly that its baseline ``should also not be seen as a prediction or forecast''. Taking that literally is what makes the adapter serve annualised scenario-minus-baseline \emph{deltas, never levels}.
+
+\paragraph{Contribution.} The contribution is threefold, and none of it is a forecast.
+
+First, \emph{a replication with its ceiling stated}. Section~\ref{sec:validation} reports four published-target gates. The manual's Table~4 macro block passes. Its emissions path diverges, and is pinned at the observed ratios so that further drift fails loudly rather than being absorbed. The two scenario targets are closed at the achievable ceiling and are explicitly not passes. Throughout, checks that cannot fail are separated from checks that can, at the point of claim: the 2025 growth pass clears a tolerance set after the run by 0.01\,pp, and the two conference-vintage anchors are bands we chose to fit a verbal quantity, one of which admits values that contradict the sentence it is named for.
+
+Second, \emph{a calibration and a multiplier reported with their non-comparabilities attached}. Section~\ref{sec:calibration} gives the baseline against ONS, DESNZ and OBR figures as a computed, regression-tested artifact rather than prose, and reports a cumulative green-public-investment multiplier of 1.78 that is ours, not the authors', and that is not like-for-like against any published multiplier on horizon, denominator or closure. A previously quoted IMF comparator is recorded as withdrawn.
+
+Third, \emph{a clean-room reimplementation in progress}. Because the upstream cannot be hosted, the route to a hostable DEFINE-UK is a from-scratch Python implementation of the published equations, with the upstream used strictly as a numerical output oracle and never read to write equations. Section~\ref{sec:reimplementation} reports milestone~1 passed and milestone~2 in progress --- 118 contiguous equations, Eqs.~(21)--(138) --- and treats the reimplementation as an instrument in its own right: reading the manual closely enough to run it has surfaced thirteen defects, which are carried as machine-readable gap records rather than papered over, and an audit of those records re-characterised four of them and withdrew one claim outright.
+
+\paragraph{Road map.} Section~\ref{sec:literature} places the model in the SFC, ecological-macroeconomics and climate-finance literatures. Section~\ref{sec:model} sets out the model as the manual specifies it. Section~\ref{sec:source} identifies the source publications, the licence position, and the replication scope. Section~\ref{sec:implementation} describes the adapter and what it serves. Section~\ref{sec:validation} is the replication record. Section~\ref{sec:calibration} covers external calibration and the multiplier. Section~\ref{sec:reimplementation} covers the clean-room track. Section~\ref{sec:limitations} concludes with limitations. The appendix reproduces the published baseline table, the full gap-record inventory, and the dated run record.
diff --git a/papers/define-uk/sections/limitations.tex b/papers/define-uk/sections/limitations.tex
new file mode 100644
index 00000000..5df53343
--- /dev/null
+++ b/papers/define-uk/sections/limitations.tex
@@ -0,0 +1,26 @@
+\section{Limitations and conclusion}
+\label{sec:limitations}
+
+Six limitations should frame any use of this model, and each is disclosed in the codebase as well as here.
+
+\paragraph{No scenario result has been replicated.} This is the limitation that matters most, because scenarios are what the model is for. No machine-readable numeric scenario results are published for version~1.1, so targets 2 and 3 are closed at the achievable ceiling rather than passed (Section~\ref{sec:targets23}). What is gated is scenario \emph{design} --- the exact policy-switch set each cached scenario toggles --- plus two coarse anchors from a pre-1.0 conference version that are regression pins, not agreement tests, one of which admits values contradicting the sentence it is named for. Nothing in this paper establishes that DEFINE-UK's scenario \emph{outputs} reproduce anything published. The condition for lifting this is specific: the authors publish scenario tables, or either SSRN paper becomes accessible.
+
+\paragraph{The baseline is not a forecast, and its levels are far from outturns.} 2025 real GDP growth of 4.66 per cent against an ONS 1.31; unemployment below the outturn; 2024 emissions some 30 MtCO$_2$e above the DESNZ actuals. The manual itself says the baseline ``should also not be seen as a prediction or forecast''. The adapter therefore serves deltas and never levels, and no DEFINE-UK number appears in any of the suite's forecast surfaces. Users who want UK levels should use the suite's OBR emulator or its Bank of England SVAR.
+
+\paragraph{One published match is a tolerance, not an agreement.} Target 1a passes, and the 0.06\,pp matches at 2030 and 2040 are real. The 2025 growth pass is not: it clears a $\pm0.31$\,pp gate set after the run by 0.01\,pp, on an annualisation convention Table~4 does not state precisely enough to resolve. It is reported as a pass of the gate and not as a match.
+
+\paragraph{The emissions path diverges, and the cause is not established.} The pinned commit runs 3.5 per cent below the published table in 2025 and 23.3 per cent below by 2040. Two checks rule out extraction error on this side, which leaves a vintage or calibration gap between the published table and this commit --- but which vintage produced Table~4 is not known here, and the question is with the authors. Anyone using the adapter's emissions deltas should know that the level path they are differenced from is not the published one.
+
+\paragraph{The multiplier is ours and is not comparable.} 1.78 is a quantity this replication defined, on a horizon, a denominator and a closure that all bias it upward relative to any published short-horizon, impulse-based multiplier (Section~\ref{sec:multiplier}). The clean number --- cumulative $\Delta$GDP over cumulative $\Delta\mathrm{GCF\_GVTG}$ --- has never been computed, and computing it is the obvious next piece of work. A previously quoted IMF comparator has been withdrawn as unlocatable.
+
+\paragraph{The reimplementation is early, and the model is local-only until it is not.} Milestone~1 is passed and milestone~2 is in progress with 118 of the manual's equations transcribed and its oracle gate unrun; \S3.3.3, the seven sectoral sections, the ecosystem block and the scenarios remain. Until it is complete, DEFINE-UK cannot be hosted at all, because the upstream is unlicensed. Two operational caveats attach: the full-notebook runtime and the upstream's R dependency set are untested in continuous integration, and the oracle half of the validation suite runs only where a cached pinned run exists.
+
+\paragraph{Not an official product.} This is an independent open-source replication and reimplementation, built from the published manual and papers. It is not produced by, endorsed by, or speaking for the DEFINE team; where this work and the publications disagree, the publications are ground truth and the disagreement is recorded as such. The model design belongs to its authors; the implementation is ours.
+
+\subsection*{Conclusion}
+
+The useful thing about DEFINE-UK for a replication programme is that it fails in an instructive way. The code runs, unmodified, at a pinned commit. One published table's macro block reproduces, two of its rows to 0.06\,pp. Another row of the same table diverges by 23 per cent at the horizon. And the results that would matter most --- what happens to output, emissions, employment and balance sheets under a fossil-fuel ban or a green investment programme --- cannot be checked at all, not because the model is wrong but because the numbers were never published.
+
+That combination is common and it is usually reported as a pass. The alternative this paper argues for is a record with three separations held throughout: between checks that can fail and checks that cannot, at the point of claim rather than in a footnote; between a number we computed and a number somebody published; and between a ceiling imposed by the publications and a limitation of the replication, with the lifting condition named in each case. Applied here, that record supports a narrow set of claims --- a baseline macro block reproduced, an emissions divergence pinned, scenario design verified, deltas served with mandatory caveats, and a clean-room reimplementation 118 equations into its second milestone --- and refuses a much larger set that the same underlying work could have been made to look like.
+
+Thirteen defects in a published manual, four of them re-characterised and one claim withdrawn on audit, is the other half of the return. Reimplementing published equations is expensive, and its usual justification is licensing. The justification on display here is different: it is the only reading of a model close enough to find out that a section as published cannot be simulated forward, that a price identity is off by a factor of three because an equation is missing from the document, and that one table's capital block is deflated at the wrong price. None of that was visible from running the code.
diff --git a/papers/define-uk/sections/literature.tex b/papers/define-uk/sections/literature.tex
new file mode 100644
index 00000000..f52c641f
--- /dev/null
+++ b/papers/define-uk/sections/literature.tex
@@ -0,0 +1,28 @@
+\section{Related literature}
+\label{sec:literature}
+
+DEFINE-UK sits at the junction of three literatures: stock-flow consistent macroeconomic modelling, ecological macroeconomics, and the climate-and-financial-stability agenda. A fourth --- the practice of replication itself --- is what this paper contributes to.
+
+\subsection{Stock-flow consistent modelling}
+
+The stock-flow consistent (SFC) tradition descends from \citet{godley2007}, whose organising insight is that accounting comes before theory: every monetary flow leaves one sector's account and enters another's, and every flow accumulates into a stock, so that a transactions matrix sums to zero in every row and a balance-sheet matrix sums to zero in every instrument row. Nothing appears from nowhere and nothing leaks out. The discipline is unusually strong for a macroeconomic modelling framework, because it converts a large class of specification errors into arithmetic failures that a matrix will not close over.
+
+Two surveys map the field the model belongs to. \citet{caverzasi2015} trace the post-Keynesian SFC programme from its Cambridge origins through the empirical and open-economy extensions, and note the characteristic methodological trade: SFC models buy accounting completeness at the cost of large parameter sets calibrated rather than estimated, which makes their published tables --- initial values, parameter provenance --- an unusually large part of what a reader must be able to check. \citet{nikiforos2017} survey the same literature with emphasis on how such models are closed, and on the recurring result that a demand-led closure delivers larger fiscal multipliers than a supply-constrained one, because output is not returned to a potential path by assumption. Section~\ref{sec:calibration} shows that this is not an abstract point for DEFINE-UK: it is one of the three reasons its green-investment multiplier cannot be read against short-horizon published figures.
+
+The empirical relevance of these surveys to the present paper is direct. In an SFC model calibrated rather than estimated, the published parameter and initial-value tables \emph{are} the specification. Section~\ref{sec:reimplementation} is in large part a report on what happens when one takes that seriously enough to transcribe those tables in full and demand that the printed equations reproduce the printed values.
+
+\subsection{Ecological macroeconomics and the DEFINE framework}
+
+The specific model replicated here belongs to the DEFINE family. \citet{dafermos2017} introduced the stock-flow-fund framework that DEFINE implements: a synthesis of Godley--Lavoie monetary accounting with the material flow-fund accounting of ecological economics, so that the same model tracks money and matter under separate but coupled consistency conditions. \citet{dafermos2018} extended it to the climate--finance channel, showing how physical damages and transition dynamics feed through firm leverage and bank balance sheets to credit availability, and how monetary policy interacts with that loop --- the mechanism DEFINE-UK's credit-rationing and default blocks inherit.
+
+DEFINE-UK \citep{georgedafermos2026manual} is the United Kingdom calibration and extension of that framework. Its distinguishing content relative to the generic case is the pair of UK-specific structural blocks described in Section~\ref{sec:model}: a housing block that tracks the dwelling stock by energy efficiency, so that retrofit policy operates through the stock rather than through an assumed elasticity, and a power-generation sector with separate fossil and non-fossil capital, an investment trigger tied to capacity utilisation, and --- new in version~1.1 --- regulatory instruments that directly depreciate fossil capital.
+
+\subsection{Climate policy, stranded assets and financial stability}
+
+The reason to want balance sheets in a climate model is the transition-risk agenda that \citet{carney2015} named: the losses from a rapid, credible decarbonisation fall not on the emitting capital alone but on the claims written against it, held by institutions whose solvency and lending behaviour then feed back into the real economy. \citet{battiston2017} made that concrete with a network stress test showing how climate-policy shocks propagate from exposed sectors through equity and credit holdings to the wider financial system. DEFINE-UK's version~1.1 regulation instruments are an in-model implementation of that mechanism: a regulatory scenario writes down power-sector capital, and the resulting change in leverage, illiquidity and credit rationing is a modelled consequence rather than an add-on. Whether it does so \emph{correctly} is not something this paper can establish for the scenarios, for reasons Section~\ref{sec:validation} sets out.
+
+\subsection{Replication as a research output}
+
+Finally, this paper belongs to the replication literature, in the sense \citet{dewald1986} established: that published results should be reproducible from published materials, and that the systematic attempt to do so is itself a finding. Their project's central lesson --- that the binding constraint is usually the completeness of what was published, not the willingness of the replicator --- describes this exercise closely. Here the constraint is threefold and unusually explicit: the code is unlicensed, the scenario results are unpublished, and the manual's parameter tables and equations do not everywhere agree with each other. Each is recorded as a ceiling with a stated route to lifting it, rather than routed around.
+
+Within the PolicyEngine Macro suite the same standard has been applied to a central bank's structural VAR \citep{ahmadi2026boesvar} and to the OBR's macroeconometric model \citep{ahmadi2026obr}. The methodological point those papers share with this one is the separation, at the point of claim, between checks that can fail and checks that cannot. A tolerance chosen after seeing the deviation it admits, an identity that holds by construction, a band fitted to a verbal quantity --- these are worth reporting, but they are not evidence of agreement, and a replication that presents them as such is overstating its result. Section~\ref{sec:validation} labels each one where it appears.
diff --git a/papers/define-uk/sections/model.tex b/papers/define-uk/sections/model.tex
new file mode 100644
index 00000000..678b501f
--- /dev/null
+++ b/papers/define-uk/sections/model.tex
@@ -0,0 +1,70 @@
+\section{The model}
+\label{sec:model}
+
+This section describes DEFINE-UK as the manual specifies it \citep{georgedafermos2026manual}. It is a description of the authors' model, not of an implementation choice of ours; where the manual's own text and tables disagree, Section~\ref{sec:reimplementation} reports the disagreement rather than resolving it here.
+
+\subsection{Accounting before theory}
+\label{sec:accounting}
+
+DEFINE (Dynamic Ecosystem FINance-Economy) is the stock-flow-fund framework of \citet{dafermos2017}; DEFINE-UK is its United Kingdom application, with the accounting structure derived directly from UK national accounts (ONS Blue Book and UK Economic Accounts) rather than assumed. Two matrices carry the discipline.
+
+The \emph{transactions flow matrix} (manual \S2.2, Table~1) records every monetary flow in the model with a sign convention in which $+$ is an inflow and $-$ an outflow to the sector in whose column it sits. Two families of identity follow:
+\begin{align}
+\sum_{s} T_{f,s} &= 0 && \text{for every flow row } f, \label{eq:trow}\\
+\sum_{f} T_{f,s} &= \mathrm{LEND}_s && \text{for every sector column } s, \label{eq:tcol}
+\end{align}
+that is, each flow leaves one sector and enters another, and each sector's column sums to its net lending position.
+
+The \emph{balance-sheet matrix} (\S2.2, Table~2) mirrors it for stocks, with $+$ an asset and $-$ a liability:
+\begin{align}
+\sum_{s} B_{i,s} &= 0 && \text{for every instrument row } i, \label{eq:brow}\\
+\sum_{i} B_{i,s} &= \mathrm{FNW}_s, \qquad \sum_{s} \mathrm{FNW}_s = 0, \label{eq:bcol}
+\end{align}
+so one sector's asset is another's liability, financial columns sum to sector financial net worth, and financial net worth sums to zero across the economy. Adding real assets to a sector's financial column gives its total net worth.
+
+Because the matrices are derived from national accounts rather than written down abstractly, they cannot close on model-determined flows alone. Following the treatment the manual attributes to Zezza and Zezza, two residual constructs absorb the difference: a \emph{residual transaction} in the flow matrix, capturing the net position of flows the model does not represent, and a \emph{residual financial instrument} $\mathrm{RES}_s = \mathrm{FNW}_s - \mathrm{FNWM}_s$ in the balance sheet, capturing the net asset position of stocks it excludes. These are part of the specification, not a plug added by this replication, and they are transcribed as such.
+
+\subsection{Sectors}
+
+The model has seven institutional sectors --- non-financial corporations (NFC), the power sector (PS), households (HH), monetary financial institutions (MFI), non-monetary financial institutions (NMFI), government (GVT) and the rest of the world (RoW) --- plus a \emph{production module} that is not a sector. The production module is where GDP expenditure (consumption, gross capital formation, exports less imports) flows in and GDP income (wages, gross operating surplus, indirect taxes) flows out; it holds no assets or liabilities, so its column in the transactions matrix sums to exactly zero rather than to a net-lending position. The split between the power sector and the production module follows the NACE classification: electricity generation (D35) is the power sector and all other productive activity, public and private, sits in the production module, with input--output intermediate consumption running between them in both directions.
+
+The rest of the world is recorded in \emph{net} terms for property income and financial stocks (\S3.4.6): net interest and net dividends in the flow matrix, and a net interest-bearing position in the balance sheet. This matters for transcription --- it is the only reading of the manual's own initialisation that is consistent with its tabulated MFI financial assets and liabilities --- and it is where one of the pinned manual defects lives (Section~\ref{sec:reimplementation}).
+
+\subsection{The macro block and the closure}
+
+The manual's \S3.2 sets out the high-level macroeconomic variables --- GDP by expenditure and income, deflators, employment, the labour force and unemployment, productivity --- in 23 equations, Eqs.~(21)--(43). Two of its properties do most of the work in interpreting every result in this paper.
+
+First, the closure is \textbf{demand-led}. Output adjusts to aggregate demand; real GDP is not held to a supply ceiling. Second, productivity growth is \textbf{Kaldor--Verdoorn} (Eq.~(31)): a demand impulse raises productivity growth, and does so persistently, in the tradition \citet{kaldor1966} set out for exactly the UK growth question. Together these mean that a sustained fiscal impulse in DEFINE-UK raises output without being returned to potential by assumption, and raises the productivity path as it goes. That is a deliberate feature of the post-Keynesian closure, and it is also the reason the model's fiscal multipliers are not comparable, without adjustment, to those of supply-constrained models such as the OBR emulator \citep{ahmadi2026obr}; Section~\ref{sec:calibration} makes that argument with the arithmetic attached.
+
+\subsection{Production, power, and the two UK-specific blocks}
+
+Production (\S3.3) divides into the domestic production module (\S3.3.1, Eqs.~(44)--(70)), the power generation sector (\S3.3.2, Eqs.~(71)--(138)) and input--output calculations (\S3.3.3). The production module carries a Leontief gross-output block, mark-up pricing over lagged unit costs, a wage-share and wage-rate distribution block, direct-energy prices and costs, and the productive capital aggregates.
+
+The power sector is where the model's climate-policy content concentrates. It carries separate fossil and non-fossil generation capital, a fossil/non-fossil cost split, marginal-cost electricity pricing, a utilisation and forward-looking-expectation block that drives investment, credit-rationed capital formation, and a full financial account running through to leverage, illiquidity and credit rationing. Government renewable investment is triggered when non-fossil capacity falls short of a utilisation threshold relative to electricity demand. Version~1.1's regulatory policies enter here: a ban or regulation directly depreciates fossil capital, which is a balance-sheet event for whoever holds the claims against it --- the transmission channel of \citet{carney2015} and \citet{battiston2017} implemented inside the accounting rather than bolted onto it.
+
+The second UK-specific block is housing: the dwelling stock is tracked by energy efficiency, so retrofit subsidies and regulation change household energy demand and emissions through the composition of the stock rather than through a reduced-form elasticity.
+
+\subsection{The ecosystem block}
+
+The ecological side (\S3.1) covers energy, emissions, and ecological efficiency and technology. Emissions decompose into electricity and non-electricity components,
+\begin{equation}
+\mathrm{EMIS} = \mathrm{EMIS}_{\mathrm{ELEC}} + \mathrm{EMIS}_{\mathrm{NELEC}},
+\label{eq:emis}
+\end{equation}
+an identity that turns out to be diagnostically useful: in the pinned run the two components sum exactly to the total, which is one of the two checks that rule out an extraction error on our side in the emissions divergence of Section~\ref{sec:validation}.
+
+\subsection{Rates of return and financial behaviour}
+
+Interest rates (\S3.4.7) run off a simple Taylor rule for the Bank Rate in logs, which enforces the zero lower bound; asset rates adjust towards a long-run mark-down on the base rate subject to a floor, and liability rates towards a mark-up over the corresponding asset rate plus a risk premium that depends on the sector's liability depreciation rate and on the MFI sector's own financial-liabilities-to-assets ratio. Default rates and credit rationing then feed investment. This is the \citet{dafermos2018} climate--finance loop in UK form: a policy shock that impairs capital raises leverage, tightens credit, and reduces investment, with the tightening depending on the state of the banking sector.
+
+\subsection{Calibration and the status of the baseline}
+
+The manual estimates its key behavioural equations econometrically on UK data and calibrates the remainder, with \S5's Tables~5 and 6 giving parameters and initial values respectively. The baseline is calibrated against external projections --- OBR macroeconomic forecasts to 2030 with growth rates extended thereafter, NESO current-policy emission pathways, and NGFS scenario data (\S4.1).
+
+Two statements from the manual govern how any of its baseline numbers may be used, and both are quoted here because the rest of this paper depends on them. First, on the emissions path: the manual itself notes that the baseline reduction ``falls significantly short'' of the UK's 2035 NDC target --- by design, since it is a current-policies baseline. Second, and more consequentially, on the baseline as a whole:
+
+\begin{quote}
+\emph{``The baseline of the model should also not be seen as a prediction or forecast''} \citep[\S4.1]{georgedafermos2026manual}.
+\end{quote}
+
+\noindent Section~\ref{sec:implementation} takes that literally. It is the reason the adapter serves deltas and never levels, and the reason no DEFINE-UK number appears anywhere in PolicyEngine Macro's forecast surfaces.
diff --git a/papers/define-uk/sections/references.tex b/papers/define-uk/sections/references.tex
new file mode 100644
index 00000000..e2f49dbe
--- /dev/null
+++ b/papers/define-uk/sections/references.tex
@@ -0,0 +1 @@
+\bibliography{references}
diff --git a/papers/define-uk/sections/reimplementation.tex b/papers/define-uk/sections/reimplementation.tex
new file mode 100644
index 00000000..83381504
--- /dev/null
+++ b/papers/define-uk/sections/reimplementation.tex
@@ -0,0 +1,88 @@
+\section{The clean-room reimplementation}
+\label{sec:reimplementation}
+
+The upstream code cannot be hosted, so the route to a hostable DEFINE-UK is to implement the published equations from scratch --- the same play the suite ran for the OBR macroeconometric model \citep{ahmadi2026obr}. This section reports where that stands, and treats it as more than an engineering task: reading a manual closely enough to run it is itself a form of validation, and it has produced findings the R source could not.
+
+\subsection{Protocol}
+
+The specification sources are the manual \citep{georgedafermos2026manual}, the companion papers, and \citet{dafermos2017} for framework-level derivations the manual references. Three rules make the exercise clean-room rather than a translation.
+
+\begin{enumerate}
+\item \textbf{The upstream repository is an output oracle only.} Its numerical outputs are compared against ours in the oracle tests. Its source code is never read to write an equation here.
+\item \textbf{Every equation carries its provenance.} Each registered equation declares a \texttt{manual\_ref} of the form ``\S3.3.2 eq.\ (84)'', validated by a regular expression at construction time, and an equation without one does not merge. Two registrations citing the same reference are rejected, because a copy-pasted provenance would make an equation \emph{look} transcribed while implementing something else --- a failure the per-section contiguity checks cannot see on their own, since a duplicate shows up there only as a hole somewhere else.
+\item \textbf{Data come from official sources}, not from the upstream repository's input spreadsheets --- again a licence consequence, and incidentally a stronger test.
+\end{enumerate}
+
+The architecture is an equation registry, a Gauss--Seidel per-period solver that iterates all registered equations to convergence, one module per manual \S3 section, the \S5 parameter and initial-value tables, and the \S2.2 accounting matrices. Stock-flow norms are asserted each period, not assumed.
+
+\subsection{Milestone 1: PASS}
+
+Milestone~1 (4 August 2026) transcribes the \S2.2 transactions and balance-sheet matrices and the full \S5 parameter and initial-value tables, and requires that every Table~1 and Table~2 row and column identity hold on the \S5 initial values within the manual's own four-significant-figure printing precision.
+
+It passes. The accounting test module collects \textbf{62} tests, of which \textbf{44} are the Table~1/2 row and column identities themselves: 20 transactions rows, 6 transactions columns, 7 balance-sheet rows and 11 balance-sheet columns. The remaining 18 are the residual-instrument and model-financial-net-worth definitions, the economy-wide closure $\sum_s \mathrm{FNW}_s = 0$, and the two tests that pin the one manual inconsistency this milestone found.
+
+That inconsistency is worth stating because it is the template for everything in Section~\ref{sec:gaps}. Table~6's rest-of-world lending entry omits the net-dividends term of Eq.~(383), so the MFI and RoW transaction columns miss lending by $\mp$5.44. This is not an inference: the manual itself tabulates overall lending at 5.44 where, by its own account, it ``should equal 0''. Rather than adjust a value or widen a tolerance until the matrix closed, the discrepancy is pinned exactly, so the implementation reproduces the manual including its defect and a future correction upstream fails the test rather than passing silently.
+
+\subsection{Milestone 2: IN PROGRESS, not passed}
+
+Milestone~2 covers the high-level macro and production blocks (\S3.2--3.3), gated on the baseline GDP path against the oracle. Three of its four sections have landed:
+
+\begin{center}
+\begin{tabular}{@{}llrl@{}}
+\toprule
+Manual section & Content & Equations & Range \\
+\midrule
+\S3.2 & High-level macroeconomic variables & 23 & (21)--(43) \\
+\S3.3.1 & Domestic production module & 27 & (44)--(70) \\
+\S3.3.2 & Power generation sector & 68 & (71)--(138) \\
+\midrule
+\multicolumn{2}{@{}l}{Total} & \textbf{118} & \textbf{(21)--(138)}, contiguous \\
+\bottomrule
+\end{tabular}
+\end{center}
+
+\noindent Each section solves as a system under its own test module, and contiguity is checked structurally rather than asserted: the equation numbers parsed out of the provenance strings must form an unbroken range, so a skipped equation fails a test rather than going unnoticed.
+
+\textbf{Milestone 2 is not passed.} \S3.3.3, the input--output calculations, is not implemented, and the oracle baseline comparison --- the milestone's actual gate --- has not been run. Reporting 118 contiguous equations as a milestone pass would be reporting effort as evidence. What can be said is narrower and true: three of four sections are transcribed and internally consistent against the manual's own initial values, at the precision those values are printed to, with every disagreement enumerated below rather than absorbed.
+
+\subsection{Thirteen gap records}
+\label{sec:gaps}
+
+Where the manual is incomplete, silent, or inconsistent with itself, the implementation does not patch, tune or tolerate. It implements what is printed and records the gap as a machine-readable entry, pinned by a test that fails if the situation changes --- including if the DEFINE team later publishes a value. There are \textbf{thirteen} such records: one in \S3.2, two in \S3.3.1 and ten in \S3.3.2. Appendix~\ref{app:gaps} lists them all. Three general observations matter more than the individual entries.
+
+\paragraph{They are not all the same kind of thing, and one is not a manual defect at all.} The thirteen cover missing parameters, symbols the body uses and the tables never define, variables the tables carry and the body never uses, a printed inequality that contradicts the prose above it, and quantities Table~6 simply omits. One of them --- the expected capital path over the planning horizon of Eq.~(89) --- records \emph{our} modelling choice rather than a defect: the manual never says what capital stock the sector expects to hold at $t+\tau$ and the model carries no forecast of it, so real capital and electricity demand are held at current values across the horizon. It is recorded in the same place as the others precisely so that a choice of ours is not mistaken for a reading of theirs. It is inert whenever the baseline credibility switch is zero.
+
+\paragraph{Some of them have consequences, and the consequences are stated.} \S3.3.2 as published \emph{cannot be simulated forward}: six parameters it uses are absent from Table~5, and two variables --- the fossil and non-fossil capital profit rates of Eq.~(99) --- are never defined anywhere in the manual. At the defaults this is not cosmetic. Credit rationing degenerates to a constant, the green/fossil investment split degenerates to 50:50 against the 69:31 that Table~6 implies, and desired power-sector investment turns negative: held at its own initial values the section returns power-sector capital formation of $-0.021$ against a tabulated $+2.15$. The 50:50 default is visibly wrong and is pinned as such rather than tuned to fit, because tuning it would replace a documented hole with an undocumented guess. This is the single largest gap in the section, since that split is what steers investment green.
+
+\paragraph{Where the manual could have been guessed, it was not.} Table~5 reuses non-financial-corporation parameter values for the power sector in at least nine places, so reusing them once more for Eq.~(138)'s credit-rationing slopes would have been a defensible inference --- but it would have been \emph{our} inference rather than the manual's instruction, and it does not reproduce Table~6 in any case. It was not done. Similarly, Eq.~(96)'s missing intercept can be backed out as roughly 0.0210 if Table~6 is read as a steady state; that value is recorded and \emph{not used}, because the equation's two lagged terms make the steady-state reading an assumption rather than a measurement.
+
+\subsection{The largest disagreements}
+
+Beyond the gaps, a number of printed identities disagree with Table~6 by one to three orders of magnitude more than its printing noise. All are implemented as printed and pinned individually. Four deserve statement here.
+
+\paragraph{Eq.~(84): a factor of 3.04.} The electricity price rule, a fixed mark-up over marginal cost, gives 0.9725 against a tabulated 0.3198, implying a mark-up of $-0.267$ --- electricity sold below marginal cost. Both sides are independently corroborated: marginal cost by Eqs.~(82) and (83) to $4.5\times10^{-5}$, and the price by Eq.~(74) to $3.7\times10^{-4}$. So the manual's two chains meet at a contradiction. The most economical account is not a wrong number but a \emph{missing equation}: \S5 calibrates an electricity-price long-run formation rule --- a switch time and a long-run price --- that \S3 never prints, and Eq.~(84) is the only price equation printed, with no long-run term and no switch. Nothing is implemented for the unprinted rule, because there is nothing printed to implement.
+
+\paragraph{One deflator, counted once.} Eq.~(27) gives real gross capital formation of 111.88 against a tabulated 112.30 --- 0.37 per cent, roughly ten times the rounding noise --- and Eq.~(25) reproduces the tabulated real GDP only using 112.30. This is emphatically \emph{one} finding and not five: gross capital formation, its NFC, household and government components, and the two capital stocks all show nominal-to-real ratios of 1.0309--1.0312 where the equations prescribe a production deflator of 1.035, and the same 0.4 per cent reappears in \S3.3.2's Eqs.~(104), (105), (132) and (133). Table~6's entire capital block is deflated at 1.031 --- most plausibly a national-accounts investment deflator where the equations want the production deflator. Counting it once matters: reported as five independent discrepancies it would look like a section riddled with errors rather than one systematic property of one table's capital rows.
+
+\paragraph{Eq.~(107): a model-wide sign problem, not a power-sector one.} Eq.~(107) has a determinate-sign right-hand side and Table~6 tabulates the opposite. This qualification belongs with the finding wherever it is quoted: \textbf{it is not a power-sector defect}. The manual prints the same rule for three other sectors --- Eqs.~(177), (231) and (284) --- every coefficient is positive, and Table~6 tabulates all four transfers negative, with magnitudes out by factors of 1.7 to 4.6 as well. Whatever is wrong is wrong in the interest-bearing asset transfer rule, or in Table~6's sign convention for it, \emph{model-wide}; \S3.3.2 is simply where it is met first. Confirming that requires \S3.4.1, \S3.4.3 and \S3.4.4, which are not implemented yet, so the claim is scoped accordingly.
+
+\paragraph{A normalisation that propagates.} Eq.~(61) gives a fuel price of 0.6788 against a tabulated 1, where Table~5 says of the relevant parameter that ``at the initial condition fuel price is normalised to equal 1''. A third witness supports the normalisation: Table~6 tabulates the power sector's nominal and real fuel intermediate consumption at the same value, which is only possible at a fuel price of 1. The gap does not contaminate any \S3.3.2 identity, because the affected quantity is tabulated. In a \emph{simulation} it cuts that input 32 per cent and carries essentially undiluted into marginal cost and the electricity price --- and it does not cancel Eq.~(84): $3.04 \times 0.68$ still leaves the price at 2.07 times Table~6's.
+
+\subsection{The audit, and what it withdrew}
+\label{sec:audit}
+
+On 12 August 2026 the pinned findings were audited against the manual PDF and recomputed independently of the sector modules: all 661 transcribed \S5 values were re-checked against the page each source comment names, and every pinned discrepancy was reproduced. The arithmetic held. The interpretations did not entirely, and the corrections are recorded here because a replication that only ever adds findings is not being audited.
+
+\paragraph{Four findings re-characterised.} Eqs.~(51), (59), (94) and (111) had been reading as the manual contradicting itself. They are not. Table~5 describes each of the parameters concerned as a \emph{historical mean} --- ``calibrated so the long-run mark-up at initial utilisation equals the mean of'' two sample windows, ``calculated using the mean over past data'', ``set as the mean of past implied values'', ``taken as the mean of past data''. A parameter described that way never claimed to reproduce the manual's own initial period. These are therefore \textbf{first-period jumps} that a run started from Table~6 will take in its first quarter, not defects. They still matter --- one of them moves a price that steers green investment, by 60 per cent --- and they are still pinned. They are simply not evidence that the manual disagrees with itself, and the distinction turns on the manual's own stated provenance for each parameter, which is why Table~5's remarks are treated as part of the specification.
+
+\paragraph{One claim withdrawn outright.} An earlier version of the reimplementation notes claimed of Eq.~(51) that ``the equation is the corroborated one'', because Eqs.~(49) and (50) imply a value close to it. That inference does not hold and is withdrawn: Eq.~(49) reads \emph{lagged} unit costs, and the corroborating value is what it gives only if unit costs were flat into the initial period. Honouring the lag, the tabulated mark-up implies $+0.91$ per cent quarterly unit-cost growth and the equation's implies $-0.12$ per cent --- deflation --- and the manual's own initial period is growing and inflating, which favours the tabulated value if anything. On the evidence a single-period snapshot provides, neither side is the outlier, and the finding now says so.
+
+\paragraph{One finding unified.} Eqs.~(27), (104), (105), (132) and (133) were counted as five and are now counted as one: the single deflator finding above.
+
+\paragraph{One new gap found.} The unprinted electricity-price rule described above was identified by the audit, and is the likeliest account of Eq.~(84).
+
+Two of the findings have been filed publicly on the authors' own tracker --- the emissions vintage divergence of Section~\ref{sec:target1b}, and \S3.3.2's missing parameters and undefined variables together with Eqs.~(84) and (61) --- so that the record here and the record there stay in step. Both were re-verified in the audit.
+
+\subsection{Attribution}
+
+The suite's standard adapted-model stance applies. The \textbf{model design belongs to its authors}; the \textbf{implementation is ours}. Every surface that presents results names DEFINE-UK and its authors, links the manual and papers, and states that this is an independent Python implementation of the published equations --- not the authors' code, and not endorsed by them. The implementation is AGPL-3.0, which is what makes hosting possible without any upstream licence.
diff --git a/papers/define-uk/sections/source.tex b/papers/define-uk/sections/source.tex
new file mode 100644
index 00000000..bdabedf3
--- /dev/null
+++ b/papers/define-uk/sections/source.tex
@@ -0,0 +1,37 @@
+\section{The source publications, the licence, and replication scope}
+\label{sec:source}
+
+\subsection{What is published}
+
+Four documents and one repository make up the published record of DEFINE-UK, and they differ sharply in what they let a replicator check.
+
+\paragraph{The manual.} \emph{DEFINE-UK Model Manual, Version 1.1} \citep{georgedafermos2026manual}, by Adam George and Yannis Dafermos of SOAS University of London, April 2026. This is the specification: the full equation listing by sector (\S3), the calibration approach (\S4) and the parameter and initial-value tables (\S5). It is also the only source of published \emph{numeric results}, and those stop at the baseline: Table~4 (p.~50) gives fifteen baseline quantities at 2025, 2030 and 2040 with 2025--40 means and standard deviations, and Table~5 embeds the design parameters of the model's scenarios. Table~6 (pp.~63--82) gives initial values for the endogenous variables, which is a large replication surface for a single period but is not a scenario result.
+
+\paragraph{The two companion papers.} Both are access-restricted. \emph{Green Fiscal Policy in the UK: A Scenario Analysis} \citep{georgedafermos2026scenario} is the version~1.0 paper (SSRN abstract 6541398); its scenario set, per the abstract, is carbon pricing, green power-sector subsidies and green housing subsidies --- a set that predates version~1.1's regulation instruments in any case. \emph{Evaluating climate policy mixes in the UK: an E-SFC approach} \citep{georgedafermos2026mixes} is the version~1.1 paper (SSRN abstract 6588778) and is likewise not obtainable. An exhaustive search, catalogued in the adapter's \texttt{validation/published\_targets.json}, found no open mirror of either on RePEc, SOAS pages or author pages.
+
+\paragraph{The conference version.} One open document survives: \emph{Green fiscal policy in an empirical UK E-SFC model} \citep{georgedafermos2023fmm}, the October 2023 FMM conference paper, marked by its authors ``preliminary and incomplete''. Its results are figure-based and it predates DEFINE-UK 1.0, so it can support qualitative or coarse checks and nothing finer. Section~\ref{sec:validation} uses exactly two numbers from it and is explicit about what they can and cannot establish.
+
+\paragraph{The repository.} \url{https://github.com/DEFINE-model/DEFINE_UK_1.1}: the authors' R implementation, public, and carrying \textbf{no licence file}.
+
+\subsection{The licence position and what follows from it}
+\label{sec:licence}
+
+Absent a licence, the default position is that no reproduction, redistribution or hosting right has been granted. This is treated as binding rather than as an oversight to be assumed away, and three consequences follow, each of which shapes the rest of this paper.
+
+First, \emph{nothing upstream is vendored}. The adapter contains no upstream code and no upstream data. It fetches the repository at pinned commit \texttt{846081a} at runtime, on the machine of whoever is running it, and executes it there.
+
+Second, \emph{nothing upstream is hosted}. DEFINE-UK is the only member of the PolicyEngine Macro suite that is local-only. Hosted tooling returns run instructions, never results computed from the upstream code, and continuous integration never fetches or runs it --- which is why the oracle comparisons of Section~\ref{sec:validation} run only where a cached pinned run exists, with a hermetic subset of the gate running everywhere.
+
+Third, \emph{the route to a hostable model is reimplementation}. Section~\ref{sec:reimplementation} describes the clean-room Python implementation of the published equations that exists for this reason, and its attribution stance: the model design belongs to its authors; the implementation is ours, AGPL-3.0, and independent.
+
+\subsection{Replication scope}
+
+The exercise has two tracks, held to the same standard applied to two different references.
+
+The \textbf{oracle track} runs the authors' unmodified R code at the pinned commit and compares its outputs against the published figures --- the manual's Table~4 for the baseline, and the scenario design parameters of Table~5 plus the two FMM 2023 anchors for the scenarios. The run is calendar-anchored on the population path: 1987Q1--2040Q4, with 2025 at $t=153$.
+
+The \textbf{clean-room track} reimplements the published equations in Python from the manual, with the upstream repository used strictly as a numerical \emph{output oracle} whose source is never read to write an equation. Its milestone gates compare the reimplementation against the manual's own tables first and against the oracle second.
+
+\paragraph{What is out of scope.} No attempt is made to re-estimate the manual's econometrically estimated behavioural equations from source data, or to reconstruct the authors' input spreadsheets. Where the reimplementation needs data it takes it from official sources rather than from the upstream repository's \texttt{input/} directory --- again a consequence of the licence, and incidentally a stronger test.
+
+\paragraph{Execution is not validation.} On 1 August 2026 the full upstream notebook rendered end to end through the adapter's runner, on R~4.3.0 under macOS, with the upstream unmodified and an \texttt{rstudioapi} shim the only accommodation, producing 151 output files: figures and tables for all four scenario blocks plus a multiplier summary. That established that the pinned code executes. It established nothing whatever about whether its outputs mean anything, and the record says so at that date. The adapter's validation gate is explicit on the point: nothing built on the repository may present model output as meaningful except by naming the target it rests on. Section~\ref{sec:validation} is the record of what that gate has since established --- including where the answer is ``nothing further can be established from what is published''.
diff --git a/papers/define-uk/sections/validation.tex b/papers/define-uk/sections/validation.tex
new file mode 100644
index 00000000..d5e4ef46
--- /dev/null
+++ b/papers/define-uk/sections/validation.tex
@@ -0,0 +1,104 @@
+\section{The replication record}
+\label{sec:validation}
+
+Four published-target gates were attempted against the pinned upstream run. One passes, one diverges and is pinned, and two are closed at the ceiling the publications allow rather than passed. This section reports each with its tolerance, its arithmetic, and --- where the check could not have come out any other way --- a statement to that effect at the point of claim rather than in a footnote.
+
+Table~\ref{tab:gates} is the summary. Every gate in it is enforced by a test in the adapter repository against the cached pinned run, so drift fails loudly rather than silently.
+
+\begin{table}[t]
+\centering
+\caption{DEFINE-UK replication gates}
+\label{tab:gates}
+\footnotesize
+\begin{tabularx}{\textwidth}{@{}>{\raggedright\arraybackslash}Xll@{}}
+\toprule
+Gate & Verdict & Enforced by \\
+\midrule
+\multicolumn{3}{@{}l}{\emph{Cannot fail as evidence of agreement --- pins and post-hoc tolerances, reported as such}} \\
+2025 real GDP growth within $\pm0.31$\,pp & clears by 0.01\,pp & tolerance set after the run \\
+FMM 2023 anchor: GPI GDP peak in $[0.7, 1.3]$\,\% & in band ($+0.92$) & band chosen by us, $\pm30$\,\% \\
+FMM 2023 anchor: 2030 baseline emissions in $[330, 352]$ & in band (342.8) & band admits values it is named against \\
+Emissions ratios within $\pm0.02$ of 0.965/0.898/0.767 & holds & pin at the \emph{observed} divergence \\
+11 scenario delta points, rel.\ $10^{-6}$ & holds & pin of the cached run against itself \\
+Reference artifact well-formed and self-consistent & holds & hermetic artifact check \\
+\addlinespace
+\multicolumn{3}{@{}l}{\emph{Can fail --- comparisons against a publication}} \\
+Manual Table~4 macro block, S1 baseline & \textbf{PASS} & \texttt{test\_replication\_baseline.py} \\
+\quad population 16+ and labour force, $\pm0.01$\,m & exact & \\
+\quad real GDP growth at 2030 and 2040 & match to 0.06\,pp & \\
+\quad unemployment, $\pm0.2$\,pp & within tolerance & \\
+Manual Table~4 total emissions, S1 baseline & \textbf{DIVERGENCE} & \texttt{test\_replication\_baseline.py} \\
+Scenario policy-switch sets vs published descriptions & exact & \texttt{test\_scenario\_design.py} \\
+Published numeric v1.1 scenario results & \textbf{CEILING} --- none exist & \texttt{published\_targets.json} \\
+Baseline vs ONS/DESNZ/OBR outturns & \textbf{COMPUTED} --- gaps material & \texttt{test\_validation.py} \\
+Clean-room milestone 1 (\S2.2 matrices on \S5 values) & \textbf{PASS} & \texttt{test\_accounting.py} \\
+Clean-room milestone 2 (\S3.2--3.3) & \textbf{IN PROGRESS} & \texttt{test\_macro.py} \\
+\bottomrule
+\end{tabularx}
+\par\smallskip
+\parbox{0.97\textwidth}{\footnotesize \emph{Notes:} The first block is separated because none of those six checks can produce evidence that this replication agrees with a publication. The growth tolerance was set after the deviation it admits was known; the two anchor bands were chosen by us to fit a verbal quantity and were never published as tolerances; the emissions gate is set \emph{at} the observed divergence, so it detects further drift and asserts no agreement; the delta-point and artifact checks compare the cached run and the committed artifacts against themselves. All verdicts are against upstream commit \texttt{846081a}, dated 4--5 August 2026 for the oracle gates and 11 August 2026 for milestone~2.}
+\end{table}
+
+\subsection{Target 1a: the baseline macro block --- PASS, with one year that proves nothing}
+\label{sec:target1a}
+
+The manual's Table~4 publishes fifteen baseline quantities at 2025, 2030 and 2040. Four of them --- real GDP growth, unemployment, population aged 16 and over, and the labour force --- form the macro block, and it is the one published target this replication passes. Tolerances were population and labour force $\pm0.01$\,m, growth $\pm0.31$\,pp, and unemployment $\pm0.2$\,pp.
+
+Population and labour force reproduce exactly to 0.01\,m at all three dates. Unemployment is within 0.2\,pp at all three. Real GDP growth matches to \textbf{0.06\,pp} at 2030 and 2040. The 2025--40 means and standard deviations of growth and unemployment reproduce the table at its printing precision. The run is calendar-anchored on the population path, 1987Q1--2040Q4, with 2025 at $t=153$.
+
+\paragraph{The 2025 growth gate is not independent of the deviation it admits.} At 2025 the run gives 4.66 per cent against Table~4's 4.96 --- a miss of $-0.30$\,pp against a $\pm0.31$\,pp gate. The gate was set \emph{after} the run, and the only year that uses any of it clears by 0.01\,pp. As a test of agreement in 2025 it is doing no work: any tolerance chosen once the deviation is known will admit that deviation. What it is doing is stopping that year drifting further, which is worth having and is not the same claim. \textbf{The informative part of this gate is the pair of 0.06\,pp matches at 2030 and 2040}, which were not fitted to anything.
+
+The recorded reason for the 2025 gap is a convention we could not resolve. Table~4's note says only that ``all quarterly values are annualised''; it does not say which annualisation, and the readings differ by more than the gap. The adapter's own comparator table uses the annual-mean convention, which is what gives 4.66. This is recorded as an unresolved convention mismatch, not as a demonstrated agreement --- and it is the reason the tolerance exists at that width rather than a tighter one.
+
+\subsection{Target 1b: the emissions path --- DIVERGENCE, pinned}
+\label{sec:target1b}
+
+The pinned code runs \emph{below} the published Table~4 emissions, and the gap widens monotonically with the horizon:
+
+\begin{center}
+\begin{tabular}{@{}lrrr@{}}
+\toprule
+& 2025 & 2030 & 2040 \\
+\midrule
+Pinned run (MtCO$_2$e/yr) & 393 & 343 & 249 \\
+Manual Table~4 (MtCO$_2$e/yr) & 407 & 382 & 324 \\
+Pinned ratio, run/table & 0.965 & 0.898 & 0.767 \\
+Deviation & $-3.5\%$ & $-10.2\%$ & $-23.3\%$ \\
+\bottomrule
+\end{tabular}
+\end{center}
+
+\noindent The percentages are computed from the pinned ratios themselves --- 0.965, 0.898 and 0.767, held in the committed reference artifact --- and not from the rounded levels printed in the first two rows, which would give slightly different figures.
+
+Two checks rule out an extraction error on this side. First, the components satisfy the model's own identity \eqref{eq:emis}: $\mathrm{EMIS}_{\mathrm{NELEC}} + \mathrm{EMIS}_{\mathrm{ELEC}}$ sums exactly to $\mathrm{EMIS}$ in the run outputs. Second, the baseline is identical across every scenario folder, so no scenario contamination is involved. What remains is a vintage or calibration gap between the published table and commit \texttt{846081a}: the table was produced by some code vintage, and it is not this one.
+
+The divergence has been raised with the upstream authors on their own issue tracker, so that the record here and the record there stay in step. Meanwhile the tests gate at the \emph{observed} ratios, within $\pm0.02$, rather than at agreement. That is a deliberate and limited claim, and it belongs in the first block of Table~\ref{tab:gates}: a gate set at the divergence cannot fail on account of the divergence. It can and will fail if the gap moves, which is the only thing it is for.
+
+\subsection{Targets 2 and 3: closed at the achievable ceiling, not passed}
+\label{sec:targets23}
+
+The two scenario targets are the ones a reader of a climate-policy model most wants replicated, and they are the ones that cannot be. An exhaustive search, catalogued with locators in the adapter's \texttt{validation/published\_targets.json}, established that \textbf{no machine-readable numeric scenario results are published for version~1.1}. The manual's results stop at Table~4's baseline plus the scenario \emph{design} parameters embedded in Table~5; both companion papers \citep{georgedafermos2026scenario, georgedafermos2026mixes} are access-restricted with no open mirror; and the version~1.0 paper's scenario set --- carbon pricing and subsidies --- predates version~1.1's regulation instruments in any case.
+
+There is therefore no scenario replication in this paper, and none is manufactured. What is verifiable has been verified, and it is design rather than results.
+
+\paragraph{Scenario design, pinned exactly.} Each cached scenario is checked to toggle \emph{exactly} the policy switches its published description claims --- no more and no fewer. The fossil-fuel-ban scenarios toggle the ban flag with its capacity and investment ban timings; the power subsidy toggles its subsidy switch alone; green public investment toggles government investment, green bonds and green power exactly; and the housing-subsidy variant pins the published 40 per cent subsidy rate. This is a real check against a published description, and it can fail --- but it constrains the scenario's \emph{inputs}, not its outputs.
+
+\paragraph{Two anchors, and what they are worth.} Two coarse numeric statements survive in the open FMM 2023 conference version \citep{georgedafermos2023fmm}, and both are used, at wide tolerances, because that document is a pre-1.0 vintage marked preliminary by its authors. The current-policies baseline is described as emitting ``just under 350 MtCO$_2$e'' in 2030; the cached run gives \textbf{342.8}, inside a band of 330--352. And the green-public-investment scenario is described in terms of ``1\% of GDP''; the cached run's real GDP delta peaks at \textbf{$+0.92$} per cent, inside a band of 0.7--1.3.
+
+\textbf{Both should be read as regression pins, not as agreement tests}, and there are three separate reasons, each of which would be sufficient on its own.
+
+First, \emph{the bands are ours}. Neither was published as a tolerance. Both were chosen by us to fit a verbal quantity after the run was in hand, which puts them in the same category as the 2025 growth gate of Section~\ref{sec:target1a}.
+
+Second, \emph{they cannot discriminate}. The GDP band is $\pm30$ per cent around ``around 1 per cent''. The emissions band runs to 352, which is \emph{above} the 350 that the phrase ``just under'' is supposed to bound --- so a run at 351 would pass a gate named for a statement it violates. A test that a value contradicting the claim would pass is not testing the claim.
+
+Third, and specific to the GPI anchor, \emph{the two sides may not be the same quantity}. The transcription of the FMM text held in \texttt{published\_targets.json} records the green-public-investment scenario as one in which ``government invests an additional 1\% of GDP in green projects by 2030'' --- that is, 1 per cent of GDP is the size of the policy \emph{input}. The gate compares it against the peak of the model's real GDP \emph{response}. Those are different objects that happen to sit near each other numerically, and this replication has not been able to obtain the conference paper's figures to establish whether a comparable published GDP-response reading exists. On the reading in the record, the anchor is not like-for-like at all; on the more generous reading it is like-for-like inside a $\pm30$ per cent band. Neither reading supports quoting $+0.92$ against $\approx1$ as a replication result, so it is not quoted as one anywhere.
+
+What the two anchors do establish is worth stating plainly, because it is not nothing: the cached run has not moved since they were recorded.
+
+\paragraph{The upgrade path.} This ceiling is not permanent and its lifting conditions are specific. If the authors publish scenario tables, or either SSRN paper becomes accessible, the published-figure gate reopens and targets 2 and 3 can be attempted as replications rather than closed as design checks. Until then, the honest verdict is the one in the adapter's own record: \emph{closed at the achievable ceiling}, which is not a pass.
+
+\subsection{What the whole record does and does not license}
+
+The adapter's validation file carries a status header, and for a period after target 1a passed that header still read ``NOT VALIDATED. No published DEFINE-UK result has been reproduced'' --- which put it in contradiction with every page that cited it as the record. The correction belongs in this paper because this paper cites that file as the record. It now reads \textbf{PARTIALLY VALIDATED} and breaks the claim down target by target: 1a pass, 1b divergence pinned, targets 2 and 3 closed at the achievable ceiling rather than passed, reimplementation milestone~1 pass and milestone~2 outstanding.
+
+Nothing in that licenses presenting DEFINE-UK scenario output as a validated result. The operative rule, held in the repository and applied throughout this paper, is that a claim about this model must name the target it rests on.
diff --git a/papers/frb-us/index.html b/papers/frb-us/index.html
index eef0ffbe..9b2315fe 100644
--- a/papers/frb-us/index.html
+++ b/papers/frb-us/index.html
@@ -22,6 +22,7 @@
 
 
 
 
@@ -84,7 +85,7 @@ 
An open Python implementation of frb-us
-

An identical 100bp funds-rate shock matches the Fed's own pyfrbus to 6.0e-9 (vs 1.0.0) and 1.4e-8 (vs 1.1.1) across all 284 variables and 20 quarters — the same scale at which the Fed's two releases disagree with each other (1.3e-8). The paper also retires its former headline: the 5.6e-17 tracking residual against LONGBASE is an add-factor identity, not evidence, and the same gate passes on randomly rescaled data.

+

An identical 100bp funds-rate shock matches the Fed's own pyfrbus to 6.0e-9 (vs 1.0.0) and 1.4e-8 (vs 1.1.1) across all 284 variables and 20 quarters — the scale at which the Fed's own two releases disagree with each other (1.3e-8). That comparison is the evidence, not the 5.6e-17 tracking residual against LONGBASE, which is an add-factor identity: the same gate passes on randomly rescaled data. On the fiscal side the government-purchases multiplier is 0.72 in year one and 0.64 in year two under the inertial Taylor rule, and 0.77 and 0.90 with the funds rate pegged — the pegged year-two figure sits below the 1.1–1.2 cross-model range for two years of accommodation. Validation →

@@ -92,8 +95,8 @@

Exact rules, estimated aggregates: the PolicyEngine tax&

- Your browser can’t display PDFs inline. Open or download the paper - instead. + Your browser can’t display PDFs inline. The paper is 25 + pages — open or download it instead.

Open the PDF
@@ -116,7 +119,7 @@

Exact rules, estimated aggregates: the PolicyEngine tax&  ·  All papers →

-

Cite this work

+

Cite this work

Ahmadi, Vahid (2026). “Exact rules, estimated aggregates: the PolicyEngine tax–benefit microsimulation in an open model suite.” PolicyEngine Macro working paper. https://policyengine-macro.vercel.app/papers/pe-microsim

@techreport{policyengine2026pemicrosim,
   title       = {Exact rules, estimated aggregates: the PolicyEngine tax-benefit microsimulation in an open model suite},
diff --git a/papers/pe-microsim/main.pdf b/papers/pe-microsim/main.pdf
index bc456309..812a1be0 100644
Binary files a/papers/pe-microsim/main.pdf and b/papers/pe-microsim/main.pdf differ
diff --git a/papers/pe-microsim/sections/limitations.tex b/papers/pe-microsim/sections/limitations.tex
index de61592b..1d7ecd96 100644
--- a/papers/pe-microsim/sections/limitations.tex
+++ b/papers/pe-microsim/sections/limitations.tex
@@ -14,7 +14,7 @@ \section{Limitations}
 
 \item \textbf{One external benchmark, one country, one tax.} The ready-reckoner comparison of Section~\ref{sec:reckoner} is the only committed check against an outside number. It is UK-only; there is no committed US benchmark. It is aggregate-only; there is no committed check of the decile impacts or the winner and loser counts against any published distributional analysis. And it is measured against another estimate produced on administrative data with embedded behavioural elasticities, so the deviation is a distance between two estimates rather than an error.
 
-\item \textbf{The benchmark's own numbers have thin provenance.} Table~\ref{tab:reckoner}'s 2028--29 figure of \pounds 6.92bn is a linear interpolation between the two endpoints, not an independently scored year, and its $-15.6$ per cent deviation inherits that construction. The 2030 row is not year-matched: the June 2025 reckoner stops at 2028--29. And the \pounds 7.38bn endpoint itself has no scored-run artifact anywhere in the project---the pair survives as prose and as \texttt{np.linspace(6.46, 7.38, 5)} in a plotting script---while the \pounds 6.46bn figure's only executable check is skipped without a microdata token and passes anything between \pounds 4bn and \pounds 9bn. Section~\ref{sec:reckonerprov} sets this out. Producing and committing a scored 2030 run, and tightening that gate to something the costing could fail, is the second-most valuable piece of outstanding work after quantifying the interval.
+\item \textbf{The benchmark's own numbers have thin provenance.} Table~\ref{tab:reckoner}'s 2028--29 figure of \pounds 6.92bn is a linear interpolation between the two endpoints, not an independently scored year, and its $-15.6$ per cent deviation inherits that construction. The 2030 row is not year-matched: the June 2025 reckoner stops at 2028--29. And the \pounds 7.38bn endpoint itself has no scored-run artifact anywhere in the project---the pair survives as prose and as \texttt{np.linspace(6.46, 7.38, 5)} in a plotting script---while the \pounds 6.46bn figure's only executable check is skipped without a microdata token. That check passed anything between \pounds 4bn and \pounds 9bn until August 2026 and now pins $\pm 5$ per cent of \pounds 6.46bn, so the 2026 endpoint is at last gated against something it could fail---when it is run. Section~\ref{sec:reckonerprov} sets this out. Producing and committing a scored 2030 run remains the second-most valuable piece of outstanding work after quantifying the interval.
 
 \item \textbf{Data vintage lag and intra-suite statute divergence.} UK population runs use \texttt{enhanced\_frs\_2023\_24} while upstream publishes a newer FRS vintage this integration has not adopted; and this integration tracks the latest country model while the OLG member is pinned to \texttt{policyengine-uk==2.88.0}, so two members of the same suite can disagree about baseline statute.
 
diff --git a/papers/pe-microsim/sections/validation.tex b/papers/pe-microsim/sections/validation.tex
index 4a9e6dbf..88235fe7 100644
--- a/papers/pe-microsim/sections/validation.tex
+++ b/papers/pe-microsim/sections/validation.tex
@@ -51,7 +51,7 @@ \subsection{What stands behind the two endpoints}
 
 The provenance of the pair is weaker than a table of four significant figures suggests, and saying so is part of the benchmark.
 
-The \pounds 6.46bn 2026 figure is the one the pipeline can in principle produce: \texttt{integration/tests/test\_population.py} scores exactly this reform through \texttt{pe\_population\_impact} for 2026 and asserts the result. But that test is skipped unless a Hugging Face token for the gated microdata is present, its assertion band is $4.0 < x < 9.0$ \pounds bn---wide enough to admit both HMRC figures and most of the plausible range besides---and it writes no artifact. It is a smoke test against gross error, not a check that reproduces 6.46.
+The \pounds 6.46bn 2026 figure is the one the pipeline can in principle produce: \texttt{integration/tests/test\_population.py} scores exactly this reform through \texttt{pe\_population\_impact} for 2026 and asserts the result. That test is still skipped unless a Hugging Face token for the gated microdata is present, and it writes no artifact, so it does not run in ordinary CI. Its assertion band was $4.0 < x < 9.0$ \pounds bn---wide enough to admit both HMRC figures and most of the plausible range besides, so it could not have distinguished this costing from the benchmark it is compared against. It now pins $\pm 5$ per cent of \pounds 6.46bn and names the surfaces that must move with it, which makes it a check on the published number rather than a smoke test against gross error---but only when it is run.
 
 The \pounds 7.38bn 2030 endpoint has no scored run behind it anywhere in this project. In the repository the pair exists in exactly two forms: as prose, and as the endpoints of \texttt{np.linspace(6.46, 7.38, 5)} in the sister paper's plotting script \citep{ahmadi2026obr}, from which the interpolated middle row is read off. The OBR emulator's own repository freezes both as named constants and says so in terms---the microsimulation is not runnable there, so the number is carried, not computed. No committed file in this project records a 2030 score, and no environment used to prepare this paper could reproduce one: the \texttt{policyengine} package is not installed and the microdata is gated. A reader should treat 6.46 as a figure with a runnable but loosely gated check behind it, and 7.38 as a figure with none.
 
diff --git a/papers/psl-og/index.html b/papers/psl-og/index.html
index 0c07f7e9..72077eb5 100644
--- a/papers/psl-og/index.html
+++ b/papers/psl-og/index.html
@@ -85,11 +85,13 @@ 
An overlapping-generations model for PolicyEngine reform
+

Every quantity in the calibration table except the capital–output ratio is a calibration input, so a small deviation on it is arithmetic, not evidence: depreciation (δ = 0.065), potential growth (1.1%/yr) and the 95% debt-to-GDP target are imposed and cannot disagree with the official aggregates, and the discount factor β = 0.965 is fixed, not estimated (estimate_beta=False), so the model publishes no achieved saving ratio to compare against the ONS 8.9%. The one substantive gap is the labour share: the retained OG-Core γ = 0.35 implies 65% against an ONS labour share near 59.5%. The genuinely over-identifying checks — the capital–output ratio and the steady-state interest rate — have no published reconciliation yet. Validation →

+
- +

Your browser can’t display PDFs inline. The paper is 34 @@ -113,7 +115,7 @@

An overlapping-generations model for PolicyEngine reform

Source & reproduction: papers/psl-og/ on GitHub -  ·  All papers → +  ·  All papers →

Cite this work

@@ -126,6 +128,7 @@

Cite this work

url = {https://policyengine-macro.vercel.app/papers/psl-og}, note = {PolicyEngine Macro working paper} }
+

The underlying model is the Policy Simulation Library’s OG-Core, deployed through its OG-UK calibration; please also credit OG-Core as the source model. No DOI is assigned. For an exact version stamp, cite the git commit of the artifact you used from github.com/PolicyEngine/macro.

diff --git a/pe/validation/index.html b/pe/validation/index.html index b82b01a3..3ca4c2cb 100644 --- a/pe/validation/index.html +++ b/pe/validation/index.html @@ -81,44 +81,34 @@

Checked rule by rule, not against a forecast.

- There is no forecast error to report, because the model is not - forecasting — it applies published rules, checkable by hand. For - the £50,000 UK earner on the code page: - £50,000 − £12,570 personal allowance = £37,430 of basic-rate income; - at 20% that is £7,486 of income tax, at 8% - £2,994 of National Insurance, leaving - £39,520 net. Raising the basic rate 5pp adds 5% of - £37,430 = £1,872. Every figure in the code sample - is deterministic arithmetic from the implemented rules — which - verifies those rules, not that every area of legislation has - complete test coverage. + No forecast error to report: the model applies published rules, and + the worked £50,000 case on the code page can be + checked with a pencil. That verifies the implemented rules — not that + every area of legislation is covered by a test.

- What is not exact is anything resting on survey microdata — - population aggregates inherit sampling error, imputation and ageing - assumptions from the enhanced FRS and the CPS, so a headline - budgetary cost is an estimate in a way a household calculation is - not. For the UK 1p basic-rate case, the - OBR emulator working paper compares - static costings against official sources: £6.46bn in 2026 rising to - £7.38bn by 2030. Every one of those years is - below HMRC's June 2025 figures at every year that can - be matched (£6.9bn for 2026–27, £8.2bn for 2028–29) — - −6.4% in the first year, widening to −10.0% by the end of - our window. Nor is the comparison like-for-like: HMRC's reckoner runs - to 2028–29, so our 2030 figure has no year-matched counterpart, - and HMRC publishes post-behavioural estimates built on administrative - Survey of Personal Incomes data against our static, survey-based one. - It is a benchmark, not a validation. No interval is published with - either of our figures, because the pipeline does not produce one. - Suite-wide evidence: validation. + Population costings are a different kind of number: survey-weighted + estimates (limits), published without an + interval because the pipeline produces none. The one external + comparison is a 1p rise in the UK basic rate, scored at + £6.46bn in 2026 against HMRC's June 2025 ready + reckoner of £6.9bn for 2026–27 — + −6.4%. HMRC's + estimate is post-behavioural on administrative SPI data where ours is + static and survey-based, so this is a benchmark, not a validation. + Our second published figure, £7.38bn in 2030, has no counterpart at + all: the reckoner stops at 2028–29. Both endpoints have thin + provenance — no scored run is committed for either, and the only + executable check pins 2026 within ±5% of £6.46bn and is skipped + without microdata access. Full account: + the paper.

- household calculation - deterministic - implemented rules applied, not estimated + 1p on the UK basic rate, 2026 + −6.4%vs HMRC + benchmark, not validation — different bases
countries @@ -145,37 +135,31 @@

Scope and vintage.

Static, no general equilibrium - Reforms are modelled statically, with optional post-hoc behavioural responses; prices, wages and output do not move. + Prices, wages and output do not move; behavioural responses are optional and post-hoc. UK data vintage - UK population runs use enhanced_frs_2023_24. Upstream now publishes an FRS 2024-25 vintage, which this integration has not adopted. + enhanced_frs_2023_24; upstream's FRS 2024-25 vintage is not adopted here. Survey-based aggregates - Population estimates inherit sampling, imputation and ageing error. HMRC's costings use administrative Survey of Personal Incomes data and embed taxable-income elasticities, so differences of a few hundred million on a broad-based rate change are expected. + Population estimates carry sampling, imputation and ageing error, and no interval is computed for them. HMRC's costings use administrative Survey of Personal Incomes data and embed taxable-income elasticities, so a few hundred million of difference on a broad-based rate change is expected. Gated UK microdata - A HUGGING_FACE_TOKEN with access is required locally. Household calculations are unaffected. + Population runs need a HUGGING_FACE_TOKEN with access. Household calculations are unaffected. Statute-vintage divergence with OLG - This package tracks the latest release; the OLG member is pinned to policyengine-uk==2.88.0, so the two can disagree on baseline statute — a declared inconsistency. - - - Two countries - UK and US only. + This package tracks the latest release; the OLG member is pinned to policyengine-uk==2.88.0, so baseline statute can disagree — a declared inconsistency.

- full write-up: the pe-microsim working - paper sets out the exactness boundary, the validation posture and - the ready-reckoner benchmark in full. Where this member feeds the macro - members, the bridges are documented in the - OBR and - OLG papers. + bridges into the macro members: + OBR, + OLG. Suite-wide evidence: + validation.

diff --git a/reports/fetch_hank_counts.py b/reports/fetch_hank_counts.py index 93c32e50..a6249bd7 100644 --- a/reports/fetch_hank_counts.py +++ b/reports/fetch_hank_counts.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Refresh the citation and star counts in reports/us-hank-open-source.html. +"""Refresh the citation and star counts in reports/us-hank-open-source/. The report said its counts "use canonical OpenAlex records" and the GitHub API, but nothing recorded when they were taken and no script existed to @@ -31,7 +31,7 @@ import urllib.request from pathlib import Path -PAGE = Path(__file__).resolve().parent / "us-hank-open-source.html" +PAGE = Path(__file__).resolve().parent / "us-hank-open-source" / "index.html" # label -> (OpenAlex work id, GitHub repo or None, substring that identifies # the row in the page). The row marker is the paper title rather than the @@ -118,7 +118,8 @@ def published() -> dict[str, int]: # A checker that silently skips a row it cannot find is the same # failure this script exists to prevent. raise SystemExit( - f"cannot find the {label} row in {PAGE.name} via " + f"cannot find the {label} row in " + f"{PAGE.parent.name}/{PAGE.name} via " f"{row_marker!r}; the table changed and this script must be " "updated with it" ) diff --git a/reports/us-hank-open-source.html b/reports/us-hank-open-source.html deleted file mode 100644 index 8f10e0ab..00000000 --- a/reports/us-hank-open-source.html +++ /dev/null @@ -1,121 +0,0 @@ - - - - - - Open US HANK models — code, data, reach - - - - - - - - - - - - - - -← PolicyEngine Macro - -
-
-
-
PolicyEngine Macro · feasibility note · us-hank model
-

Open US HANK models

-

Public code, data, citations, repository activity and analytical coverage. Citation counts use canonical OpenAlex records; stars and last-push dates use the GitHub API. Both are a snapshot, retrieved 12 August 2026 and reproducible with reports/fetch_hank_counts.py; they move continuously, so read them as orders of magnitude rather than as figures. Nothing in this table was run. Every coverage claim is read off the paper and the repository's own documentation, so this is a survey of what is published, not a comparison of what the models do.

-
-
SURVEY AS OF 21 JUL 2026
COUNTS AS OF 12 AUG 2026
A4 · ONE-PAGE EDITION
-
- -

Best complete candidate: Bayer–Born–Lütticke provides the clearest combination of a US-calibrated model, public data, estimation machinery, replication targets and an explicit GPL-3.0 licence — on the published record; none of it has been executed here. Two qualifications belong with the verdict rather than under it. The replication repository is frozen: the authors' own copy was last pushed in October 2022, against a paper published in 2024, and their maintained line has since moved to a general toolbox (BASEtoolbox.jl) rather than this model. And Sequence-Space Jacobian is the only candidate whose repository is still actively pushed — it is the strongest Python foundation, though not itself a fully estimated institutional US model.

- -
- - - - - - - - - - -
Model / resourceYearCitationsStarsCode and dataCoverage
Bayer–Born–Lütticke
Shocks, Frictions, and Inequality in US Business Cycles		20244670
(fork: 4)		Both available
Julia · GPL-3.0
last push Oct 2022		US business cycles; income and wealth inequality; one-/two-asset households; portfolio choice; fiscal and monetary shocks; Bayesian estimation. Aggregate quarterly data approximately 1954Q3–2019Q4.
FRBNY Estimating HANK
Central-bank estimation and forecast evaluation		2023105Code + BBL data
Julia · no licence declared
last push Jun 2024		Recursive Bayesian estimation and pseudo-out-of-sample forecasts; output, consumption, investment, wages, hours, inflation, policy/shadow rates and inequality; comparison with Smets–Wouters.
Kaplan–Moll–Violante
Monetary Policy According to HANK		20181,377N/ABoth available
Fortran · ICPSR; licence unclear		US monetary transmission; liquid and illiquid assets; hand-to-mouth households; earnings risk; wealth and MPC distributions; aggregate consumption responses. Scientific replication rather than forecasting.
Sequence-Space Jacobian
One- and two-asset HANK examples		2021248353Python framework + example inputs
MIT; no full US dataset
last push Feb 2025		Steady states, Jacobians, likelihoods, impulse responses and perfect-foresight transitions. Broad modelling framework, not a specific official US empirical model.
Fed Board HANK Comes of Age
Monetary policy with heterogeneous overlapping generations		2024/250*N/ADownloadable materials
No maintained official GitHub repo; licence unclear		US life cycle: age, labour supply, wealth, retirement, MPCs, monetary transmission, asset repricing, consumption and welfare by age.
-
- -

Source links

-
    -
  1. 1ABBL model code (authors' repository) · their maintained successor · data · paper · citations
    2. -
  2. 1BFRBNY code · report · citations
    3. -
  3. 2Kaplan–Moll–Violante code/data · paper · citations
    4. -
  4. 3Sequence-Jacobian code · one-asset · two-asset · citations
    5. -
  5. 4HANK Comes of Age paper/materials · DOI · citations
    6. -
- -
* OpenAlex reports zero citations on the revised DOI record; citations to earlier versions may be split or delayed.“N/A” means no official GitHub repository—not zero popularity. BBL stars are the authors' own repository (70); this report previously counted FRBNY-DSGE/HANK_BusinessCycleAndInequality, which is a fork of it and holds 4 — fetch_hank_counts.py still points at the fork and needs the same correction.
-
- - diff --git a/reports/us-hank-open-source/index.html b/reports/us-hank-open-source/index.html new file mode 100644 index 00000000..d1fec2f5 --- /dev/null +++ b/reports/us-hank-open-source/index.html @@ -0,0 +1,143 @@ + + + + + +Open US HANK models — code, data, reach — PolicyEngine Macro + + + + + + + + + + + + + + + + + + + + + + +
+ + PolicyEngine Macro + + +
+ + + +
+
+
+

feasibility note — us-hank · survey 21 July 2026 · counts 12 August 2026

+

+ Open US HANK models: code, data, reach. +

+

+ Which published US HANK models ship code and data anyone can run, how + far each is cited and maintained, and what its own documentation says + it covers. +

+
+
+ + + +
+
+ 01 — the field +

Five models, on the published record.

+
+
+

Best complete candidate: Bayer–Born–Lütticke provides the clearest combination of a US-calibrated model, public data, estimation machinery, replication targets and an explicit GPL-3.0 licence — on the published record; none of it has been executed here. Two qualifications belong with the verdict rather than under it. The replication repository is frozen: the authors' own copy was last pushed in October 2022, against a paper published in 2024, and their maintained line has since moved to a general toolbox (BASEtoolbox.jl) rather than this model. And Sequence-Space Jacobian is the only candidate whose repository is still actively pushed — it is the strongest Python foundation, though not itself a fully estimated institutional US model.

+ +

+ Nothing in this table was run. Every coverage claim is + read off the paper and the repository's own documentation, so this is a + survey of what is published, not a comparison of what the models do. +

+ +
+ + + + + + + + + +
Citation counts use canonical OpenAlex records; stars and last-push dates use the GitHub API. Both are a snapshot, retrieved 12 August 2026 and reproducible with reports/fetch_hank_counts.py; they move continuously, so read them as orders of magnitude rather than as figures.
Model / resourceYearCitationsStarsCode and dataCoverage
Bayer–Born–Lütticke
Shocks, Frictions, and Inequality in US Business Cycles		20244670
(fork: 4)		Both available
Julia · GPL-3.0
last push Oct 2022		US business cycles; income and wealth inequality; one-/two-asset households; portfolio choice; fiscal and monetary shocks; Bayesian estimation. Aggregate quarterly data approximately 1954Q3–2019Q4.
FRBNY Estimating HANK
Central-bank estimation and forecast evaluation		2023105Code + BBL data
Julia · no licence declared
last push Jun 2024		Recursive Bayesian estimation and pseudo-out-of-sample forecasts; output, consumption, investment, wages, hours, inflation, policy/shadow rates and inequality; comparison with Smets–Wouters.
Kaplan–Moll–Violante
Monetary Policy According to HANK		20181,377N/ABoth available
Fortran · ICPSR; licence unclear		US monetary transmission; liquid and illiquid assets; hand-to-mouth households; earnings risk; wealth and MPC distributions; aggregate consumption responses. Scientific replication rather than forecasting.
Sequence-Space Jacobian
One- and two-asset HANK examples		2021248353Python framework + example inputs
MIT; no full US dataset
last push Feb 2025		Steady states, Jacobians, likelihoods, impulse responses and perfect-foresight transitions. Broad modelling framework, not a specific official US empirical model.
Fed Board HANK Comes of Age
Monetary policy with heterogeneous overlapping generations		2024/250*N/ADownloadable materials
No maintained official GitHub repo; licence unclear		US life cycle: age, labour supply, wealth, retirement, MPCs, monetary transmission, asset repricing, consumption and welfare by age.
+ +

+ * OpenAlex reports zero citations on the revised DOI record; citations + to earlier versions may be split or delayed. “N/A” means no + official GitHub repository — not zero popularity. BBL stars are the + authors' own repository; the parenthesised figure is the + FRBNY-DSGE fork of it. +

+
+
+ +
+
+ 02 — sources +

Where each row comes from.

+
+
+ +
+
+
+ + + + + diff --git a/site_contract.py b/site_contract.py index 909325d6..e4081aa2 100644 --- a/site_contract.py +++ b/site_contract.py @@ -29,7 +29,7 @@ "frb-us": ("frb-us", "papers/frb-us"), "us-hank": ("us-hank",), "psl-og": ("olg", "papers/psl-og"), - "define-uk": ("define",), + "define-uk": ("define", "papers/define-uk"), } # /docs is a permanent redirect to /models#compare in vercel.json; the model # inventory lives on the pages below. diff --git a/site_nav.py b/site_nav.py index ed0814a2..d343c01f 100644 --- a/site_nav.py +++ b/site_nav.py @@ -82,7 +82,9 @@ "/papers/boe-svar": "boe-svar", "/papers/frb-us": "frb-us", "/papers/psl-og": "psl-og", + "/papers/define-uk": "define-uk", "/reports/define-uk-replication": "define-uk", + "/reports/us-hank-open-source": "Open models", "/forecasts": "Forecasts", "/forecasts/us": "United States", "/notes/releases": "Releases", @@ -105,7 +107,11 @@ "/papers/boe-svar": "/models", "/papers/frb-us": "/models", "/papers/psl-og": "/models", + "/papers/define-uk": "/models", "/reports/define-uk-replication": "/models", + # The open-models survey is one of the five us-hank tabs, so its pathway + # runs through the model rather than straight to /models. + "/reports/us-hank-open-source": "/us-hank", # /notes was folded into /forecasts#notes; the surviving note pages hang # off Forecasts so no crumb links to the retired /notes page. "/notes/releases": "/forecasts", @@ -239,11 +245,12 @@ def header(path: Path) -> str: # Standalone documents allowed to ship without the global nav header. -# Anything else without
fails the check instead of -# being silently skipped forever. -NAV_EXEMPT = { - Path("reports/us-hank-open-source.html"), -} +# Empty: reports/us-hank-open-source was the last one, and a reader clicking +# through from /us-hank landed on a page with no header, no breadcrumb and no +# footer, which reads as leaving the site. It now carries the global chrome +# like every other page. Anything without
fails the check +# instead of being silently skipped forever. +NAV_EXEMPT: set[Path] = set() # Directories that never contain public pages. SKIP_ROOTS = {"vendor", "reveal.js", "audit", "assets", "data"} diff --git a/sitemap.xml b/sitemap.xml index 0392a2e1..e546f9ae 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -102,6 +102,7 @@ https://policyengine-macro.vercel.app/papers/boe-svar0.6 https://policyengine-macro.vercel.app/papers/psl-og0.6 https://policyengine-macro.vercel.app/papers/frb-us0.6 + https://policyengine-macro.vercel.app/papers/define-uk0.6 https://policyengine-macro.vercel.app/reports/us-hank-open-source0.5 https://policyengine-macro.vercel.app/reports/define-uk-replication0.5 diff --git a/style.css b/style.css index 956b45bc..92fa6192 100644 --- a/style.css +++ b/style.css @@ -2420,7 +2420,13 @@ h2 { /* honest-warning cells in validation tables (over-band, imposed, targeted) */ /* Was --brand-strong: the same teal as links, series 1 and .metric-val, so "Weak" read in the site's affirmative accent. --brand-warning is AA-tested. */ -.vchart .vc-warn, td.vc-warn, th.vc-warn { color: var(--brand-warning); font-weight: 600; } +/* SVG text takes `fill`, not `color`, so the shared rule below set a + property nothing reads and `.vchart text { fill: var(--paper-dim) }` won: + every vc-warn line inside a chart rendered as muted grey at the browser + default 16px instead of the warning colour. The HTML cells were always + fine, so the bug was invisible wherever both appeared together. */ +.vchart .vc-warn { fill: var(--brand-warning); font-weight: 600; font-size: 12px; } +td.vc-warn, th.vc-warn { color: var(--brand-warning); font-weight: 600; } /* Methodology step diagrams: dashboard schematics remapped to site tokens, click to expand in a modal. */ diff --git a/svar/validation/index.html b/svar/validation/index.html index 6fb43872..9fe8e37e 100644 --- a/svar/validation/index.html +++ b/svar/validation/index.html @@ -81,13 +81,12 @@

Replicates the paper; forecast evidence remains limited.

- Two independent checks: replication against the published paper, and - — rarer — a 2024Q2-frozen forecast study scored against ONS outturns - that arrived since, a genuine out-of-sample test that exists - because the data edge is frozen. + Two independent checks: replication against the published paper, and a + forecast study frozen at the 2024Q2 data edge and scored against the + ONS outturns that arrived since.

- + @@ -111,19 +110,9 @@

Replicates the paper; forecast evidence remains limited.

- - - - - - - - - - - + - +
This replication against Brignone & Piffer (2025), check by checkCheck by check against Brignone & Piffer (2025); the forecast evidence is below
checkthis replicationthe paper
qualitative match
Out-of-sample forecast error, 2024Q3–2026Q10.32pp RMSE for both GDP growth and CPI inflationno published counterpart
68% band coverage over the same seven quarters14 of 14 outturns inside—
Forecast-revision adding-up identity (software check, not an agreement)Forecast-revision adding-up identity exact per draw (max abs error 5.23e-12)holds by construction — it cannot disagree, and is here only to catch a coding regressionholds by construction — a regression check, not an agreement
@@ -137,22 +126,20 @@

Replicates the paper; forecast evidence remains limited.

68% band coverage 14/14 - no outturn escaped even the 68% band — though the 49-origin rolling evaluation finds the bands under-cover on average + over-dispersion, not a pass — a correct 68% band should hold about 9.5
- tests - comprehensive - estimation, identification, and analysis coverage + vs a drifting random walk + 0.36min q + no variable beats it at any horizon once the 64 tests are adjusted together

- The honest reading: the model was right that the 2025 inflation hump - would mean-revert and wrong about its peak, by up to 0.6 points — - outturns of 3.5 and 3.8% against medians of 3.2%, inside the 68% band - but on its upper half. It never saw the Ofgem cap increases, food - inflation, or the April 2025 administered price rises. Charts and - source data below; full quarterly - scorecard in the working paper. + The model was right that the 2025 inflation hump would mean-revert and + wrong about its peak by up to 0.6pp — outturns of 3.5 and 3.8% against + medians of 3.2%, inside the 68% band but on its upper half. It never saw + the Ofgem cap increases or the April 2025 administered price rises. Full + quarterly scorecard in the working paper.

This is an independent replication built from Bank of England Macro @@ -161,17 +148,12 @@

Replicates the paper; forecast evidence remains limited.

should not be presented as Bank of England estimates.
- How to read the evidence below. The replication - invariants (sign/zero restrictions, FEVD shares) and the public - forecast archive's append-only rule are gated — CI - fails if they break. The 49-origin rolling evaluation and the - band-coverage study were computed once on real data - and are refreshed manually when the code or data change. Only the - first kind is continuously enforced — and within it, the - sign and zero restrictions are imposed by the identification - scheme, so that gate protects the software and cannot be evidence - about the economy. The FEVD shares and the two studies are the - substantive evidence here. + Gated in CI: the replication invariants and the + forecast archive's append-only rule. Computed once and + refreshed by hand: the 49-origin rolling evaluation and the + coverage study. The sign and zero restrictions are imposed by + the identification scheme, so that gate protects the software and + cannot be evidence about the economy.
@@ -182,14 +164,17 @@

Against a Bank of England paper, and against outturns.

- Against Brignone & Piffer (2025), the one-year global-shock - shares of UK forecast-error variance land within about a point of the - paper's benchmarks. + On the paper's own definition — the posterior mean of the per-draw + group share of total four-quarter-ahead variance, which leaves + about a fifth unexplained — identified global shocks explain + 37.4% of UK GDP forecast-error variance against + Brignone & Piffer's ~40%, and 42.3% of UK CPI + against ~50%.

boe-svar: global-shock FEVD shares, ours vs Brignone & Piffer (2025) -Grouped bar chart. Share of UK forecast-error variance attributed to identified global shocks (world demand, energy and supply) four quarters ahead, as a share of total variance — the statistic the paper's Figure 4 plots. For GDP, our 10,000-draw production run gives 37.4% against the paper's 40.0%, a match; for CPI, 42.3% against 50.0%, about eight points short. The posterior band on each is roughly plus or minus fourteen points, wider than the shortfall. The paper's values are approximate. +Grouped bar chart. Share of UK forecast-error variance attributed to identified global shocks (world demand, energy and supply) four quarters ahead, as a share of total variance — the statistic the paper's Figure 4 plots. For GDP, our production artifact gives 37.4% against the paper's 40.0%, 2.6 points short; for CPI, 42.3% against 50.0%, 7.7 points short. Neither gap is resolvable: on the 600-draw check configuration the 68 per cent posterior band on the same share runs 25 to 56 per cent for GDP and 22 to 47 per cent for CPI, both far wider than the shortfall, and the paper's values are read off its figure and approximate. This comparison and the fan chart elsewhere on this page are different runs: the fan chart is the frozen 2024Q2-edge run of 3,000 draws at seed 20250717. 0% @@ -213,77 +198,41 @@

Against a Bank of England paper, and against outturns.

paper UK CPI, 1-yr horizon -
Global shocks = world demand + energy + supply. Ours = 10,000-draw production run; paper values approximate. Global-shock FEVD shares at the one-year horizon: the production run against the paper's published values. Source: papers/boe-svar/figures/comparison_numbers.json and the paper's validation table.
+
Global-shock (world demand + energy + supply) FEVD shares four quarters ahead: the production run against the paper's approximate published values. No committed artifact records that run's draw count, so none is claimed. Source: papers/boe-svar/figures/comparison_numbers.json.
- What this comparison is measuring, and how noisy it is -

- The paper's Figure 4 plots the mean of the sum of the - decomposition, and its note attributes the gap to 100% to the two - unidentified shocks — so its shares are posterior means of - shares of total forecast-error variance, with roughly a fifth - left unexplained. Earlier rounds here published a different quantity: - the sum of per-shock posterior medians, renormalised to 100%. - Those medians sum to about 0.62 for GDP and 0.71 for CPI, so - renormalising inflated every identified share by about a third. That - step, not the model, produced the previous 49.5% CPI figure against - the paper's ~50%. The horizon was also read one column late: column - h of the decomposition holds the (h+1)-step variance, so - “one year” was five quarters. -

+ How noisy that comparison is

- The CI configuration is deliberately cheap — 600 unweighted - draws, of which about 50 are accepted — and it is a noisy - estimator rather than a second opinion: across sampling seeds its - global GDP share has a standard deviation of about 6.5 percentage - points. Both configurations sit inside the wide [30, 60]% acceptance - band CI enforces, which is chosen to catch gross regressions without - being flaky to Monte-Carlo variation. The production run is adequately - converged: quadrupling it to 40,000 draws moves both shares by about a - point. + The 68% posterior band is about ±14pp wide, so the 8pp CPI + shortfall sits inside a single interval: a real gap on the paper's + definition, reported as one, but not resolvable at this sample size.

-
- - - - - - - -
boe-svar global-shock FEVD shares against Brignone & Piffer (2025), on the paper's definition
Global share, 4-quarter FEVDPaperCI config (fast, ~50 draws)Production (10k draws)Production 68% band
UK GDP~40%42.0%37.4%[23.4, 52.2]
UK CPI~50%37.6%42.3%[25.5, 60.3]
-

- The band is the point: it is about ±14pp wide, so the 8pp CPI - shortfall is inside a single posterior interval. It is a real gap on - the paper's own definition and is reported as one, but it is not - resolvable at this sample size. + The CI gate uses a cheap ~50-draw configuration whose global GDP share + moves about ±6.5pp across sampling seeds, checked against a wide + [30, 60]% band. It catches gross regressions; it is not a second + opinion on the production run.

- Out-of-sample. Forecasting seven quarters ahead from - a frozen 2024Q2 edge, RMSE (root-mean-square error) is - 0.32pp for both GDP growth and CPI inflation - (what the median got right and wrong is in the - honest reading above). + Out-of-sample. Seven quarters ahead from the frozen + 2024Q2 edge, RMSE is 0.32pp for both GDP growth and + CPI inflation.

- Fourteen of fourteen outturns inside the 68% band is a - calibration failure, not a pass — a correct 68% interval - should contain roughly 9.5 of 14, and the bands are about three times - wider than the errors warrant, so this is evidence of over-dispersion, - not the validation win it was once presented as. But one frozen origin - yields perhaps two or three effectively independent observations, so - neither reading carries much weight; the test to weight is the - empirical coverage across all 49 rolling origins and eight horizons - below. + Fourteen of fourteen outturns inside the 68% band is + over-dispersion, not a pass — a correct interval should + contain about 9.5 of 14. One frozen origin yields two or three + effectively independent observations, so the reading to weight is the + 49-origin coverage below, which finds the opposite problem.

- + boe-svar: out-of-sample forecast fan from the frozen 2024Q2 edge vs ONS outturns -Two-panel fan chart. Left panel: year-on-year UK GDP growth; right panel: year-on-year UK CPI inflation. Each shows the posterior median forecast from the frozen 2024Q2 data edge as a line, the 68 per cent credible band as a shaded region over thirteen quarters 2024Q3 to 2027Q3, and ONS outturns for the seven evaluated quarters 2024Q3 to 2026Q1 as dots. All fourteen outturn dots fall inside the 68 per cent band; RMSE 0.32 percentage points for both variables. GDP medians run 1.3, 1.8, 1.3, 0.9, 0.8, 0.9, 0.9 per cent over the evaluated quarters against outturns of 1.0, 1.5, 1.3, 1.4, 1.3, 1.0, 0.9; CPI medians 2.3, 2.7, 2.8, 3.2, 3.2, 3.2, 3.2 against outturns of 2.0, 2.5, 2.8, 3.5, 3.8, 3.4, 3.1. Medians and 68 per cent bands from papers/boe-svar/figures/figure_numbers.json (forecast_table, entries [median, lo68, hi68]); ONS outturns from papers/boe-svar/figures/make_figures.py. Coordinates: GDP panel maps value v to y = 292 - (v + 1) * 59 for the -1 to 3 per cent axis; CPI panel y = 292 - (v - 1) * 59 for the 1 to 5 per cent axis; quarter i of 13 maps to x = 58 + i * 25.667 (GDP) or 416 + i * 25.667 (CPI). +Two-panel fan chart. Left panel: year-on-year UK GDP growth; right panel: year-on-year UK CPI inflation. Each shows the posterior median forecast from the frozen 2024Q2 data edge as a line, the 68 per cent credible band as a shaded region over thirteen quarters 2024Q3 to 2027Q3, and ONS outturns for the seven evaluated quarters 2024Q3 to 2026Q1 as dots. All fourteen outturn dots fall inside the 68 per cent band; RMSE 0.32 percentage points on GDP and 0.32 on CPI. GDP medians run 1.3, 1.8, 1.3, 0.9, 0.8, 0.9, 0.9 per cent over the evaluated quarters against outturns of 1.0, 1.5, 1.3, 1.4, 1.3, 1.0, 0.9; CPI medians 2.3, 2.7, 2.8, 3.2, 3.2, 3.2, 3.2 against outturns of 2.0, 2.5, 2.8, 3.5, 3.8, 3.4, 3.1. A dashed rule in each panel marks 2026Q1, the last quarter with an outturn: the six quarters to its right are forecast with nothing yet to check them against, so the fit shown covers seven of the thirteen quarters plotted. This run is 3,000 posterior draws at seed 20250717, 217 of which were accepted by the sign restrictions; it is a different run from the FEVD comparison elsewhere on this page, which comes from the production artifact. Medians and 68 per cent bands from papers/boe-svar/figures/figure_numbers.json (forecast_table, entries [median, lo68, hi68]); ONS outturns from papers/boe-svar/figures/make_figures.py. Coordinates: GDP panel maps value v to y = 292 - (v + 1) * 59 for the -1 to 3 per cent axis; CPI panel y = 292 - (v - 1) * 59 for the 1 to 5 per cent axis; quarter i of 13 maps to x = 58 + i * 25.667 (GDP) or 416 + i * 25.667 (CPI). median forecast + 68% band -ONS outturn -frozen 2024Q2 edge at left of each panel +ONS outturn +frozen 2024Q2 edge at left of each panel -1% @@ -308,6 +257,8 @@

Against a Bank of England paper, and against outturns.

+ + @@ -332,63 +283,74 @@

Against a Bank of England paper, and against outturns.

24Q3 26Q1 27Q3 +dashed rule = last quarter with an ONS outturn (2026Q1); the 6 quarters to its right are not evaluated
14/14 inside the 68% band — see the over-dispersion caveat above. Frozen 2024Q2 data edge; source: papers/boe-svar/figures/figure_numbers.json.

- Rolling-origin audit. A stricter expanding-window test - evaluates 49 historical origins without future-data leakage; the full - benchmark treatment is in the - forecast-accuracy section below. The short - version: no variable clears a 10% false-discovery rate against a - drifting random walk once the 64 tests are adjusted together, and the - single frozen-edge result must not be read as broad forecast - superiority. + Rolling-origin audit. Across 49 expanding-window + origins, no variable beats a drifting random walk at any horizon once + the 64 tests are adjusted together — the + benchmark section below. One frozen origin + is not evidence of broad forecast superiority.

- The svar-unemployment satellite. Unemployment is not - a VAR variable; its published outlook comes from an estimated - GDP-growth-to-unemployment mapping: a regression of the - quarterly change in the ONS unemployment rate on year-on-year GDP - growth, fitted 1992Q1–2025Q1 with furlough dummies for - 2020Q1–2021Q2 (which otherwise attenuate the coefficient about - threefold). In a 73-origin rolling test fed by the VAR's own GDP - forecasts it beats no-change at horizons 1–4 (relative RMSE - 0.82–0.99, excluding furlough targets, not statistically - significant) and is clearly worse beyond, so the published path is - capped at four quarters; the bands carry GDP-forecast uncertainty - only. Full specification and skill table: - unemployment_satellite_validation.json - in the model repository. + The svar-unemployment satellite. Unemployment is not a + VAR variable: its published outlook comes from a regression of the + quarterly change in the ONS rate on year-on-year GDP growth, fitted + 1992Q1–2025Q1 with furlough dummies. It beats no-change only at + horizons 1–4 (relative RMSE 0.82–0.99, not significant), so + the published path is capped there, and its bands carry GDP-forecast + uncertainty only. Specification and skill table: + unemployment_satellite_validation.json.

boe-svar: empirical band coverage across 49 origins -Line chart of empirical interval coverage by forecast horizon, averaged across the eight model variables, against the nominal 68 and 90 percent levels. 68% band, horizons one to eight: 71%, 68%, 64%, 61%, 60%, 60%, 58%, 57%; 90% band, horizons one to eight: 87%, 81%, 81%, 76%, 76%, 73%, 72%, 72%. Both bands under-cover, and coverage worsens with horizon. The window includes the Covid quarters and the evaluation model carries no Covid dummies, which depresses coverage. - -40% - -50% - -60% - -70% - -80% - -90% - -100% - - - - - - +Line chart of empirical interval coverage by forecast horizon, averaged across the eight model variables, against the nominal 68 and 90 percent levels; a vertical bar at each horizon spans the best and worst variable. 68% band, mean coverage at horizons one to eight: 71%, 68%, 64%, 61%, 60%, 60%, 58%, 57%; 90% band, mean coverage at horizons one to eight: 87%, 81%, 81%, 76%, 76%, 73%, 72%, 72%. The 68 per cent band covers 71% at one quarter, just above nominal, and falls below nominal from horizon two onward to 57% at eight; the 90 per cent band under-covers at every horizon, from 87% down to 72%. The mean hides how bad the worst variable is: the lowest cell in the grid is CPI energy at the 68 per cent band, horizon 4, covering 29%. The window includes the Covid quarters and the evaluation model carries no Covid dummies, which depresses coverage. + +30% + +40% + +50% + +60% + +70% + +80% + +90% + +100% + + + + + + + + + + +90% + + + + + + + + + + +68% 68% band, mean coverage across variables 90% band -nominal 68% -nominal 90% +vertical bars span the best and worst of the eight variables · lowest cell 29% (CPI energy, 68% band, h4) +nominal 68% +nominal 90% h1 h2 h3 @@ -398,19 +360,8 @@

Against a Bank of England paper, and against outturns.

h7 h8 -
Empirical interval coverage across 49 expanding-window origins, mean across the eight model variables, by horizon. Predictive bands combine parameter and shock uncertainty (100 posterior draws × 5 paths per origin), percentile intervals on series levels. Both bands under-cover — the 68% band contains outturns 57–71% of the time and the 90% band 72–87%, worsening with horizon. The window includes the Covid quarters and the evaluation model carries no Covid dummies, which depresses coverage; pseudo- rather than real-time out-of-sample, as above. So the frozen-origin over-dispersion reading and this under-coverage reading disagree — a single origin is simply too little data, and the rolling result is the one to weight. Source: papers/boe-svar/figures/coverage_evaluation.json, generated by make_coverage.py.
+
Empirical interval coverage across 49 expanding-window origins, mean across the eight variables, by horizon. The 68% band over-covers at one quarter (71%) and falls below nominal from two onward; the 90% band under-covers at every horizon. Both worsen with horizon — the opposite of the frozen-origin reading above, and the one to weight. The evaluation model carries no Covid dummies over a window that includes the Covid quarters, which depresses coverage. Source: papers/boe-svar/figures/coverage_evaluation.json.
-
- Why there is no official yardstick for those forecast errors -

- No official counterpart exists for those RMSEs. The authors' - companion paper (Staff Working Paper No. 1,165, January 2026) is a - methods paper on a different, four-variable SVAR; it publishes no - root-mean-squared errors, coverage rates or benchmark comparisons, so - the forecast numbers here stand without an official yardstick and are - reported as such. -

-

working paper (pdf) → boe-svar model page → @@ -425,270 +376,236 @@

How does it compare to other forecasters?

- Three different statistics answer three different questions, and they - are not averaged, ranked, or plotted on one axis: this - SVAR's pseudo-out-of-sample accuracy against a naive benchmark - (computed here, quarterly), the OBR's and external forecasters' own - published real-time errors (cited, annual), and the - OBR emulator's tracking of the official - forecast (replication accuracy, not forecasting accuracy). + Three statistics answer three different questions and are + not averaged or ranked: this SVAR's + pseudo-out-of-sample accuracy against naive benchmarks (computed here, + quarterly), the OBR's and external forecasters' published real-time + errors (cited, annual), and the OBR + emulator's tracking of the official forecast (replication, not + forecasting).

No forecasting win survives the number of tests run. - Bank Rate is the closest thing to one — 0.79 against a drift - benchmark at one quarter — but that is the smallest of 64 - variable×horizon tests, and once they are adjusted together - nothing clears a 10% false-discovery rate. CPI's advantage is a - benchmark artefact, and GDP's full-sample loss turns out to be a Covid - artefact. + Bank Rate comes closest — 0.79 against a drifting random walk at one + quarter, and the smallest of 64 variable×horizon tests at + p = 0.018 — but adjusted together + (Benjamini–Hochberg) the minimum q is 0.36, so + nothing clears a 10% false-discovery rate on any variable at any + horizon.

Benchmark and sample diagnostics

boe-svar against a random walk, computed from our own runs

- An expanding-window pseudo-out-of-sample exercise re-fits the BVAR at - 49 quarterly origins (2012Q1–2024Q1, data sample - 1992Q1–2026Q1) and scores forecasts one to eight quarters ahead - against three naive benchmarks. A ratio below 1.0 means the model beats - the benchmark. -

-

- The benchmark decides the answer. A no-change random - walk on a trending log level forfeits the whole trend as forecast - error, so beating it on a price index is close to uninformative. - Against a random walk with drift — the textbook naive - for a trending series — the CPI result largely evaporates: - 0.63 becomes 0.83 at one quarter and - 0.67 becomes 1.03 at eight, where the model is no - longer ahead at all. The pattern in the original numbers, wins only on - the two trending price series and ties or losses on the four series - where a random walk is a genuinely hard benchmark, was an artefact of - the benchmark rather than a finding about the model. -

-

- Bank Rate comes closest, and still does not clear the bar. - It is the one series here that does not trend, so no-change is - the right naive for it, and it is the one series that improves under - the harder benchmark: 0.79 against drift at one quarter - and 0.85 at eight. Its Diebold–Mariano p of 0.018 is the number - this page used to lead with — but the evaluation runs 64 tests - per benchmark, and the smallest of 64 p-values is not a discovery. - Adjusting the whole grid together (Benjamini–Hochberg), the - minimum q-value against drift is 0.36, and 0.40 under - the published specification below. Nothing survives at a 10% - false-discovery rate, on any variable at any horizon. + Expanding-window design: 49 quarterly origins + (2012Q1–2024Q1), estimation from 1992Q1, data through 2026Q1, + horizons 1–8. A ratio below 1.0 beats the benchmark.

- The evaluation was scoring a model this site does not - publish. Every forecast shown here is estimated with six - Covid dummies (2020Q1–2021Q2); the rolling evaluation had always - left them out, so 2020Q2 entered the likelihood as an ordinary - observation. Re-run on the published specification, UK GDP against - drift goes 1.06 → 0.99 at one quarter and - 1.12 → 0.95 at eight, and its ex-Covid one-quarter - ratio goes 1.21 → 0.97. CPI moves by less than 0.02 and Bank Rate - by less than 0.02, so the inflation weakness is not a Covid - artefact while the GDP weakness substantially was. The dummies are - real-time safe: a dummy column for a quarter after the origin is - identically zero in training. Both blocks are published in the source - file; the charts below still show the headline (no-dummy) block, and - the published-spec ratios are the ones to weight. + The benchmark decides the answer. A no-change walk on + a trending log level forfeits the whole trend as error, so beating it + on a price index is close to uninformative: UK CPI's 0.63 at one + quarter becomes 0.83 against a walk with + drift, and 0.67 at eight becomes 1.03. Bank Rate + is the one series that does not trend, and the one that improves under + the harder benchmark.

- UK GDP is still not distinguishable from the benchmark. - Its drift-benchmark ratios are not statistically separable at any - horizon (p = 0.33 to 0.43), and that is true in both - specifications — moving from worse-than-naive to level with it is - not the same as beating it. Two numbers this page previously placed in - a drift-benchmark paragraph belong to the weaker no-change benchmark: - the p = 0.38–0.67 range, and the ex-Covid - 0.77, which against drift is 1.21. - Under squared loss the 2020Q2 collapse dominates a 49-origin average; - both the full-sample and the excluding-Covid figures are published, and - neither is the preferred number. + Covid dummies matter for GDP, not for inflation. The + rolling evaluation omits the six Covid dummies (2020Q1–2021Q2) + that every published forecast here carries. On the published + specification UK GDP against drift goes 1.06 → 0.99 + at one quarter and 1.12 → 0.95 at eight, while + CPI and Bank Rate move by less than 0.02. Level with a naive benchmark + is still not beating it: UK GDP is statistically inseparable from drift + at every horizon (p = 0.33–0.43) in both + specifications.

- The AR(1) comparison is reported in the source file but is - not usable at long horizons: 94.5% of its eight-step - squared error for UK GDP comes from a single origin, because an AR(1) - extrapolates the Covid collapse. The apparent 3:1 win against it is - arithmetic, not evidence. Every benchmark now carries a - worst_origin_mse_share diagnostic for exactly this reason. + The AR(1) comparison in the source file is not usable at long + horizons: 94.5% of its eight-step squared error for UK GDP + comes from one origin, because an AR(1) extrapolates the Covid + collapse. Every benchmark carries a worst_origin_mse_share + diagnostic for that reason.

- + boe-svar forecast skill against a random walk with drift, all eight variables -Dot matrix of forecast error relative to a random walk with drift, eight variables by eight quarterly horizons, from 49 expanding-window origins. A ratio below 1.0 means the model beats the benchmark; filled dots mark differences significant at 5 per cent by a Diebold-Mariano test, hollow dots differences that are not statistically distinguishable; those p-values are pairwise and the grid runs 64 tests, none of which clears a 10 per cent false-discovery rate once adjusted together. Bank Rate runs 0.79 at one quarter to 0.85 at 8; UK CPI runs 0.83 at one quarter to 1.03 at 8; World CPI runs 0.94 at one quarter to 1.07 at 8; Oil price runs 1.01 at one quarter to 0.98 at 8; CPI energy runs 0.98 at one quarter to 1.11 at 8; UK real GDP runs 1.06 at one quarter to 1.12 at 8; World GDP runs 1.07 at one quarter to 1.10 at 8; Exchange rate runs 1.03 at one quarter to 1.33 at 8. Bank Rate is the best at the longest horizon and Exchange rate the worst. Against this harder benchmark only four of eight variables beat naive at one quarter and two at 8. -RMSE ÷ drifting-random-walk RMSE · horizons 1–8 quarters -filled = difference significant at 5% (Diebold–Mariano); hollow = not distinguishable - -0.8 - -0.9 - -1.1 - -1.2 - -1.3 - -1.0 -← model better -benchmark better → -Bank Rate - - - - - - - - - -0.85 -UK CPI - - - - - - - - - -1.03 -World CPI - - - - - - - - - -1.07 -Oil price - - - - - - - - - -0.98 -CPI energy - - - - - - - - - -1.11 -UK real GDP - - - - - - - - - -1.12 -World GDP - - - - - - - - - -1.10 -Exchange rate - - - - - - - - - -1.33 +Dot matrix of forecast error relative to a random walk with drift, eight variables by eight quarterly horizons, from 49 expanding-window origins. Position alone carries better or worse: a dot left of the 1.0 rule means the model beats the benchmark, right of it means the benchmark wins. Filled dots mark differences significant at 5 per cent by a Diebold-Mariano test, hollow dots differences that are not statistically distinguishable; those p-values are pairwise and the grid runs 64 tests, none of which clears a 10 per cent false-discovery rate once adjusted together — the smallest adjusted q-value is 0.36. Bank Rate runs 0.79 at one quarter to 0.85 at 8; UK CPI runs 0.83 at one quarter to 1.03 at 8; World CPI runs 0.94 at one quarter to 1.07 at 8; Oil price runs 1.01 at one quarter to 0.98 at 8; CPI energy runs 0.98 at one quarter to 1.11 at 8; UK real GDP runs 1.06 at one quarter to 1.12 at 8; World GDP runs 1.07 at one quarter to 1.10 at 8; Exchange rate runs 1.03 at one quarter to 1.33 at 8. Bank Rate is the best at the longest horizon and Exchange rate the worst. Against this harder benchmark only four of eight variables beat naive at one quarter and two at 8. The number at the end of each row is that row's ratio at horizon 8. +RMSE ÷ drifting-random-walk RMSE · horizons 1–8 quarters +filled = significant at 5% pairwise · hollow = not distinguishable +large dot and row-end number = h=8 · smallest adjusted q over the 64 cells: 0.36 + +0.8 + +0.9 + +1.1 + +1.2 + +1.3 + +1.0 +← model better +benchmark better → +Bank Rate + + + + + + + + + +0.85 +UK CPI + + + + + + + + + +1.03 +World CPI + + + + + + + + + +1.07 +Oil price + + + + + + + + + +0.98 +CPI energy + + + + + + + + + +1.11 +UK real GDP + + + + + + + + + +1.12 +World GDP + + + + + + + + + +1.10 +Exchange rate + + + + + + + + + +1.33 -
Expanding-window pseudo-out-of-sample error ratios against a random walk
with drift, on the level of each series. A driftless walk is too weak a benchmark for a trending series, so this is the fair comparison; the no-change ratios are in the table below. 49 forecast origins, 2012Q1–2024Q1 (the window includes the Covid quarters); estimation sample from 1992Q1, data through 2026Q1; 4 lags. Estimation uses final revised data, so this is pseudo- rather than real-time out-of-sample. These ratios are concentrated in a few origins: in 16 of the 64 variable–horizon cells a single origin contributes more than 35% of the drift-benchmark squared error, peaking at 63% for UK GDP at h=1, so individual ratios rest on far fewer effective observations than 49 origins suggests. Source: papers/boe-svar/figures/rolling_evaluation.json (worst_origin_mse_share.drift). +
Error ratios against a random walk with drift, on the level of each series — the fair benchmark for a trending series. Estimated without the Covid dummies the published specification carries; no-change ratios are in the table below. Estimation uses final revised data, so this is pseudo- rather than real-time out-of-sample. The ratios rest on fewer effective observations than 49 origins suggests: in 16 of the 64 cells one origin contributes over 35% of the squared error, peaking at 63% for UK GDP at h=1. Source: papers/boe-svar/figures/rolling_evaluation.json (worst_origin_mse_share.drift).
- + How many of eight variables beat a drifting random walk, by horizon -Stacked column chart. Of the model's eight forecast variables, how many have lower root mean squared error than a random walk with drift, at horizons one to eight quarters. The counts are 4, 4, 3, 2, 3, 3, 2, 2 respectively. Each column also separates wins whose difference is statistically significant at 5 per cent by a Diebold-Mariano test from wins that are not: significant wins number 2, 1, 1, 1, 1, 0, 0, 0. From horizon 6 onward no variable beats the benchmark by a statistically significant margin. +Column chart, one column per forecast horizon from one to eight quarters. Column height is the number of the model's eight forecast variables whose root mean squared error is below that of a random walk with drift; a dashed reference line across the top marks all eight. The counts are 4, 4, 3, 2, 3, 3, 2, 2, so between 2 and 4 of the 8 beat the benchmark at any horizon and the remaining 4, 4, 5, 6, 5, 5, 6, 6 do not. The shaded lower part of each column is the subset whose difference is significant at 5 per cent on an unadjusted pairwise Diebold-Mariano test: 2, 1, 1, 1, 1, 0, 0, 0. Those p-values are pairwise and the grid runs 64 tests; under a Benjamini-Hochberg adjustment none of them reaches a 10 per cent false-discovery rate and the smallest adjusted q-value is 0.36, so on the adjusted reading the significant-win count is zero at every horizon. At horizons 7 and 8 one variable is significantly worse than the benchmark. Of 8 forecast variables, how many beat a random walk with drift - -2 - -4 - -6 - -8 - -0 - - - -4 -h=1 - - - -4 -h=2 - - - -3 -h=3 - - - -2 -h=4 - - - -3 -h=5 - - -3 -h=6 - - -2 -h=7 - - -2 -h=8 -beats it, and the difference is significant -beats it, not significantly -does not beat it +column height = variables with lower RMSE than the benchmark · the rest do not beat it + +2 + +4 + +6 + +0 + +8 +all 8 variables + + +4 +h=1 +2 + + +4 +h=2 +1 + + +3 +h=3 +1 + + +2 +h=4 +1 + + +3 +h=5 +1 + +3 +h=6 +0 + +2 +h=7 +0 + +2 +h=8 +0 +p<0.05 +beats it, p<0.05 unadjusted +beats it, not distinguishable +Adjusted together, none of the 64 tests clears a 10% false-discovery rate (smallest q = 0.36). -
Wins are counted at a ratio strictly below 1.0, which is a hard cut: a variable at 0.99 counts as a win and one at 1.01 does not, even though neither is distinguishable from the benchmark. That is why the significance split matters more than the count — and the significance shown is pairwise, unadjusted for the 64 tests the grid runs. Under a Benjamini–Hochberg adjustment the significant-win count is zero at every horizon. Same 49 origins and benchmark as the chart above. Source:
papers/boe-svar/figures/rolling_evaluation.json. +
A win is a ratio strictly below 1.0 — a hard cut that counts 0.99 and rejects 1.01 though neither is distinguishable from the benchmark, so the significance split matters more than the count. That significance is pairwise: under a Benjamini–Hochberg adjustment the significant-win count is zero at every horizon. Same origins and benchmark as above. Source: papers/boe-svar/figures/rolling_evaluation.json.
- + @@ -704,22 +621,19 @@

boe-svar against a random walk, computed from our own runs

boe-svar forecast error relative to two naive benchmarks (ratio; below 1.0 beats the benchmark). Bold marks a win that is statistically significant; * marks any significant difference. These are unadjusted pairwise p-values: the grid runs 64 tests per benchmark, and adjusting them together leaves nothing below a 10% false-discovery rate, so read a star as "worth a second look", not as a result. Ratios are the headline block, estimated without Covid dummies; the published specification carries them, and its UK GDP ratios are 0.99 / 0.99 / 0.97 / 0.95 against drift. All eight variables are shown — the four previously omitted were among the weakest.Forecast error relative to two naive benchmarks (below 1.0 beats it). Bold marks a significant win, * any significant difference — unadjusted, so read a star as "worth a second look". Ratios omit the Covid dummies the published specification carries (see the diagnostics above). The ex-Covid column is against no-change — UK GDP's 0.77 there is 1.21 against drift — and neither it nor the full-sample figure is preferred.
variableh=1h=2h=4h=8h=1 ex-Covid
(vs no-change)
vs driftvs no-changevs driftvs no-changevs driftvs no-changevs driftvs no-change

- Bank Rate crosses parity beyond h=3 - (source file has all eight variables and horizons). These are - quarterly, level-basis, pseudo-out-of-sample ratios computed with - hindsight-final data — they are not comparable - with the real-time annual-growth errors the OBR publishes, which is - why they get separate tables. + These are quarterly, level-basis ratios computed with hindsight-final + data. They are not comparable with the real-time + annual-growth errors the OBR publishes, which is why those get a + separate table.

What the OBR and external forecasters report about themselves

- The OBR's Forecast - Evaluation Report (July 2025) publishes its own real-time errors - on annual growth rates since 2010, beside the median of external - forecasters compiled by HM Treasury. Cited, not recomputed; the - statistic is the median absolute error on annual rates, so it cannot - be compared with the quarterly RMSE ratios above. + Cited, not recomputed: the OBR's + Forecast + Evaluation Report (July 2025) reports its own real-time errors on + annual growth rates since 2010, beside the HM Treasury median of + external forecasters.

@@ -734,21 +648,21 @@

What the OBR and external forecasters report about themselves

- The nearest in-house analogue is the frozen-edge experiment above: - 0.32pp RMSE on quarterly year-on-year GDP growth and - CPI inflation over seven quarters. Different statistic, frequency, - and origin count, so it sits beside the OBR's numbers, not against - them. For the Bank of England, the only precisely sourceable - comparison is the matched-vintage episode on the - validation index (August 2024 MPR - modal CPI of ~2.4% for late 2025 vs an outturn peak of 3.8%); the MPR - publishes no RMSE-by-horizon table. + The nearest in-house analogue is the frozen-edge + 0.32pp above — a different statistic, frequency and + origin count, so it sits beside these numbers, not against them. For + the Bank of England the only sourceable comparison is the + matched-vintage episode on the + validation index; the MPR publishes no + RMSE-by-horizon table.

- Omitted for lack of a defensible source: NIESR or consensus RMSE - tables by horizon, a Bank of England RMSE-by-horizon table, and an - AR(1) benchmark for the SVAR exercise. Each needs a new computation - or source before it can appear here. + Omitted for lack of a defensible source: NIESR or consensus RMSE tables + by horizon, a Bank of England RMSE-by-horizon table, and an AR(1) + benchmark for this exercise. The authors' companion paper (Staff + Working Paper No. 1,165) is a methods paper on a different SVAR and + publishes no errors, so the frozen-edge RMSEs stand without an official + yardstick.
@@ -765,23 +679,23 @@

Where this departs from the Bank's model.

Frozen data edge - Coefficient estimation to 2025Q1; conditioning data to 2026Q1 (refreshed July 2026). Results shift with data revisions. The replication claims are scored on the extended 1992Q1–2025Q1 sample, not the paper's original 1992Q1–2023Q2 window. + Estimation to 2025Q1, conditioning to 2026Q1 — the extended sample, not the paper's 1992Q1–2023Q2 window. Results shift with revisions. Proxied world aggregates - The Bank's internal UK-trade-weighted world GDP and CPI are unpublished, so they are rebuilt as chain-weighted US + euro-area + Japan + China aggregates with time-varying UK trade weights. + The Bank's UK-trade-weighted world GDP and CPI are unpublished, so they are rebuilt as chain-weighted US + euro-area + Japan + China with time-varying UK trade weights. One ranking does not replicate - The paper ranks UK monetary policy as the largest domestic contributor to CPI variance; here UK supply (13.5%) exceeds monetary policy (10.1%). Attributed to the proxy world aggregates and the smaller accepted sample, and documented rather than tuned away. + The paper's prose ranks UK monetary policy as the largest domestic contributor to CPI variance; here UK supply (13.9%) and UK demand (11.4%) both exceed it (8.7%). The paper publishes no FEVD table, so treat this as unresolved rather than a defect; the proxy aggregates are a candidate cause. Assumed lag length and simplified pandemic prior - p = 4 is assumed, not selected; the pandemic treatment is simplified to exogenous Covid dummies. + p = 4 is assumed, not selected; the pandemic is handled by exogenous Covid dummies. Sign-restriction critiques apply - Pointwise medians mix structural models (Fry–Pagan); the Haar prior over rotations is informative about impulse responses (Baumeister–Hamilton). These apply to the Bank's own outputs identically. + Pointwise medians mix structural models (Fry–Pagan); the Haar prior over rotations is informative about impulse responses (Baumeister–Hamilton). These hit the Bank's own outputs identically. diff --git a/us-hank/validation/index.html b/us-hank/validation/index.html index f8aa7f79..265025b1 100644 --- a/us-hank/validation/index.html +++ b/us-hank/validation/index.html @@ -83,88 +83,93 @@

A replication gate, not a forecast-accuracy claim.

- The ground truth is the published paper. The model repository's - replication suite gates the parameters the model - solves for internally against the published Econometrica 2021 table, - plus market clearing and the signs and timing of the shock responses. - The steady-state calibration targets are gated too, but they - are imposed inputs rather than results — the figure below reads - the top five rows that way. The - implementation builds on the authors' own - sequence-jacobian toolkit rather than re-deriving the - model. Fidelity is strong, the claim narrow: evidence the replication - is right, not that the model predicts the US economy — no predictive - validation exists or is claimed. The hosted steady state is set - against the paper's published calibration targets, value by value, - below. + The ground truth is the published paper, not data. The parameters the + model solves for internally reproduce Auclert et al. (2021) + Table B.III to the last published digit on six of seven: + β 0.9762739 against 0.976, χ1 6.4164196 + against 6.416 — solved numerically from a 6.5 starting guess, the + strongest single check — and Z, α, μp and τ + analytically. The seventh is unresolved: this + implementation returns vphi (the wage Phillips curve + disutility scale) of 1.7135 where Table B.III publishes 2.073, + −17.3%, left unasserted in the test suite rather than laundered. +

+

+ Signs and timing of the shock responses are gated too. + Magnitudes are not — the paper publishes no impulse + response to match them against.

- -us-hank: steady state vs published calibration targets -Table-style chart of the hosted two-asset steady state against the published calibration targets of Auclert, Bardóczy, Rognlie and Straub (2021). Output Y: target 1, achieved 1, deviation 0; Capital K: target 10, achieved 10, deviation 0; Real rate r (quarterly): target 0.0125, achieved 0.0125, deviation 0; Government spending G: target 0.2, achieved 0.2, deviation 0; Government debt Bg: target 2.8, achieved 2.8, deviation 0; Total wealth A+B: target 14, achieved 14, deviation 2.7e-13; Liquid assets B: target 1.04, achieved 1.04, deviation 9.5e-14; Discount factor beta: target 0.976, achieved 0.976274, deviation 2.7e-04. Goods and asset market clearing residuals are 3.3e-08 and 3.1e-13. Beta is the calibrated free parameter that hits the wealth targets. -target -achieved -|deviation| -Output Y -1 -1 -0 - -Capital K -10 -10 -0 - -Real rate r (quarterly) -0.0125 -0.0125 -0 - -Government spending G -0.2 -0.2 -0 - -Government debt Bg -2.8 -2.8 -0 - -Total wealth A+B -14 -14 -2.7e-13 - -Liquid assets B -1.04 -1.04 -9.5e-14 - -Discount factor beta -0.976 -0.976274 -2.7e-04 -Market clearing: goods 3.3e-08, assets 3.1e-13. Achieved values solved from the hosted adapter; targets from the upstream test suite citing the paper. + +us-hank: internally solved parameters vs Auclert et al. (2021) Table B.III +Table-style chart of the parameters the us-hank two-asset model solves for internally, against the values Auclert, Bardóczy, Rognlie and Straub (2021) print in Table B.III. These are not imposed: two are solved numerically and four analytically from the calibration inputs, so they had somewhere to miss. beta (discount factor): published 0.976, ours 0.9762739, solved numerically; chi1 (portfolio adjustment cost scale): published 6.416, ours 6.41642, solved numerically; Z (TFP): published 0.468, ours 0.4677898, solved analytically; alpha (capital share): published 0.33, ours 0.3299492, solved analytically; mu_p (price markup): published 1.015, ours 1.015228, solved analytically; tau (labour tax rate): published 0.356, ours 0.3560606, solved analytically. All six agree to the last digit the paper prints. The seventh, vphi (wage Phillips curve disutility scale), does not: the paper publishes 2.073 and this implementation returns 1.713476, a gap of -17.3 per cent, left unasserted in the upstream test suite rather than laundered. Separately, the 5 rows Output Y, Capital K, Real rate r (quarterly), Government spending G, Government debt Bg are calibration inputs held fixed through the solve, so their zero deviations are identities and carry no information, and Total wealth A+B and Liquid assets B are reached by the solver, so their residuals of 2.7e-13 and 9.5e-14 are tolerances rather than accuracies. Market clearing: goods 3.3e-08, which is untargeted and holds only by Walras' law, and assets 3.1e-13, which is the root-finder's own convergence target. +Solved by the model, compared against the published table +solved +published +ours +|diff| + +beta (discount factor) +numerically +0.976 +0.9762739 +2.7e-04 + +chi1 (portfolio adjustment cost scale) +numerically +6.416 +6.41642 +4.2e-04 + +Z (TFP) +analytically +0.468 +0.4677898 +2.1e-04 + +alpha (capital share) +analytically +0.33 +0.3299492 +5.1e-05 + +mu_p (price markup) +analytically +1.015 +1.015228 +2.3e-04 + +tau (labour tax rate) +analytically +0.356 +0.3560606 +6.1e-05 + +vphi (wage Phillips curve disutility scale) +unresolved +2.073 +1.713476 +3.6e-01 + +Six of seven agree to the last published digit; vphi misses by -17.3%. +Calibration inputs — held fixed through the solve, so their deviations are identities +Constants equalling themselves: Output Y 1, Capital K 10, Real rate r 0.0125, Government spending G 0.2, +Government debt Bg 2.8. +Solver targets: Total wealth A+B 14 hit to 2.7e-13, Liquid assets B 1.04 hit to 9.5e-14 — tolerances, not +accuracies. +Market clearing: goods 3.3e-08 untargeted (Walras’ law), assets 3.1e-13 is the solver’s own target. -
Hosted two-asset steady state against the paper's published calibration targets, solved 2026-07-29 at the production grids (n
B=50, nA=70, nK=50). Read the top five rows as inputs, not results: Y, K, r, G and Bg are literal entries in the calibration dictionary, held fixed through the solve, so their zero deviation is an arithmetic tautology and carries no information. The informative rows are the last three — total wealth and liquid assets, which the solver has to hit through market clearing, and β, the one free parameter, whose 0.976274 is the only value here compared against a number the paper reports. Source: papers/us-hank/figures/replication.json. +
Calibration, not replication. Y, K, r, G, Bg and the wealth rows are imposed and held fixed through the solve, so their deviations are identities or solver tolerances rather than accuracies; The rows the model solves for are the replication: six reproduce Table B.III to the last published digit and vphi misses by 17.3%. Same for the residuals: asset-market clearing at 3.1e−13 is the solver's own target, while goods-market clearing at 3.3e−08 is untargeted — it holds only by Walras' law, which makes it the honest measure. Solved 2026-07-29 at production grids (nB=50, nA=70, nK=50), re-solved independently 2026-08-13. Source: papers/us-hank/figures/replication.json.
-
- This validates replication fidelity, not prediction. - The model generates deterministic, first-order responses around a - calibrated steady state. It does not forecast the US economy, quantify - parameter uncertainty, or map detailed tax reforms into its stylized - instruments. -

- An independent replication of a published academic model — not - produced or endorsed by the paper's authors. Results are model + An independent replication of a published academic model, built on the + authors' own sequence-jacobian + toolkit — not produced or endorsed by them. Results are model counterfactuals, not official estimates or forecasts.

-

- For how this model sits among other open-source HANK implementations, - see the open US HANK - model landscape report. The toolkit it builds on is at - shade-econ/sequence-jacobian. +

+ where this sits among other open HANK implementations: + the landscape report.

@@ -181,23 +186,23 @@

Known limitations.

Not a forecaster - No data, no estimation, no forecast origin: every result is an impulse response around the paper's calibrated steady state. Use boe-svar (UK) for forecasts; nothing in this suite forecasts the US. + No data, no estimation, no forecast origin: every result is an impulse response around the paper's calibrated steady state. Nothing in this suite forecasts the US. Stylized instruments only - Three shock kinds — monetary, fiscal spending, productivity. The labor tax is endogenous (the fiscal block balances the budget): no transfer or tax-rate instrument, and score_reform refuses model="hank" rather than inventing a reform mapping. + Monetary, fiscal-spending and productivity shocks; the labor tax is endogenous. No transfer or tax-rate instrument, so score_reform refuses model="hank". First-order responses - IRFs are linear in shock size: no state dependence, no zero lower bound, no asymmetry between large/small or positive/negative shocks. + IRFs are linear in shock size: no state dependence, no zero lower bound, no asymmetry. Parameter uncertainty is not quantified — the calibration is a single point. Approximate distributional output - The by-quartile consumption response allocates the aggregate response in proportion to steady-state MPCs — the MPC-heterogeneity channel, not full household-level dynamics along the transition. + By-quartile consumption splits the aggregate response in proportion to steady-state MPCs — the MPC-heterogeneity channel, not household-level dynamics along the transition. Calibration, not vintage - The parameterisation is the published paper's; there is no data vintage to refresh. Revisiting the calibration would be a deliberate model change, not a data update. + The parameterisation is the paper's; there is no data vintage to refresh. Changing it would be a model change. diff --git a/validation/figures/chart_data.json b/validation/figures/chart_data.json index 29c1792c..99a807ab 100644 --- a/validation/figures/chart_data.json +++ b/validation/figures/chart_data.json @@ -103,16 +103,17 @@ }, "frbus_residuals": { "source": "papers/frb-us/sections/validation.tex, Tables tab:tracking and tab:refnoise", + "_series_comment": "series is the row's KIND of comparison, not a decoration: 1 = ours against the Fed's implementation under shock (the evidence), 2 = the Fed's own two releases against each other (the reference-noise yardstick our rows are judged against), 3 = a tracking identity that re-solves to its own input and therefore cannot fail. Series 3 was previously 1 on the first row, which painted the row the page calls 'not evidence' in the accent colour.", "rows": [ { - "label": "ours vs LONGBASE (identity \u2014 not evidence)", + "label": "ours vs LONGBASE \u2014 tracking identity, not evidence", "value": 5.6e-17, - "series": 1, + "series": 3, "source": "validation.tex tab:tracking, max abs error", "note": "init_trac defines add-factors as minus the residuals at the input data, so re-solving reproduces it up to float64 for ANY input \u2014 verified: the same test passes at 6.7e-9 on randomly scrambled data. Shown for completeness; it measures arithmetic, not economics." }, { - "label": "pyfrbus 1.1.1 vs LONGBASE", + "label": "pyfrbus 1.1.1 vs LONGBASE \u2014 tracking identity", "value": 1.1e-08, "series": 3, "source": "validation.tex subsec:refnoise prose" @@ -130,11 +131,20 @@ "source": "validation.tex tab:refnoise row 2" }, { - "label": "pyfrbus 1.1.1 vs 1.0.0 \u2014 the Fed\u2019s own two releases", + "label": "pyfrbus 1.1.1 vs 1.0.0 \u2014 the Fed\u2019s own releases", "value": 1.3e-08, "series": 2, "source": "validation.tex tab:refnoise row 3" } ] + }, + "define_emissions": { + "_comment": "What the define-uk divergence gate actually pins is the RATIO of the pinned run to the manual's Table 4, not the two levels: the levels in validation/figures/data/define_emissions_divergence.csv are rounded to whole MtCO2e, so recomputing the percentage from them gives -3.4 / -10.2 / -23.1 where the gate and the validation page say -3.5 / -10.2 / -23.3. The bars plot the rounded levels; the percentages come from the gated ratios below.", + "source": "validation/reference_outputs.json in the PolicyEngine define-uk adapter repository (the divergence gate, pinned to \u00b10.02), as recorded in that repository's VALIDATION.md target 1b and published on define/validation/", + "gated_ratios": { + "2025": 0.965, + "2030": 0.898, + "2040": 0.767 + } } } diff --git a/validation/figures/make_charts.py b/validation/figures/make_charts.py index 66edd948..d198681f 100644 --- a/validation/figures/make_charts.py +++ b/validation/figures/make_charts.py @@ -39,6 +39,26 @@ def esc(s: str) -> str: return s.replace("&", "&").replace("<", "<").replace(">", ">") +def wrap(text: str, width: int = 105): + """Greedy word wrap for a run of lines. + + SVG text does not wrap, and these charts carry no foreignObject, so a note + long enough to overflow the viewBox silently spills outside the card border + (.vchart sets overflow: visible). Anything variable-length gets wrapped. + """ + lines, cur = [], "" + for w in text.split(): + trial = f"{cur} {w}".strip() + if len(trial) > width and cur: + lines.append(cur) + cur = w + else: + cur = trial + if cur: + lines.append(cur) + return lines + + def n(x: float) -> str: """Format a coordinate the way the committed markup does: 1dp, no trailing .0 loss.""" return f"{x:.1f}" @@ -206,9 +226,12 @@ def ymap(v): # legend out.append('') - out.append(f'real GDP (peak {max(gdp):+.2f}%)') + # "peak" is the largest deviation in either direction, not the largest + # positive one: max() would quietly report the least-negative value on a + # series that never rose above the EFO. + out.append(f'real GDP (peak {max(gdp, key=abs):+.2f}%)') out.append('') - out.append(f'consumption (peak {max(cons):+.2f}%)') + out.append(f'consumption (peak {max(cons, key=abs):+.2f}%)') # y grid for tick in (-0.6, -0.3, 0.0, 0.3, 0.6): @@ -308,6 +331,11 @@ def chart_pe_costing(): def chart_svar_fevd(): ours, paper = load_svar() + d = json.loads((ROOT / "papers" / "boe-svar" / "figures" / "comparison_numbers.json").read_text()) + fast = d["fast_config"] + fan = json.loads( + (ROOT / "papers" / "boe-svar" / "figures" / "figure_numbers.json").read_text() + ) groups = [ {"label": "UK GDP, 1-yr horizon", "bars": [{"name": "ours", "value": ours["gdp"], "series": 1}, @@ -316,13 +344,24 @@ def chart_svar_fevd(): "bars": [{"name": "ours", "value": ours["cpi"], "series": 1}, {"name": "paper", "value": paper["cpi"], "series": 2}]}, ] + # The draw count of the production run is not recorded in the artifact, so + # it is not asserted here. The 68% bands are the 600-draw check + # configuration's, and are labelled as such rather than attached to the + # production bars as if they had been measured on them. desc = (f"Grouped bar chart. Share of UK forecast-error variance attributed to identified " f"global shocks (world demand, energy and supply) four quarters ahead, as a share of " - f"total variance — the statistic the paper's Figure 4 plots. For GDP, our 10,000-draw " - f"production run gives {ours['gdp']:.1f}% against the paper's {paper['gdp']:.1f}%, a " - f"match; for CPI, {ours['cpi']:.1f}% against {paper['cpi']:.1f}%, about eight points " - f"short. The posterior band on each is roughly plus or minus fourteen points, wider " - f"than the shortfall. The paper's values are approximate.") + f"total variance — the statistic the paper's Figure 4 plots. For GDP, our production " + f"artifact gives {ours['gdp']:.1f}% against the paper's {paper['gdp']:.1f}%, " + f"{abs(ours['gdp'] - paper['gdp']):.1f} points short; for CPI, {ours['cpi']:.1f}% " + f"against {paper['cpi']:.1f}%, {abs(ours['cpi'] - paper['cpi']):.1f} points short. " + f"Neither gap is resolvable: on the {fast['n_draws']}-draw check configuration the " + f"68 per cent posterior band on the same share runs {fast['gdp_68_band_pct'][0]:.0f} " + f"to {fast['gdp_68_band_pct'][1]:.0f} per cent for GDP and " + f"{fast['cpi_68_band_pct'][0]:.0f} to {fast['cpi_68_band_pct'][1]:.0f} per cent for " + f"CPI, both far wider than the shortfall, and the paper's values are read off its " + f"figure and approximate. This comparison and the fan chart elsewhere on this page " + f"are different runs: the fan chart is the frozen 2024Q2-edge run of " + f"{fan['n_draws']:,} draws at seed {fan['seed']}.") return grouped_bars( "svar-fevd", "boe-svar: global-shock FEVD shares, ours vs Brignone & Piffer (2025)", @@ -330,19 +369,36 @@ def chart_svar_fevd(): def chart_frbus_residuals(): + """Residuals against the Fed's own pyfrbus, on a log axis. + + The fills used to run brand / faint / brand / brand / mid down the rows, + which encoded nothing a reader could recover: the tracking identity — the + row the page itself says is *not* evidence — was painted in the accent + colour and read as the headline. The three fills now name three kinds of + comparison, each row label carries the same distinction as text, and the + two identity rows also carry a dashed outline so the grouping survives + without colour. + """ data = load_transcribed()["frbus_residuals"] rows = data["rows"] - W, H = 760, 300 - x0, x1 = 380.0, 692.7 # 1e-18 .. 1e-8 - lo_exp, hi_exp = -18, -8 + W, H = 760, 330 + x0, x1 = 380.0, 692.7 # 1e-18 .. 1e-7 + lo_exp, hi_exp = -18, -7 per_decade = (x1 - x0) / (hi_exp - lo_exp) + def group(r): + return {1: "ours vs the Fed", 2: "the Fed vs itself"}.get( + r["series"], "tracking identity") + desc = ("Horizontal bar chart on a base-10 logarithmic axis of maximum absolute residuals; " - "shorter is closer. " + - "; ".join(f"{r['label']}: {r['value']:.1e}" for r in rows) + - ". The framing comparison is the last row: the Federal Reserve's own two pyfrbus " - "releases disagree with each other by as much as this implementation disagrees with " - "either, so our agreement sits at the scale of the reference implementation's own " + "shorter is closer. The rows fall into three kinds. " + + "; ".join(f"{r['label']}: {r['value']:.1e} ({group(r)})" for r in rows) + + ". The two tracking rows are an identity — init_trac defines the add-factors as " + "minus the residuals at the input data, so re-solving reproduces the input for any " + "input at all — and are shown for completeness, not as evidence. The framing " + "comparison is the last row: the Federal Reserve's own two pyfrbus releases " + "disagree with each other by as much as this implementation disagrees with either, " + "so our agreement sits at the scale of the reference implementation's own " "numerical noise rather than at a chosen tolerance.") out = svg_open(W, H, "frbus-residuals", @@ -358,10 +414,23 @@ def chart_frbus_residuals(): y = 34.0 + 48 * i ty = y + 21 w = (math.log10(r["value"]) - lo_exp) * per_decade + dash = (' stroke="currentColor" stroke-dasharray="3 2" stroke-width="1"' + if r["series"] == 3 else "") out.append(f'{esc(r["label"])}') - out.append(f'') + out.append(f'') out.append(f'{r["value"]:.1e}') + out.append('' + 'ours vs the Fed’s pyfrbus, ' + 'under shock') + out.append('' + 'the Fed’s two releases vs each ' + 'other') + out.append('' + 'tracking identity — round-off, ' + 'not fidelity') out.append("") return "\n".join(out) @@ -405,11 +474,14 @@ def ymap(v): "obr-macro: real GDP level, anchored vs free-running vs the March 2026 EFO (£bn/qtr)", desc) - out.append('') + # Legend swatches use the classes the marks are actually drawn with: these + # three series are plain paths, so no dots, and the EFO swatch is vc-s3 — + # it was drawn as a hairline vc-grid, which is not the line on the chart. + out.append('') out.append(f'anchored ({mad_a:.2f}% MAD)') - out.append('') + out.append('') out.append(f'free-running ({mad_f:.2f}% MAD)') - out.append('EFO Mar 2026') + out.append('EFO Mar 2026') tick = lo while tick <= hi + 1e-9: @@ -442,7 +514,7 @@ def chart_obr_outturn(): group_step = 165.0 first_centre = 138.5 - parts = "; ".join(f"{q}: emulator {m:.2f}, EFO {e:.2f}, ONS {o:g}" for q, m, e, o in rows) + parts = "; ".join(f"{q}: emulator {m:.2f}, EFO {e:.2f}, ONS {o:.2f}" for q, m, e, o in rows) desc = ( f"Grouped bar chart, percentage quarter-on-quarter real GDP growth for the four " f"quarters with ONS outturns since anchoring. {parts}. The emulator tracks the three " @@ -467,15 +539,20 @@ def chart_obr_outturn(): out.append(f'{t:g}') t += 0.2 + # Three bars per group, two of them neutral greys: the legend is not enough + # on its own, so each bar also carries its own name under it. for g, (q, m, e, o) in enumerate(rows): centre = first_centre + g * group_step - for k, (cls, v) in enumerate((("vc-b1", m), ("vc-b3", e), ("vc-b2", o))): + for k, (cls, name, v) in enumerate((("vc-b1", "emul.", m), ("vc-b3", "EFO", e), + ("vc-b2", "ONS", o))): x = centre - 60.0 + k * (bar_w + bar_gap) h = v * scale y = base_y - h + cx = x + bar_w / 2 out.append(f'') - out.append(f'{v:.2f}') - out.append(f'{q}') + out.append(f'{v:.2f}') + out.append(f'{name}') + out.append(f'{q}') out.append("") return "\n".join(out) @@ -483,7 +560,10 @@ def chart_obr_outturn(): def chart_svar_fan(): quarters, series, outturns = load_svar_fan() - W, H = 760, 340 + meta = json.loads( + (ROOT / "papers" / "boe-svar" / "figures" / "figure_numbers.json").read_text() + ) + W, H = 760, 360 base_y, top_y = 292.0, 56.0 per_unit = (base_y - top_y) / 4.0 # 4-unit span on each panel panels = {"gdp": {"x0": 58.0, "x1": 366.0, "v_lo": -1.0}, @@ -509,9 +589,16 @@ def chart_svar_fan(): f"edge as a line, the 68 per cent credible band as a shaded region over thirteen quarters " f"{quarters[0]} to {quarters[-1]}, and ONS outturns for the seven evaluated quarters " f"{quarters[0]} to {quarters[n_out - 1]} as dots. All fourteen outturn dots fall inside " - f"the 68 per cent band; RMSE {rmse['gdp']:.2f} percentage points for both variables. GDP " + f"the 68 per cent band; RMSE {rmse['gdp']:.2f} percentage points on GDP and " + f"{rmse['cpi']:.2f} on CPI. GDP " f"medians run {med_g} per cent over the evaluated quarters against outturns of {out_g}; " - f"CPI medians {med_c} against outturns of {out_c}. Medians and 68 per cent bands from " + f"CPI medians {med_c} against outturns of {out_c}. A dashed rule in each panel marks " + f"{quarters[n_out - 1]}, the last quarter with an outturn: the six quarters to its right " + f"are forecast with nothing yet to check them against, so the fit shown covers seven of " + f"the thirteen quarters plotted. This run is {meta['n_draws']:,} posterior draws at seed " + f"{meta['seed']}, {meta['accepted_forecast']} of which were accepted by the sign " + f"restrictions; it is a different run from the FEVD comparison elsewhere on this page, " + f"which comes from the production artifact. Medians and 68 per cent bands from " f"papers/boe-svar/figures/figure_numbers.json (forecast_table, entries [median, lo68, " f"hi68]); ONS outturns from papers/boe-svar/figures/make_figures.py. Coordinates: GDP " f"panel maps value v to y = {base_y:g} - (v + 1) * {per_unit:g} for the -1 to 3 per cent " @@ -524,8 +611,8 @@ def chart_svar_fan(): desc) out.append('median forecast + 68% band') - out.append('ONS outturn') - out.append('frozen 2024Q2 edge at left of each panel') + out.append('ONS outturn') + out.append('frozen 2024Q2 edge at left of each panel') gdp_labels = {-1: "-1%", 0: "0", 1: "+1%", 2: "+2%", 3: "+3%"} for var, labels in (("gdp", gdp_labels), ("cpi", None)): @@ -548,6 +635,14 @@ def chart_svar_fan(): for var in ("gdp", "cpi"): p = panels[var] out.append(f'') + # Where the evidence stops. Without this the six unevaluated quarters on the + # right of each panel read as part of the fit. Drawn before the median and + # the dots so it does not strike through the last outturn. + for var in ("gdp", "cpi"): + p = panels[var] + x_end = p["xs"][n_out - 1] + out.append(f'') for var in ("gdp", "cpi"): p = panels[var] pts = " L".join(f"{n(p['xs'][i])} {n(p['ymap'](v))}" for i, v in enumerate(series[var]["median"])) @@ -563,6 +658,9 @@ def chart_svar_fan(): p = panels[var] for i in (0, (npts - 1) // 2, npts - 1): out.append(f'{quarters[i][2:]}') + out.append(f'dashed rule = last quarter with an ONS ' + f'outturn ({quarters[n_out - 1]}); the {npts - n_out} quarters to its right are ' + f'not evaluated') out.append("") return "\n".join(out) @@ -580,13 +678,14 @@ def chart_obr_computed_share(): computed, passthrough = d["computed"], d["passthrough"] grades = d["grades"] - W, H = 760, 290 + W, H = 760, 314 x0, x1 = 64.0, 724.0 span = x1 - x0 per_var = span / total in_band = sum(g["count"] for g in grades if g["in_band"]) trivial = sum(g["count"] for g in grades if g.get("trivial")) + nontrivial = computed - trivial off = next(g for g in grades if g["key"] == "off") desc = ( @@ -600,8 +699,9 @@ def chart_obr_computed_share(): for g in grades ) + f". {in_band} of the {computed}, or {100 * in_band / computed:.0f} per cent, land " - f"within band, and {word(trivial)} of those is a trivial accounting identity, so only " - f"{in_band - trivial} non-trivial computed variables are in band. The worst are " + f"within band, but {word(trivial)} of those is an accounting identity that closes " + f"over passthrough inputs, so {in_band - trivial} of the {nontrivial} non-trivial " + f"computed variables are in band. The worst are " + " and ".join(off["examples"]) + "." ) @@ -609,37 +709,49 @@ def chart_obr_computed_share(): out = svg_open(W, H, "obr-computed-share", "How much of the OBR emulator scorecard the model actually computes", desc) - # Row 1 — computed vs passthrough, out of the full scorecard. + # Row 1 — computed vs passthrough, out of the full scorecard. Segment labels + # sit below their segment rather than on it: a label printed over a filled + # bar has to clear the fill in both themes, and these did not. split = x0 + computed * per_var out.append(f'{total} headline scorecard variables') - out.append(f'') - out.append(f'') - out.append(f'' + out.append(f'') + out.append(f'') + out.append(f'' f'{computed} computed') - out.append(f'' + out.append(f'' f'{passthrough} passthrough — held at the OBR value') - out.append(f'') - out.append(f'') + out.append(f'') + out.append(f'') # Row 2 — the computed variables broken down by grade, on the same scale, so # the second bar reads as a zoom into the first bar's left-hand segment. - out.append(f'of which, the {computed} the model ' + out.append(f'of which, the {computed} the model ' f'computes') x = x0 - for g in grades: + for i, g in enumerate(grades): w = g["count"] * per_var + cx = x + w / 2 + # 'poor' and 'off' share a fill, so 'off' also carries a dashed outline + # and both carry their label as text under the segment. Labels alternate + # between two baselines with a leader on the lower one: a one-variable + # segment is 31px wide and its label is not, so on a single baseline the + # narrow ones run into their neighbours. dash = ' stroke="currentColor" stroke-dasharray="3 2" stroke-width="1"' if g["key"] == "off" else "" - out.append(f'') - out.append(f'' + ly = 219 if i % 2 == 0 else 237 + if ly > 219: + out.append(f'') + out.append(f'' f'{esc(g["label"])} {g["count"]}') x += w - out.append(f'Only {in_band - trivial} of {computed} ' - f'non-trivial variables are in band.') - out.append('A fourth pass is an identity over ' - 'passthrough inputs.') - out.append(f'{esc(d["bands_note"])}') + out.append(f'Only {in_band - trivial} of the ' + f'{nontrivial} non-trivial computed variables are in band.') + out.append(f'{word(trivial).capitalize()} of the ' + f'{in_band} in-band passes is an accounting identity over passthrough ' + f'inputs.') + out.append(f'{esc(d["bands_note"])}') out.append("") return "\n".join(out) @@ -664,7 +776,7 @@ def chart_obr_computed_share(): def chart_svar_skill_all(): horizons, ratio, pval, meta = load_rolling_eval() - W, H = 760, 404 + W, H = 760, 420 x_one, per_unit = 370.8, 883.0 # value 1.0 at x_one def xs(v): return x_one + (v - 1.0) * per_unit @@ -674,15 +786,18 @@ def xs(v): worst = max(SKILL_ROWS, key=lambda r: ratio[last][r[0]]) n_win_1 = sum(1 for k, _ in SKILL_ROWS if ratio[horizons[0]][k] < 1) n_win_last = sum(1 for k, _ in SKILL_ROWS if ratio[last][k] < 1) + adj = meta["multiplicity"]["headline"]["drift"] desc = ( f"Dot matrix of forecast error relative to a random walk with drift, " f"{word(len(SKILL_ROWS))} variables by {word(len(horizons))} quarterly horizons, from " - f"{meta['origins']} expanding-window origins. A ratio below 1.0 means the model " - f"beats the benchmark; filled dots mark differences significant at 5 per cent by a " + f"{meta['origins']} expanding-window origins. Position alone carries better or worse: " + f"a dot left of the 1.0 rule means the model beats the benchmark, right of it means " + f"the benchmark wins. Filled dots mark differences significant at 5 per cent by a " f"Diebold-Mariano test, hollow dots differences that are not statistically " - f"distinguishable; those p-values are pairwise and the grid runs 64 tests, none of " - f"which clears a 10 per cent false-discovery rate once adjusted together. " + f"distinguishable; those p-values are pairwise and the grid runs {adj['tests']} " + f"tests, none of which clears a 10 per cent false-discovery rate once adjusted " + f"together — the smallest adjusted q-value is {adj['min_q']:.2f}. " + "; ".join( f"{label} runs {ratio[horizons[0]][key]:.2f} at one quarter to " f"{ratio[last][key]:.2f} at {last}" @@ -690,29 +805,33 @@ def xs(v): ) + f". {best[1]} is the best at the longest horizon and {worst[1]} the worst. " f"Against this harder benchmark only {word(n_win_1)} of {word(len(SKILL_ROWS))} " - f"variables beat naive at one quarter and {word(n_win_last)} at {last}." + f"variables beat naive at one quarter and {word(n_win_last)} at {last}. The number " + f"at the end of each row is that row's ratio at horizon {last}." ) out = svg_open(W, H, "svar-skill-all", "boe-svar forecast skill against a random walk with drift, " "all eight variables", desc) - out.append('RMSE ÷ drifting-random-walk RMSE ' + out.append('RMSE ÷ drifting-random-walk RMSE ' '· horizons 1–8 quarters') - out.append('filled = difference significant at 5% ' - '(Diebold–Mariano); hollow = not distinguishable') + out.append('filled = significant at 5% pairwise ' + '· hollow = not distinguishable') + out.append(f'large dot and row-end number = h={last} ' + f'· smallest adjusted q over the {adj["tests"]} cells: ' + f'{adj["min_q"]:.2f}') for tick in (0.8, 0.9, 1.1, 1.2, 1.3): x = xs(tick) - out.append(f'') - out.append(f'{tick}') - out.append(f'') - out.append(f'1.0') - out.append(f'' + out.append(f'') + out.append(f'{tick}') + out.append(f'') + out.append(f'1.0') + out.append(f'' f'← model better') - out.append(f'benchmark better →') + out.append(f'benchmark better →') for i, (key, label) in enumerate(SKILL_ROWS): - y = 78.0 + 38 * i + y = 94.0 + 38 * i values = [ratio[h][key] for h in horizons] out.append(f'' f'{esc(label)}') @@ -720,14 +839,17 @@ def xs(v): f'x2="{n(xs(max(values)))}" y2="{n(y)}"/>') for h in horizons: v, p = ratio[h][key], pval[h][key] - r = 4.0 if h == last else 2.6 + r = 4.0 if h == last else 3.0 + # One colour for every dot: better-or-worse is already carried by + # which side of the 1.0 rule the dot sits on, so colouring by that + # again would be an encoding that adds nothing. Fill vs outline is + # the only categorical channel here, and it means significance. if p < SIGNIF: - series = 1 if v < 1 else 2 - out.append(f'') else: out.append(f'') + f'stroke="currentColor" stroke-width="1.2" opacity="0.7"/>') out.append(f'' f'{ratio[last][key]:.2f}') @@ -736,7 +858,20 @@ def xs(v): def chart_svar_winrate(): - horizons, ratio, pval, _meta = load_rolling_eval() + """How many of the eight variables beat the drift benchmark, by horizon. + + This was a stacked column of wins / significant wins / losses summing to + eight at every horizon, so every column was the same height and the height + carried nothing; the count label was also drawn above the top of the wins + stack, i.e. inside the losses segment it was not describing. What actually + varies is the win count, so that is what the bar height encodes now, with + the eight-variable ceiling drawn as a reference line rather than as a + segment. Losses are the gap between the column and that line, and are + stated in the description; the significance split stays as the shaded lower + part of the column, with the count repeated as text under each column so + the distinction never rests on fill alone. + """ + horizons, ratio, pval, meta = load_rolling_eval() keys = [k for k, _ in SKILL_ROWS] total = len(keys) @@ -744,67 +879,95 @@ def chart_svar_winrate(): for h in horizons: wins = [k for k in keys if ratio[h][k] < 1] sig = [k for k in wins if pval[h][k] < SIGNIF] - counts.append({"h": h, "wins": len(wins), "sig": len(sig)}) + lost_sig = [k for k in keys if ratio[h][k] >= 1 and pval[h][k] < SIGNIF] + counts.append({"h": h, "wins": len(wins), "sig": len(sig), + "lost_sig": len(lost_sig)}) + + # The pairwise p-values above are not the last word: the same artifact + # carries the Benjamini-Hochberg adjustment over the whole grid, and + # against this benchmark nothing survives it. + adj = meta["multiplicity"]["headline"]["drift"] + worse = [c["h"] for c in counts if c["lost_sig"]] - W, H = 760, 300 + W, H = 760, 344 x0, x1 = 64, 724 - y_zero, y_top = 250.0, 60.0 - per_var = (y_zero - y_top) / total + y_zero, y_full = 248.0, 76.0 + per_var = (y_zero - y_full) / total - first_dry = next((c["h"] for c in counts if c["sig"] == 0), None) desc = ( - f"Stacked column chart. Of the model's {word(total)} forecast variables, how many have " - f"lower root mean squared error than a random walk with drift, at horizons one to " - f"{word(horizons[-1])} quarters. The counts are " + f"Column chart, one column per forecast horizon from one to {word(horizons[-1])} " + f"quarters. Column height is the number of the model's {word(total)} forecast " + f"variables whose root mean squared error is below that of a random walk with " + f"drift; a dashed reference line across the top marks all {word(total)}. The counts " + f"are " + ", ".join(str(c["wins"]) for c in counts) - + " respectively. Each column also separates wins whose difference is statistically " - "significant at 5 per cent by a Diebold-Mariano test from wins that are not: " - "significant wins number " + + f", so between {min(c['wins'] for c in counts)} and " + f"{max(c['wins'] for c in counts)} of the {total} beat the benchmark at any horizon " + f"and the remaining " + + ", ".join(str(total - c["wins"]) for c in counts) + + " do not. The shaded lower part of each column is the subset whose difference is " + "significant at 5 per cent on an unadjusted pairwise Diebold-Mariano test: " + ", ".join(str(c["sig"]) for c in counts) - + "." - + (f" From horizon {first_dry} onward no variable beats the benchmark by a " - f"statistically significant margin." if first_dry else "") + + f". Those p-values are pairwise and the grid runs {adj['tests']} tests; under a " + f"Benjamini-Hochberg adjustment none of them reaches a 10 per cent false-discovery " + f"rate and the smallest adjusted q-value is {adj['min_q']:.2f}, so on the adjusted " + f"reading the significant-win count is zero at every horizon." + + (f" At horizon{'s' if len(worse) > 1 else ''} " + + " and ".join(str(h) for h in worse) + + " one variable is significantly worse than the benchmark." if worse else "") ) out = svg_open(W, H, "svar-winrate", "How many of eight variables beat a drifting random walk, by horizon", desc) - out.append(f'Of {total} forecast variables, how many ' + out.append(f'Of {total} forecast variables, how many ' f'beat a random walk with drift') + out.append(f'column height = variables with lower ' + f'RMSE than the benchmark · the rest do not beat it') - for k in (2, 4, 6, 8): + for k in (2, 4, 6): y = y_zero - k * per_var out.append(f'') out.append(f'{k}') out.append(f'') out.append(f'0') + # The eight-variable ceiling is a reference, not a segment: no column's + # height is fixed by it, and the empty space above each column is the + # count of variables that do not beat the benchmark. + out.append(f'') + out.append(f'{total}') + out.append(f'' + f'all {total} variables') bar_w, pitch = 56, 80 for i, c in enumerate(counts): bx = x0 + 12 + i * pitch mid = bx + bar_w / 2 - # Bottom-up: significant wins, then the rest of the wins, then the losses. y_sig = y_zero - c["sig"] * per_var y_win = y_zero - c["wins"] * per_var if c["sig"]: out.append(f'') if c["wins"] > c["sig"]: + # 2px of surface between the two parts so they read as two. + top = y_sig - 2 if c["sig"] else y_sig out.append(f'') - out.append(f'') - out.append(f'') + # Above the column it counts, not above a segment it does not. + out.append(f'{c["wins"]}') - out.append(f'' + out.append(f'' f'h={c["h"]}') - - out.append('' - 'beats it, and the difference is ' - 'significant') - out.append('' - 'beats it, not significantly') - out.append('' - 'does not beat it') + out.append(f'' + f'{c["sig"]}') + out.append('p<0.05') + + out.append('' + 'beats it, p<0.05 unadjusted') + out.append('' + 'beats it, not distinguishable') + out.append(f'Adjusted together, none of the ' + f'{adj["tests"]} tests clears a 10% false-discovery rate (smallest q = ' + f'{adj["min_q"]:.2f}).') out.append("") return "\n".join(out) @@ -821,50 +984,82 @@ def mean_cov(level, h): return sum(vals) / len(vals) series = {lv: [mean_cov(lv, h) for h in horizons] for lv in (68, 90)} + # The mean is not the whole story: at every horizon the eight variables are + # spread across tens of points, and the worst of them is far below the mean. + # Plot that spread, and set the axis floor below it rather than above it. + spread = {lv: [(min(cov[str(lv)][f"h{h}"].values()), + max(cov[str(lv)][f"h{h}"].values())) for h in horizons] + for lv in (68, 90)} + worst_lv, worst_h, worst_key, worst_v = min( + ((lv, h, k, v) for lv in (68, 90) for h in horizons + for k, v in cov[str(lv)][f"h{h}"].items()), + key=lambda t: t[3], + ) + worst_var = dict(SKILL_ROWS).get(worst_key, worst_key) W, H = 760, 330 x0, x1 = 58.0, 724.0 step = (x1 - x0) / (len(horizons) - 1) xs = [x0 + i * step for i in range(len(horizons))] - lo_v, hi_v = 0.4, 1.0 - base_y, top_y = 282.0, 40.0 + # Floor the axis below the worst cell rather than above it, so nothing is + # clipped and the shortfall is not compressed against the frame. + hi_v = 1.0 + lo_v = math.floor((worst_v - 0.03) * 20) / 20 + base_y, top_y = 282.0, 56.0 def ymap(v): return base_y - (v - lo_v) * (base_y - top_y) / (hi_v - lo_v) desc = ("Line chart of empirical interval coverage by forecast horizon, averaged " "across the eight model variables, against the nominal 68 and 90 percent " - "levels. " + + "levels; a vertical bar at each horizon spans the best and worst variable. " + "; ".join( - f"{lv}% band, horizons one to eight: " + + f"{lv}% band, mean coverage at horizons one to eight: " + ", ".join(f"{v:.0%}" for v in series[lv]) for lv in (68, 90) ) + - ". Both bands under-cover, and coverage worsens with horizon. The window " + f". The 68 per cent band covers {series[68][0]:.0%} at one quarter, just above " + f"nominal, and falls below nominal from horizon two onward to {series[68][-1]:.0%} " + f"at eight; the 90 per cent band under-covers at every horizon, from " + f"{series[90][0]:.0%} down to {series[90][-1]:.0%}. The mean hides how bad the " + f"worst variable is: the lowest cell in the grid is {worst_var} at the " + f"{worst_lv} per cent band, horizon {worst_h}, covering {worst_v:.0%}. The window " "includes the Covid quarters and the evaluation model carries no Covid " "dummies, which depresses coverage.") out = svg_open(W, H, "svar-coverage", "boe-svar: empirical band coverage across 49 origins", desc) - for v in (0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0): + for k in range(int(math.ceil(lo_v * 10)), 11): + v = k / 10 out.append(f'') out.append(f'{v:.0%}') - for lv, cls in ((90, "vc-s2"), (68, "vc-s1")): + for lv, cls, dodge in ((90, "vc-s2", 4.0), (68, "vc-s1", -4.0)): # nominal level as a reference line out.append( f'' ) + for x, (v_lo, v_hi) in zip(xs, spread[lv]): + out.append(f'') pts = " ".join(f"{n(x)},{n(ymap(v))}" for x, v in zip(xs, series[lv])) out.append(f'') out.append( - f' ' + f'{lv}%' ) out.append(f'68% band, mean coverage across variables') out.append(f'90% band') - out.append(f'nominal 68%') - out.append(f'nominal 90%') + out.append(f'vertical bars span the best and ' + f'worst of the eight variables · lowest cell {worst_v:.0%} ' + f'({worst_var}, {worst_lv}% band, h{worst_h})') + # Mid-plot, where neither reference label can be mistaken for a series + # end-label: at the right-hand edge "nominal 68%" landed beside the 90% + # line's own label. + x_nom = (xs[3] + xs[4]) / 2 + out.append(f'nominal 68%') + out.append(f'nominal 90%') for i, h in enumerate(horizons): out.append(f'h{h}') out.append("") @@ -872,57 +1067,132 @@ def ymap(v): def chart_hank_targets(): + """The us-hank replication that could actually fail. + + This chart used to tabulate the eight rows of ``targets``, six of which are + literal entries in the calibration dictionary held fixed through the solve. + "target 10, achieved 10" on capital is a constant equalling itself: its zero + deviation is an identity, so the chart was a column of eight passes that no + implementation could ever fail, and the description told screen-reader users + that beta was "the calibrated free parameter that hits the wealth targets" — + the framing the artifact's own ``beta_note`` retired. + + The comparison that has somewhere to miss is ``published_table_b3``: the + parameters the model solves for internally, against the values Auclert, + Bardóczy, Rognlie and Straub (2021) print in Table B.III — including the one + that misses by 17 per cent. That is what the rows show now; the calibration + inputs are summarised underneath as what they are. + """ data = json.loads( (ROOT / "papers" / "us-hank" / "figures" / "replication.json").read_text() ) - rows = data["targets"] - W, H = 760, 300 - row_h = (264.0 - 40.0) / len(rows) + b3 = data["published_table_b3"] + params = b3["parameters"] + miss = b3["unexplained_discrepancy"] + targets = data["targets"] + res = data["residuals"] - def dev(r): - d = abs(r["achieved"] - r["target"]) - return "0" if d == 0 else f"{d:.1e}" + def how(p): + return "numerically" if p["solved"].startswith("numerically") else "analytically" + + rows = [{"name": p["name"], "tag": how(p), "pub": p["published"], "ours": p["ours"]} + for p in params] + rows.append({"name": miss["name"], "tag": "unresolved", + "pub": miss["published"], "ours": miss["ours"]}) + + imposed = [t for t in targets if "imposed" in t["kind"]] + solver = [t for t in targets if "through a solver" in t["kind"]] + agree = len(params) + + col_tag, col_pub, col_ours, col_dev = 470, 570, 660, 736 + row_h = 26.0 + top = 56.0 + def short(name): + return name.split(" (")[0] + + notes = [] + for text in ( + "Constants equalling themselves: " + + ", ".join(f'{short(t["name"])} {t["target"]:g}' for t in imposed) + ".", + "Solver targets: " + + ", ".join(f'{short(t["name"])} {t["target"]:g} hit to ' + f'{abs(t["achieved"] - t["target"]):.1e}' for t in solver) + + " — tolerances, not accuracies.", + f'Market clearing: goods {res[0]["value"]:.1e} untargeted (Walras’ law), ' + f'assets {res[1]["value"]:.1e} is the solver’s own target.', + ): + notes.extend(wrap(text)) + W = 760 + H = int(top + row_h * (len(rows) + 2) + 20 + 16 * len(notes) + 12) - desc = ("Table-style chart of the hosted two-asset steady state against the " - "published calibration targets of Auclert, Bardóczy, Rognlie and Straub (2021). " + - "; ".join( - f"{r['name']}: target {r['target']:g}, achieved {r['achieved']:.6g}, " - f"deviation {dev(r)}" - for r in rows - ) + - ". Goods and asset market clearing residuals are " - f"{data['residuals'][0]['value']:.1e} and {data['residuals'][1]['value']:.1e}. " - "Beta is the calibrated free parameter that hits the wealth targets.") + desc = ( + "Table-style chart of the parameters the us-hank two-asset model solves for " + "internally, against the values Auclert, Bardóczy, Rognlie and Straub (2021) " + "print in Table B.III. These are not imposed: two are solved numerically and " + "four analytically from the calibration inputs, so they had somewhere to miss. " + + "; ".join( + f"{r['name']}: published {r['pub']:g}, ours {r['ours']:.7g}, solved " + f"{r['tag']}" for r in rows[:agree] + ) + + f". All {word(agree)} agree to the last digit the paper prints. The seventh, " + f"{miss['name']}, does not: the paper publishes {miss['published']:g} and this " + f"implementation returns {miss['ours']:.7g}, a gap of {miss['gap_pct']:g} per cent, " + "left unasserted in the upstream test suite rather than laundered. Separately, the " + f"{len(imposed)} rows " + + ", ".join(t["name"] for t in imposed) + + " are calibration inputs held fixed through the solve, so their zero deviations are " + "identities and carry no information, and " + + " and ".join(t["name"] for t in solver) + + " are reached by the solver, so their residuals of " + + " and ".join(f"{abs(t['achieved'] - t['target']):.1e}" for t in solver) + + " are tolerances rather than accuracies. Market clearing: goods " + f"{res[0]['value']:.1e}, which is untargeted and holds only by Walras' law, and " + f"assets {res[1]['value']:.1e}, which is the root-finder's own convergence target." + ) out = svg_open(W, H, "hank-targets", - "us-hank: steady state vs published calibration targets", desc) + "us-hank: internally solved parameters vs Auclert et al. (2021) Table B.III", + desc) - hdr_y = 30.0 - col_target, col_achieved, col_dev = 400, 545, 700 - out.append(f'target') - out.append(f'achieved') - out.append(f'|deviation|') + out.append('Solved by the model, compared against ' + 'the published table') + out.append(f'solved') + out.append(f'published') + out.append(f'ours') + out.append(f'|diff|') for i, r in enumerate(rows): - y = 40.0 + row_h * i + y = top + row_h * i ty = y + row_h / 2 + 4 - if i: - out.append(f'') + out.append(f'') out.append(f'{esc(r["name"])}') - out.append(f'{r["target"]:g}') - out.append(f'{r["achieved"]:.6g}') - out.append(f'{dev(r)}') - res = data["residuals"] - out.append( - f'Market clearing: goods {res[0]["value"]:.1e}, ' - f'assets {res[1]["value"]:.1e}. Achieved values solved from the hosted adapter; ' - "targets from the upstream test suite citing the paper." - ) + out.append(f'' + f'{r["tag"]}') + out.append(f'{r["pub"]:g}') + out.append(f'{r["ours"]:.7g}') + out.append(f'' + f'{abs(r["ours"] - r["pub"]):.1e}') + last_y = top + row_h * len(rows) + out.append(f'') + + out.append(f'{word(agree).capitalize()} ' + f'of seven agree to the last published digit; {miss["name"].split()[0]} misses by ' + f'{miss["gap_pct"]:g}%.') + out.append(f'Calibration inputs — ' + f'held fixed through the solve, so their deviations are identities') + for i, line in enumerate(notes): + out.append(f'{line}') out.append("") return "\n".join(out) def chart_define_emissions(): rows = load_define_emissions() + # The percentage is the gated quantity and is pinned in the adapter's + # reference_outputs.json; recomputing it from the whole-MtCO2e levels in the + # CSV drifts by up to 0.2 points and disagreed with the number the page and + # VALIDATION.md publish. Levels come from the CSV, percentages from the gate. + gated = load_transcribed()["define_emissions"]["gated_ratios"] + gap = {yr: (gated[yr] - 1) * 100 for yr, _, _ in rows} W, H = 760, 304 base_y, top_y = 258.0, 26.0 y_max = 450.0 @@ -936,15 +1206,18 @@ def chart_define_emissions(): # on the same page — a screen-reader user got a different number on a # validation page whose whole point is that the numbers are gated. parts = "; ".join(f"{yr}: pinned run {p:g}, published table {m:g}, " - f"a gap of {(p / m - 1) * 100:.1f} per cent" + f"a gated ratio of {gated[yr]:.3f}, a gap of {gap[yr]:.1f} per cent" for yr, p, m in rows) desc = ( "Grouped bar chart of S1 baseline total UK emissions in MtCO2e per year, " "the cached pinned run of upstream commit 846081a against the manual's " f"published Table 4. {parts}. The pinned code runs below the published " - "table and the gap widens with horizon; the divergence is gated at the " - "observed ratios so any further drift fails loudly. Data from " - "validation/figures/data/define_emissions_divergence.csv." + "table and the gap widens with horizon; what is gated is the ratio of the " + "two, pinned to plus or minus 0.02, so any further drift fails loudly. " + "Bar heights are the levels in " + "validation/figures/data/define_emissions_divergence.csv, rounded to whole " + "MtCO2e; the percentages are the gated ratios themselves, which is why they " + "differ slightly from the ratio of the rounded bars." ) out = svg_open(W, H, "define-emissions", "define-uk: S1 baseline total emissions — pinned run vs manual Table 4 (MtCO2e/yr)", @@ -970,7 +1243,7 @@ def chart_define_emissions(): out.append(f'{v:g}') out.append(f'{name}') out.append(f'' - f'{esc(yr)} ({(p / m - 1) * 100:.0f}%)') + f'{esc(yr)} ({gap[yr]:+.1f}%)') out.append("") return "\n".join(out)