Model-repo review round: correct what the site could not support - #174
Merged
Conversation
Three model-repo reviews and two recovered page audits. The findings that
were verifiable and mine to fix.
Two published commands could not run. /connect's dynamic-scoring recipe gave
`pe-macro og-score --json > og.json` and `pe-macro dynamic-score --og-payload
og.json`; --reform is required=True on both, so each exits 2 with "Missing
option '--reform'" before doing anything. Confirmed by running them. core.py's
own error text has carried the correct two-step form all along; the page was
the only place that dropped it.
The site's name for a model was rejected by the site's own API. Every page
says psl-og; get_status("psl-og") raised, and the error listed only registry
keys, so nothing connected the two. site_id was added to render one name per
model but the alias was one-directional. Lookups now resolve site names, the
error offers both vocabularies, and the returned model_id is the registry key
so a caller who passed a site name learns the contract id. Pinned by a test.
/connect claimed "no PolicyEngine account and no API key" beside a UK CLI
example that needs a HUGGING_FACE_TOKEN for a private dataset. The claim is
true of the hosted route and false of that example. Three other pages
document the token; the page carrying the example did not. Both now scoped
and stated.
It also said the hosted server "points you to the local CLI" for
score_reform(model='og'). dynamic_reform_impact does return the local recipe;
model='og' returns an import error pointing at an install the registry
documents as unusable in-process. Reworded to what actually happens.
FRB/US: the headline was the weakest evidence in the repo, presented as the
strongest. A review proved the tracking invariant is tautological by running
it on randomly scrambled data — every accounting identity destroyed — where
it still passes at 6.7e-9, because init_trac defines add-factors as minus the
residuals at the input, so re-solving is the identity on any input. 282 of
284 equations have bit-exactly zero residual before Newton runs. models/ now
leads with the real evidence, agreement with the Fed's own pyfrbus across all
284 variables at ~1e-8 — the Board's two releases differ from each other by
1.3e-8 — and the residual chart's tautological bar is relabelled "identity —
not evidence" with the scrambled-data result in its note, instead of being
plotted as the shortest bar beside genuine cross-implementation residuals.
And one of mine: replacing "Seven model classes" left "Seven models, side by
side, side by side." in an h2. The earlier attempt at the same line had also
silently failed on a literal apostrophe rather than the entity.
Twelve gates, 1107 site tests, 275 integration tests.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…the one described The FRB/US "fixed funds rate" fiscal experiment was not pegged. A review found the year-two multiplier of 0.99 could not be reproduced by running the model, which returns 0.90, and the cause is not a definitional difference or a stale artifact: results.json regenerates bit-identically, so the artifact faithfully encodes a bad experiment. run_experiments.py flipped dmpintay to 0 and dmpex to 1 after init_trac, and only on the shocked frame, then took deviations against the inertial-Taylor baseline. Tracking had already absorbed the wedge between the rule's own value and the actual funds rate into the rff add-factor, so it kept being subtracted: the "pegged" rate settled 0.07-0.12pp below its own rfffix target. That is an unintended monetary easing stacked on the fiscal stimulus, and it inflated the multiplier. The tell needed no external comparison — a pegged run whose funds rate moves is self-evidently not pegged. It now takes its own pegged baseline, tracked under the peg, and asserts max |drift| < 1e-10, so a peg that is not a peg fails loudly instead of returning a plausible number. Three independent correct implementations agree at 0.904. Regenerated, and the site corrected in five places across four pages: the year-two pegged multiplier is 0.90, and it sits below the 1.1-1.2 cross-model accommodation range rather than just below it. Year one under the inertial Taylor rule is unaffected at 0.72 — no switches were flipped post-tracking there, so only the pegged run was wrong. The script also hardcoded one developer's home directory, so nobody else could regenerate these figures — which is how a bad experiment stayed committed. It now resolves from US_FRB_MODEL or a default checkout and fails with a clear message rather than a stack trace. Twelve gates, 1107 tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two classes of defect found by the site and model-repo reviews. **A dead button on the catalog page.** `models/index.html#score` is the target of fifteen "how to score a reform" links from across the site, and it renders a copy button. The handler had been pasted by hand into fourteen pages plus a variant inside /connect's own IIFE, and was missing here — the button looked live and did nothing. Lifted the one handler into `reveal.js`, where it is delegated from the document, and removed all fifteen copies. Two tests keep it honest: a page that renders the button must load the file, and no page may rebind it (two listeners on one button is two clipboard writes per click). **A test count nobody can keep true.** Four surfaces published the size of the us-hank suite as a fact about another repo. It was "18" while that repo had 24 test functions; an earlier pass here changed it to "47", which was also wrong — the true committed figure is 24, and in-flight work takes it to 43. A number that lives in this repo and describes a different one drifts on every commit over there, and the page said "an 47-test replication gate", which is what a mechanical replace looks like. Removed the count from all four surfaces and from the assertion pinning it; what matters — that steady-state targets, market clearing and shock responses are gated — is unchanged and still stated. Also from this round: the us-hank status on `forecasts/us` said "research prototype" (that is psl-og's registry status), `frb-us` — the id the site prints — fell through to the bare enum error instead of its written refusal, the HANK productivity shock documented 0.01 as "~1%" when steady-state Z = 0.468 makes it ~2.1%, and the six /economy topic pages carried a breadcrumb through /economy/topics, a redirect source rather than a page. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… quoting one of 64 p-values as a result Fresh reviews of the model repositories found the site publishing four things it cannot support. All four model repos are re-pinned to the merged fixes. **The FEVD replication was measuring the wrong quantity.** The paper's Figure 4 "decomposes the mean of the sum of the decomposition", with the note attributing the gap to 100% to the unidentified shocks, and its text says the identified shocks explain only "around 80%". So its ~40% / ~50% are posterior means of shares of *total* variance. This site summed per-shock posterior *medians* and renormalised them to 100% — and those medians sum to about 0.62 for GDP and 0.71 for CPI, so the renormalisation inflated every identified share by roughly a third. It also read the decomposition one column late, since column h holds the (h+1)-step variance. On the paper's own definition: UK GDP 37.4% against ~40%, which is a match, and UK CPI **42.3% against ~50%, about 8pp short**. The 49.5% that matched was an artefact of the renormalisation. Both are published, with the 68% band (roughly ±14pp) that makes the shortfall unresolvable at this sample size. **"Bank Rate is the only defensible forecasting claim" rested on the smallest of 64 p-values.** The evaluation runs 64 variable×horizon Diebold–Mariano tests per benchmark and had always noted it needed a multiplicity adjustment without supplying one. Adjusted together (Benjamini–Hochberg), the minimum q against a drifting random walk is 0.36. **Nothing clears a 10% false-discovery rate, on any variable at any horizon.** Also corrected: the ex-Covid 0.77 and the p = 0.38–0.67 range were quoted inside a drift-benchmark paragraph but belong to the weaker no-change benchmark; against drift they are 1.21 and 0.33–0.43. **The forecast evaluation scored a model this site does not publish.** Every fan here is estimated with six Covid dummies; the rolling evaluation had always left them out, so 2020Q2 sat in the likelihood as an ordinary observation. Under the published specification UK GDP against drift goes 1.06 → 0.99 at h=1 and 1.12 → 0.95 at h=8 — level with naive rather than worse, and still not better. CPI barely moves, so that weakness is not a Covid artefact. **The published fan bands were calibrated on a different model.** The same no-dummy problem reached the widening factors in `band_calibration.json`: measured without dummies, applied to fans that have them, in the same direction at every horizon. UK GDP's 68% factor at one quarter should be 0.86, not 0.64 — the fan was being narrowed by 36% where the model actually shown wants 14%. Regenerated from the production-spec coverage run, with the ±6.7pp standard error recorded so four decimals are not read as four decimals of precision. On DEFINE: the multiplier of 1.78 was set beside IMF and OBR multipliers as if comparable. It is a cumulative ratio to 2040 against short-horizon benchmarks, its denominator is endogenous and moves the wrong way (social benefits fall when unemployment does, biasing the ratio up), and a demand-led closure with Kaldor–Verdoorn productivity implies above 1 over fifteen years anyway. Now stated as upper-leaning, with the unidentifiable "Batini et al. 2021" citation dropped rather than left as a bare name. Two pages said "thirty-five manual defects are pinned in tests"; thirteen are. The Table 4 growth gate is ±0.31pp against an observed −0.30pp miss, which is a pass by 0.01pp and now says so. And `make_comparison.py` pointed at a developer's home directory, so the figure could not be regenerated by anyone else — which is how a superseded statistic stayed in the committed artifact. Merged upstream and re-pinned: boe-var-model#13, us-hank-model#3, us-frb-model#8, define-uk-model#12. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…hecks that cannot fail forecasts/score.py --check caught the fan charts left stale by the regenerated band_calibration.json — which is what that gate is for. The GDP fan is visibly wider at short horizons now: the applied 68% factor at one quarter goes 0.64 to 0.86. Also on svar/validation: the forecast-revision adding-up identity sat as a row in a table captioned 'this replication against Brignone & Piffer, check by check', where a check that holds by construction reads as a fifth agreement with the Bank. And the reading guide called the gated sign and zero restrictions 'real evidence' — they are imposed by the identification scheme and cannot break, so that gate protects the software, not a claim about the economy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ying one silently CI caught svar_summary() serving an empty FEVD table: _parse_md_table returns [] when it cannot find its heading, the summary is assembled without checking, and the upstream heading gained '(4-quarter-ahead forecast error)' when the off-by-one was fixed. Matching a substring rather than most of the heading, plus a test that asserts the table is non-empty, so the next rename fails loudly instead of publishing nothing. The summary now carries both FEVD tables and says which is which. The renormalised sum-of-medians is kept for continuity; the per-draw group share with its posterior mean and bands is the statistic the paper's ~40% / ~50% refer to, and a caller comparing to the paper needs that one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fresh independent reviews of the model repositories (boe-svar, frb-us, us-hank, define-uk) and of the site's validation pages and examples. The model-side fixes are merged upstream and re-pinned here.
Headline corrections:
reveal.js.Upstream: PolicyEngine/boe-var-model#13, PolicyEngine/us-hank-model#3, PolicyEngine/us-frb-model#8, PolicyEngine/define-uk-model#12 and #13.
🤖 Generated with Claude Code