From 8835e16cbfea2e7d02b8c8c1a2b3c4eb6cf74e52 Mon Sep 17 00:00:00 2001 From: vahid-ahmadi Date: Wed, 12 Aug 2026 15:09:46 +0100 Subject: [PATCH 1/6] Fix broken examples, a rejected identifier, and an overstated headline MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three model-repo reviews and two recovered page audits. The findings that were verifiable and mine to fix. Two published commands could not run. /connect's dynamic-scoring recipe gave `pe-macro og-score --json > og.json` and `pe-macro dynamic-score --og-payload og.json`; --reform is required=True on both, so each exits 2 with "Missing option '--reform'" before doing anything. Confirmed by running them. core.py's own error text has carried the correct two-step form all along; the page was the only place that dropped it. The site's name for a model was rejected by the site's own API. Every page says psl-og; get_status("psl-og") raised, and the error listed only registry keys, so nothing connected the two. site_id was added to render one name per model but the alias was one-directional. Lookups now resolve site names, the error offers both vocabularies, and the returned model_id is the registry key so a caller who passed a site name learns the contract id. Pinned by a test. /connect claimed "no PolicyEngine account and no API key" beside a UK CLI example that needs a HUGGING_FACE_TOKEN for a private dataset. The claim is true of the hosted route and false of that example. Three other pages document the token; the page carrying the example did not. Both now scoped and stated. It also said the hosted server "points you to the local CLI" for score_reform(model='og'). dynamic_reform_impact does return the local recipe; model='og' returns an import error pointing at an install the registry documents as unusable in-process. Reworded to what actually happens. FRB/US: the headline was the weakest evidence in the repo, presented as the strongest. A review proved the tracking invariant is tautological by running it on randomly scrambled data — every accounting identity destroyed — where it still passes at 6.7e-9, because init_trac defines add-factors as minus the residuals at the input, so re-solving is the identity on any input. 282 of 284 equations have bit-exactly zero residual before Newton runs. models/ now leads with the real evidence, agreement with the Fed's own pyfrbus across all 284 variables at ~1e-8 — the Board's two releases differ from each other by 1.3e-8 — and the residual chart's tautological bar is relabelled "identity — not evidence" with the scrambled-data result in its note, instead of being plotted as the shortest bar beside genuine cross-implementation residuals. And one of mine: replacing "Seven model classes" left "Seven models, side by side, side by side." in an h2. The earlier attempt at the same line had also silently failed on a literal apostrophe rather than the entity. Twelve gates, 1107 site tests, 275 integration tests. Co-Authored-By: Claude Opus 5 (1M context) --- connect/index.html | 17 ++- frb-us/validation/index.html | 31 +++-- .../src/policyengine_macro/capabilities.py | 24 +++- integration/tests/test_capabilities.py | 24 ++++ models/index.html | 16 +-- obr/index.html | 6 +- obr/validation/index.html | 6 +- pe/validation/index.html | 15 ++- reports/define-uk-replication/index.html | 4 +- svar/validation/index.html | 2 +- us-hank/validation/index.html | 2 +- validation/figures/README.md | 47 ++++--- validation/figures/chart_data.json | 123 ++++++++++++++---- validation/figures/make_charts.py | 2 - 14 files changed, 241 insertions(+), 78 deletions(-) diff --git a/connect/index.html b/connect/index.html index 2c3a88ef..35eb427d 100644 --- a/connect/index.html +++ b/connect/index.html @@ -285,7 +285,9 @@

Three routes to the same models. Start with MCP below — it needs no install, no PolicyEngine account and no API key, though some clients - gate custom connectors by plan. Use the CLI or Python when you want the + gate custom connectors by plan. That is true of the hosted route; a + local UK population or reform-scoring run additionally needs a + HuggingFace token, as below. Use the CLI or Python when you want the models in a script, or need one of the local-only members.

@@ -368,10 +370,19 @@

Connect in three steps.

takes around 20 seconds while the tax-benefit models load; a UK forecast runs a couple of minutes on first call. Other tools respond faster.

+

uk microdata + Running a UK population or reform score locally needs + HUGGING_FACE_TOKEN from an account with access to the + private policyengine/policyengine-uk-data-private + dataset. UK household calls, every US call, and every hosted call + need nothing.

local only OG-UK long-run scoring (score_reform with model='og') and dynamic_reform_impact take - tens of minutes — the hosted server points you to the local CLI. + tens of minutes and are excluded from the hosted image; + dynamic_reform_impact returns the two-step local recipe, + while model='og' returns an import error — use the local + CLI for both. The three define_* tools likewise return run instructions rather than results when called on the hosted server — the unlicensed upstream is never hosted. Everything else here runs hosted.

@@ -443,7 +454,7 @@

Use Python directly.