Skip to content

Fix what five page-level audits found, including two self-contradictions - #172

Merged
vahid-ahmadi merged 1 commit into
mainfrom
fix/audit-round-two
Aug 12, 2026
Merged

Fix what five page-level audits found, including two self-contradictions#172
vahid-ahmadi merged 1 commit into
mainfrom
fix/audit-round-two

Conversation

@vahid-ahmadi

Copy link
Copy Markdown
Contributor

The five section audits that died when the machine slept were rerun, covering every page. Their top findings, fixed.

Two self-contradictions — the site arguing against itself

  • /forecasts headlined an accuracy number, the one thing its own generator says it must never do. The record band's third tile read "0.12pp — latest absolute error", level with the round and scored counts, while score.py's docstring argues "a track record that reports an impressive figure derived from one observation is worse than an empty one" — and the disclaimer saying so sat below the tile. Now shows the next scored period; the error stays on the scored card beside its benchmarks.
  • The same page called its one favourable result "beats naive" using the driftless walk that its own §06 calls too weak for a trending series. Now: "beats driftless-naive error".

A promise of something that does not exist

frb-us/index said "Run the shock explorer on this page." There is no explorer — the page contains no form, input or button, and neither does any other page on the site.

Every equation and code block on 14 pages was invisible to screen readers

.meth-inline and .scrolly-sticky are a responsive pair — exactly one visible per viewport — and both carried aria-hidden="true", so in every viewport the only visible copy was hidden from assistive technology. On pe/code and define/code, whose entire subject is the commands, not one was exposed. 58 blocks fixed; the copy buttons in the still-hidden aside are no longer focusable.

Stale and wrong numbers

  • /economy carried the only hand-typed live number on the page, and it had rotted: "the model expects it back above 3% by Q4", from the 2026-07-21 round, against a current round putting 2026Q4 at 2.94% and an inflation topic one click away saying not until 2027Q2.
  • olg/validation published debt-to-GDP as 95.1% (May 2026). The committed store says May = 94.8, June = 94.9, and the site's own economy page says 94.9. Corrected, deviation sign flipped to match.
  • models/index called all 284 FRB/US equations behavioural (the paper says 120 behavioural + 164 identities); counted seven model classes where there are six (obr-macro and frb-us share one); linked the January HMRC bulletin under June 2025 text; and attached "machine precision" to the ~1e-8 cross-validation figure, which is solver tolerance, not the 5.6e-17 tracking invariant.
  • /connect said "everything else here runs hosted" — the three define_* tools return run instructions, since the unlicensed upstream is never hosted.

Missing disclosure on two overview pages

  • svar/index cited the 49-origin rolling evaluation as a validation credential. That study is precisely what found the model no better than naive — against a random walk with drift, UK GDP is not statistically distinguishable (p = 0.38–0.67) — and the page published a GDP fan with no qualifier.
  • obr/index presented 0.15% anchored MAPE as validation. Anchored accuracy is a by-construction invariant (its own validation page says so) and free-running the same equations miss by 5.75%.

A broken primary journey

/models#score landed on a closed accordion, because the id sits on the <details> element and browsers only auto-expand when the fragment targets a descendant. Three high-traffic links point there. reveal.js now opens a targeted details element.

Two findings that were mine

  • The sticky column blanked through step 1 on five pages. My simulation of the data-step pairing skipped the active === 0 case — exactly the case that fails — so I reported "no page blanks mid-article", which was narrowly true and missed the defect.
  • The DEFINE clean-room status was stale everywhere: I merged §3.3.1 and §3.3.2 today and updated neither the site nor the registry. Four surfaces said "§3.2, 23 equations, three defects" against 118 equations and ~35. The §3.3.2 blocker — six untabulated parameters and two variables defined nowhere, so the section as published cannot be simulated forward — was published nowhere, while those same pages call the clean room the path to hosting.

One finding not acted on, because it was wrong

define/validation cites upstream issue DEFINE_UK_1.1#1 for the emissions divergence; an audit flagged it as unsupported, "the same failure class as the fabricated figures". The issue exists, is open, and its title matches the finding exactly. Verified before deleting.

Verification

Twelve generator and contract gates, 1038 site tests, 274 integration tests.

🤖 Generated with Claude Code

The five section audits killed when the machine slept were rerun. Their
findings, worst first.

/forecasts headlined an accuracy number, which is the one thing its own
generator says it must never do. The record band's third tile read "0.12pp —
latest absolute error", level with the round and scored counts, while
score.py's module docstring argues that "a track record that reports an
impressive figure derived from one observation is worse than an empty one",
and the disclaimer saying so sat below the tile. The tile now shows the next
scored period. The error still appears on the scored card, beside its
benchmarks.

The same page called its one favourable result "beats naive" using the
driftless walk that its own section 06 calls too weak for a trending series.
The footer now names the benchmark: "beats driftless-naive error".

frb-us/index promised a "shock explorer on this page". There is none, on
that page or anywhere on the site — the page contains no form, input or
button. A reader was told to do something impossible.

/economy carried the only hand-typed live number on the page, and it had
already rotted: "the model expects it back above 3% by Q4", from the
2026-07-21 round, against a current round putting 2026Q4 at 2.94% and an
inflation topic one click away saying not until 2027Q2. It now describes the
note as the dated artifact it is and points at the live view.

svar/index cited the 49-origin rolling evaluation as a validation
credential. That study is precisely what found the model no better than
naive: against a random walk with drift, UK GDP is not statistically
distinguishable (p=0.38-0.67). The page published a GDP fan with no
qualifier. Both now sit in the fact.

obr/index presented 0.15% anchored MAPE as validation. Anchored accuracy is
a by-construction invariant — the page's own validation page says so — and
free-running the same equations miss by 5.75%. Both now stated.

Every equation and code block on fourteen pages was invisible to screen
readers. .meth-inline and .scrolly-sticky are a responsive pair, exactly one
visible per viewport, and both carried aria-hidden="true" — so in every
viewport the only visible copy was hidden from assistive technology. On
pe/code and define/code, whose entire subject is the commands, not one was
exposed. 58 blocks across 14 pages now are; the copy buttons in the hidden
aside are no longer focusable.

The sticky column blanked through step 1 on five pages, and that one is
mine: my simulation of the data-step pairing skipped the active === 0 case,
which is exactly the case that fails. Panels now pin from the first
available step.

models/index called all 284 FRB/US equations behavioural; the paper says 120
behavioural and 164 identities. It counted seven model classes where there
are six — obr-macro and frb-us share one. Its HMRC link said June 2025 and
pointed at the January bulletin. And it attached "machine precision" to the
~1e-8 cross-validation figure, which is solver tolerance, not the 5.6e-17
tracking invariant.

/connect said "everything else here runs hosted"; the three define_* tools
return run instructions, as the unlicensed upstream is never hosted.

/models#score landed on a closed accordion because the id sits on the
<details> element, which browsers do not auto-expand — breaking the primary
"score a reform" journey from three high-traffic links. reveal.js now opens
a details element a fragment targets.

olg/validation published debt-to-GDP as 95.1% for May 2026. The committed
store says May is 94.8 and June 94.9, and the site's own economy page says
94.9. Corrected, with the deviation sign flipped to match.

And the DEFINE clean-room status was stale everywhere, which is my doing: I
merged §3.3.1 and §3.3.2 today and updated neither the site nor the
registry. Four surfaces said "§3.2, 23 equations, three defects" against a
reality of 118 equations and some thirty-five. The §3.3.2 blocker — six
untabulated parameters and two variables defined nowhere, so the section as
published cannot be simulated forward — was published nowhere at all, while
the same pages call the clean room the path to hosting. Now stated.

One audit finding was wrong and is not acted on: define/validation cites
upstream issue DEFINE_UK_1.1#1 for the emissions divergence, flagged as
unsupported. The issue exists, is open, and its title matches the finding
exactly. Verified before deleting anything.

Twelve gates, 1038 site tests, 274 integration tests.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vercel

vercel Bot commented Aug 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
macromod Ready Ready Preview Aug 12, 2026 1:11pm

Request Review

@vahid-ahmadi
vahid-ahmadi merged commit 954ac00 into main Aug 12, 2026
7 checks passed
@vahid-ahmadi
vahid-ahmadi deleted the fix/audit-round-two branch August 12, 2026 13:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant