UK source packages, wave 3: local geography

[juaristi22]

Wave 3 of the UK migration: source packages for the local-geography surface.
Complete. Draft — the profile-correction PR follows separately.

Part of #134.

What is here

Twenty-four packages, 95,571 facts, all at the publisher's own geography.

batch	family	packages	facts
1	Population by age	ons-pcon24-population-by-age-2024, nrs-pcon24-population-by-age-2024, nisra-pcon24-population-by-age-2024, ons-lad-population-by-age-2024	23,749
2	SPI income by area	hmrc-spi-income-by-area-2023-24	19,816
3	Universal Credit by area	dwp-uc-households-by-constituency-may-2025, dwp-uc-households-by-local-authority-may-2025, dwp-uc-households-by-constituency-children-may-2025	6,038
4	Small-area income	ons-small-area-income-msoa-fye2023	29,056
4	Tenure (households)	ons-census2021-ts054-tenure-lad, nrs-census2022-uv404-tenure-council-area, nisra-census2021-tenure-lgd	3,271
4	Dwelling stock by tenure	ons-subnational-dwellings-by-tenure-2024	4,425
4	Private rents	ons-pipr-rents-by-area-june-2026	348
5	Household counts	ons-census2021-ts041-households-pcon24, ons-census2021-ts041-households-lad, nrs-census2022-households-ukpc24, nisra-census2021-households-pcon24, nisra-census2021-households-lgd	979
5	LA council tax stock	voa-council-tax-stock-by-lad-2025	2,881
6	Council tax levels and taxbase	mhclg-council-tax-levels-england-2026-27, scotgov-band-d-equivalents-2025, scotgov-band-d-council-tax-rates-2026-27, welshgov-council-tax-levels-2026-27	5,008

The constituency families land on the same geography: 650 Westminster seats on
2024 boundaries. The age family needs three publishers to cover them (ONS via
Nomis for the 575 England and Wales seats, NRS for 57, NISRA for 18); the
local-authority level needs one, because Nomis NM_31_1 spans the whole UK
including the 11 Northern Ireland districts.

The batch-4 families are local-authority and below, and each lands at whatever
grain its publisher actually releases — 361 UK local authorities for tenure
across three census legs, 7,264 MSOAs for small-area income, 295 English
districts for the dwelling series, and 348 areas for rents. Nothing is
re-aggregated to make the grains match; that is downstream build work.

Row-count mechanics, as #134 asked

The compact shape for a whole table, confirmed on a 650-row one and now written
into docs/agent-source-package-harness.md:

• one record set per publisher category, not per geography;
• rows carry geometry only — row_number, a row-header guard on the code
  column, the per-row geography overrides, and a guard proving the row belongs
  to the category its record set claims;
• the category's filters and constraints go in shared_filters /
  shared_constraints, declared once and merged into every fact.

That is ~11–16 YAML lines per fact against ~26 for the per-geography
orientation, which is what produced the repo's two 200k-line packages. Where the
layout is ours to choose (Nomis, Stat-Xplore, NISRA PxStat) the artifact is
long format; where the publisher fixes it (the SPI ODS, the NRS workbook) we
parse their grid as published and never re-shape it.

Ten-year bands are a declared sum, never arithmetic here

The profile declares eight ten-year age bands. ONS publishes quinary groupings
and nothing coarser, so banding at download stops at five years. The ten-year
step is carried by a publisher-neutral record_set_spec_id
(uk.local_geography.population.age_0_10.v1 and siblings) that the ONS, NRS and
NISRA packages share: the profile's sum-only operation then adds exactly the
publisher facts that make up the band. The grain differs by publisher — quinary
in England and Wales, single year in Scotland, five-year bands in Northern
Ireland — and the spec-id grouping absorbs that without any package restating
another's classification.

Tenure is UK-wide, and the household and dwelling series stay apart

All three census legs are ported, so household tenure covers the UK: TS054 for
318 England and Wales districts, NRS UV404 for 32 Scottish councils, NISRA
HH_TENURE for 11 Northern Ireland districts at its full eleven-category
classification rather than the four-category roll-up the profile could not use.

The legs classify differently — Scotland has one social-rented category where
England and Wales has two, Northern Ireland splits private renting five ways,
and Scotland's "Shared Equity (LIFT or Help-to-Buy)" has no counterpart at all.
Concepts that genuinely match share a publisher-neutral record_set_spec_id
bucket so one target sums its bucket across legs; unmatched categories keep
their own bucket rather than being forced into one.

SPREE is the same tenure language over a different entity, so it carries
entity: dwelling and its own uk.local_geography.dwellings.* spec ids. A
target pinned to entity household can never tie-break against it — the
separation requirement from the plan, enforced structurally rather than by
convention.

The Scottish census table does not add up, by design

NRS applies statistical disclosure control cell by cell. In 26 of 32 councils
the eight detail categories differ from the published all-households total, by
−6 to +10 households (median 3); the publisher's own roll-ups differ from their
own details in 17 councils. Scotland-wide the gap is 8 households in 2,509,275.

Every value is carried exactly as published and nothing is reconciled toward
anything else. The England and Wales leg, by contrast, adds up in all 318
districts. Each package header states which behaviour its leg has, because a
consumer that assumes additivity across the UK would be wrong in Scotland.

Household counts are UK-wide at both levels

575 + 57 + 18 = 650 constituencies; 318 + 11 in batch 5 plus the 32 Scottish
councils batch 4 already carries = 361 local authorities. Three census legs share
one publisher-neutral record_set_spec_id per level, so a single profile target
spans them despite three different census days — England and Wales 2021-03-21,
Scotland 2022-03-20, Northern Ireland 2021-03-21 — each fact keyed to its own
publisher's period.

Scotland's local-authority household count is deliberately not added again:
it is already the all-occupied-households row of batch 4's UV404 council-area
package, and a second package would collide on fact_key. The same reasoning
keeps CTSOP1.1's region and country rows out of the council tax package, where
wave 2's voa-council-tax-bands-2025 already ports that concept from CTSOP2.0.

Every publisher input uk-data reads is here

Batch 6 exists because a coverage audit against
policyengine-uk-data/targets/sources/la_council_tax.py found three publisher
values wave 3 had missed: the average Band D per authority, the council taxbase,
and the Welsh council tax income. Facts-only means those have to be in Chronicle
even when the target a consumer materialises from them is computed later —
otherwise the computation cannot be rebuilt from facts.

They are ported now, for all three jurisdictions. England's net council tax stays
excluded (computed) because it is taxbase × Band D, but both inputs are facts.
Wales publishes its council tax income directly, so that is a fact rather than a
derivation.

Three publisher quirks, each of which would have been a silent drop:

• Footnote markers welded onto year headers. ScotGov writes '2024-251' and
  '2022a'. A strict year regex rejects them; the first pass lost 99 facts that
  way. The parser now takes the leading year for the period and keeps the raw
  label as the column guard.
• Scottish council names do not match ONS. Four use & for and, and
  Shetland Islands carries a double space. An alias map does the code lookup;
  the row guards carry the publisher's exact string.
• The after-CTR taxbase series starts in 2013, not 1996, because council tax
  reduction replaced council tax benefit that April. 13 years against 30 is
  publisher behaviour, not missing data.

Findings that are the profile's decision, not a sourcing gap

• The SPI tables publish no income-amount column. Count, mean and median per
  income type, and an amount only for total tax. hmrc.employment_income.amount
  and hmrc.self_employment_income.amount have no publisher fact;
  policyengine-uk-data forms them as count × mean, excluded (computed).
• All eight age targets share one selector ({ons, population}), so as the
  profile stands each would sum the same fact set and every band would resolve
  to total population. They need to select the spec ids.
• ONS publishes no equivalised-housing-costs measure, so
  ons.equiv_housing_costs has no publisher fact either; uk-data forms one as
  BHC minus AHC, excluded (computed).
• Small-area income exists at MSOA only — there is no local-authority cut —
  so the three ons_income targets move to geography_level: msoa.

All of these are for the profile-correction PR, together with the households
target batch 5 will back and the entity pin the tenure targets need.

Enumerated exclusions

• Northern Ireland UC — DWP's UC statistics are Great Britain; DfC publishes
  Northern Ireland separately.
• Isles of Scilly SPI — HMRC suppresses every column of Table 3.14 for it,
  so it carries no facts. [Not available] is a statement that nothing was
  published; rows are grouped by which measures they carry, so a suppressed cell
  shows as a missing fact rather than a zero.
• Stat-Xplore's "Unknown" area row (7,054 households) — a real publisher
  count with no GSS code, recorded rather than forced into a geography.
• ASHE constituency earnings — feed an interpolated band distribution
  loss.py never reads; not a target input.
• Publisher roll-up categories that are derivable sums of ported details
  (Scotland's "Owned: Total" and "Private rented: Total"; Nomis's Owned / Social
  rented / Private rented aggregates) — porting both layers would double-count
  inside the tenure buckets. Verified to be sums of ported details, so the
  exclusion loses nothing.
• Northern Ireland private rents — the publisher prints [x] at June 2026
  because the NI series lags; the December-2025 NI level is already ported in the
  wave-2 bulletin package. City of London and Isles of Scilly are absent from the
  publisher's rent table entirely.
• Confidence-limit, percentage and percentile restatements — publisher
  re-presentations of ported estimates that no declared target needs.
• Devolved constituency rent anchors — hardcoded in uk-data with no
  citation; blocked until a publisher table surfaces.
• uk-data's national consistency adjustments, boundary mapping matrix, ×0.9
  age scaling, and count × mean amounts — computed, populace-side.

Lane changes

• xlsx_table_full_rows, a single-sheet full-row lane. The PIPR monthly
  workbook is one ~49,000-row sheet whose used-range cell parse is ~2M cell
  records per build — the operational blocker recorded on UK wave 2: housing base facts — PIPR bulletin rents and EHS weekly housing costs (#133) #159. The whole sheet
  is preserved as source rows and selected_rows restrict cell emission, so the
  348 rent facts cost 13,960 cells with every row still queryable.
• header_row on both full-row lanes, for export tools that write a
  metadata preamble above the table (NRS's SuperWEB2 export puts ten lines
  there). Authors beware: cell emission renumbers preserved rows virtually, so
  a package whose artifact has a preamble must cite the virtual row, not the
  physical line. They coincide only when the header is line 1 — the first UV404
  attempt cited physical lines and a tenure guard caught it, ten rows off.
• ALLOWED_GEOGRAPHY_LEVELS: parliamentary_constituency → constituency. The
  old name was unused; the profile and populace's runtime both say
  constituency. Renamed rather than adding a synonym. msoa added for the
  small-area income grain.
• _source_row_age_range reads three more publisher wordings — Nomis's
  "Aged 85+" and "Aged under 1 year", and NISRA's "Five year age bands" column —
  so those bands evidence the same numeric bounds every other band carries.
• source_cells_from_xlsx takes an optional sheets allowlist. The NRS
  workbook is 963k cells across sixteen geography worksheets of which this
  package reads one; naming it keeps the per-build cost proportionate. Tested
  both ways, including that a renamed sheet fails loudly.

Verification

Per package: validate-package, then publish-raw (facts need raw_r2_uri),
then build-suite — all green, lineage coverage 1.0, zero agent-acceptance
errors.

Every fact was re-derived independently against its artifact: fact count equals
published-cell count, nothing dropped, nothing invented, every value matching
after publisher scaling. Additivity anchors hold where the publisher gives one —
the age bands sum to each area's published total, and NISRA's own Northern
Ireland total (1,927,855) equals the sum of its 18 seats to the person.

Two cross-checks recorded rather than corrected:

• The two UC cubes disagree slightly by design: 571 of 632 constituencies differ
  between the totals cube and the sum of the children cube, median 4 households
  and at most 15 (GB 6,577,205 against 6,576,846). DWP adjusts cells at source.
• Chronicle's local-authority set is 361 to uk-data's 360; the extra is
  N09000011 Ards and North Down, which uk-data's committed CSV drops.

Scotland's census export carries council names only, so each row's S12 code is
authored from the publisher name — cross-checked 32/32 against the batch-1
local-authority age package, which carries both — and guarded by that exact name
as the row header.

The bundle expectation surface was regenerated whole from one measured build.

The checklist is now one table

docs/pe-uk-source-checklist.md carried wave-3 in its own section while the
families were being ported. They are ported, so there is one inventory for every
uk-data calibration-target family, national and local — forty rows, every one
with an explicit status, and nothing in uk-data absent from it.

Two rows came off blocked, because wave 3 made the blocking false: UK-wide
tenure counts and the devolved constituency rent anchors both now have every
publisher input ported, so they are excluded (computed) with base facts rather
than sourceless. Only two blocked rows survive, both genuinely unobtainable — the
£1,100/month mortgage payment whose citation points at a bulletin carrying no
mortgage figure, and the SPP Review PDF that link-rotted with no Wayback copy.

The status vocabulary now separates three things the old wording blurred: a
computation whose publisher inputs are all ported, an artifact that is pinned and
parsed but does not print the quantity uk-data uses (the SPI amounts and the ONS
housing costs, previously mislabelled "blocked"), and a source that cannot be
obtained at all.

Next

The profile-correction PR, which now has a settled list: the eight age selectors,
the three targets with no publisher fact, the ons_income move to MSOA, the
entity pin on tenure, and a new households target these five packages back.

[juaristi22]

Adversarial review (Codex): approve, no material findings

Ran an adversarial Codex review over the branch diff against main — framed to
challenge the approach and assumptions, not just hunt for defects.

Verdict: approve. No defensible ship-blocking issue in the branch diff.

What it exercised

The approval is worth what it checked, so for the record:

• All 24 new package aliases validated structurally; alias-map drift clean.
• The lane changes read in full — chronicle/source_package.py,
  sources/rows.py, sources/cells.py, core.py.
• Representative end-to-end builds through both new parser paths, resolving
  facts and lineage. It confirmed the PIPR workbook parses 49,266 rows and emits
  its facts with lineage intact — the path that exists because a used-range parse
  of that sheet is ~2M cell records per build.
• The specific things that could have gone wrong: the
  parliamentary_constituency → constituency vocabulary rename (searched for
  surviving references), virtual row numbering in compact cell emission, and the
  Welsh package's fy2025 record-set identifiers.

Limits of this pass

• Full pytest did not run in its read-only sandbox — Python, openpyxl and
  pytest all need a writable temp directory. It substituted direct parser and
  package checks. The full suite was run locally instead: 635 passed, 1
  skipped, including the bundle test against the re-measured expectation
  surface.
• It did not reach the profile-resolution question. An earlier adversarial
  pass over this same branch returned needs-attention on two findings, and
  they still stand — this pass simply did not cover that ground, so treat them as
  open rather than retracted:
  1. uk_local_geography still selects self_employment_income_amount and
     employment_income_amount, which have no publisher fact — the SPI tables
     publish count, mean and median per income type and an amount only for total
     tax.
  2. The three ons_income targets are still scoped to local_authority while
     the ported facts are at msoa, because MSOA is the only grain ONS releases.

Both are the profile-correction PR's work: #134's plan puts profile changes in the
ledger-target-profile-author lane, landing after the packages their selectors
point at. This PR is the source-porting half. The profile PR's list is settled —
the eight age-band selectors, the three targets with no publisher fact
(hmrc.*_income.amount ×2 and ons.equiv_housing_costs), the ons_income grain
move, the entity pin on the tenure targets, and a new households target these
packages now back.

[vahid-ahmadi]

Now that all five batches are in, this reads well. The reviewable surface is ~540 lines against a million lines of generated packages, which is the right ratio for a wave PR, and the new lane is justified by a measured problem rather than a preference — a ~49k-row sheet whose used-range parse would cost millions of cell records per build is exactly the case that needs a full-row lane.

Two things the new lane gets right that its predecessors didn't: source_rows_from_xlsx_table raises on a missing declared sheet rather than returning [], and read_only=True, data_only=True is the correct pairing for the memory problem it exists to solve. Flattening datetimes to ISO text with the midnight rule for date-only is a good call — guards and criteria comparing against a stable string form is worth more than preserving the type.

Three things.

1. header_row isn't validated, and header_row=0 silently produces empty rows.

Both lanes do:

if row_number < header_row:
    continue
if row_number == header_row:
    header = [...]
    continue
values = {... for index, column in enumerate(header or ())}

with enumerate(..., start=1). With header_row: 0 — a plausible slip from zero-indexed thinking, and the kind of thing an authoring agent does — no line is ever < 0 or == 0, so every physical line falls through to the values branch with header still None. header or () then yields an empty column set, and the package emits one value-less SourceRow per line. No error, no zero-row signal, just a table of empty rows.

The other end is quieter still: header_row past the end of the sheet yields zero rows and no complaint, which is the return [] failure mode from the Stat-Xplore lane in a new place.

A if header_row < 1: raise ValueError(...) plus a refusal when the header line is never reached would close both, and the second is the one that matters — a package silently contributing nothing is invisible until someone diffs fact counts.

2. The workbook is never closed.

openpyxl.load_workbook(..., read_only=True) documents that read-only mode leaves resources open until close() is called — it's the one mode where openpyxl says so explicitly. This lane exists precisely because of memory pressure, so leaking the handle in the function whose purpose is to not hold the sheet in memory is worth fixing:

workbook = openpyxl.load_workbook(BytesIO(content), read_only=True, data_only=True)
try:
    ...
finally:
    workbook.close()

Worth doing even though BytesIO softens the consequence, because the next caller may not pass one.

3. The parser dispatch chain is now 26 branches, and this is the third PR to grow it.

I raised this on #141 when it was around ten; chronicle/source_package.py on main now carries 26 if self.parser == ... branches across build_source_rows and build_source_cells, and this PR adds more. Each new lane means a near-verbatim stanza in both methods, so "which parsers exist and what does each do to sheet_name" is now a question nobody can answer without reading both chains end to end.

A {parser_name: (rows_fn, cells_fn, sheet_default)} table would collapse both, make the set enumerable, and remove the class of bug where a lane is registered in one method and not the other. Not for this PR — but the cost is now compounding at roughly one lane per wave, and wave 3 isn't the last wave.

Smaller

• The source_rows_from_delimited_text restructure preserves the empty-file behaviour (returns []) and keeps row numbers physical, which is the property the docstring promises. Good that the preamble-skipping is explained by naming SuperWEB2 rather than left as a magic parameter.
• Three publishers for the constituency age family (ONS/NRS/NISRA covering 575/57/18 of the 650) with one publisher at LA level because Nomis NM_31_1 spans the whole UK — worth the sentence it got in the body; that asymmetry would otherwise look like an inconsistency.
• The record-set-per-category shape with rows carrying geometry only, and the category's filters in shared_filters/shared_constraints, is the right compact for a 650-row table. Documenting it in the harness doc rather than leaving it as precedent in one package is what makes it reusable for wave 4.

[juaristi22]

Thanks — both fixes are in b064989.

1. header_row validation. Both lanes now refuse header_row < 1 up front,
and refuse when the content ends before the declared header line — the message
names the last physical line/row reached and the header row it never got to, so
the mis-set package fails at build rather than contributing nothing.

One deliberate asymmetry: the delimited lane keeps its documented
empty-content → [] behaviour (it predates this PR, and a test now pins it),
so its refusal fires only when lines exist. The new xlsx lane refuses even on
an empty sheet — it already refuses a missing sheet, and a declared sheet with
no rows is the same class of authoring error.

Five tests pin the contract: header_row=0 and past-the-end refusals in both
lanes, plus the preserved empty-content promise.

2. Workbook close. try/finally around everything after the load, with a
comment stating why it's there — read-only mode holds the source open until
close(), and this lane exists for memory pressure.

3. Dispatch table. Agreed, and agreed it's follow-up scope — filed as #169
so it doesn't get lost. Wave 4 is the natural place to do it, before the chain
grows again.

[vahid-ahmadi]

b064989 closes both. Verified:

• Both lanes refuse header_row < 1 up front, and both refuse when content ends before the declared header line, with the message naming the last line reached and the header it never got to. That turns the silent-empty-table and silent-zero-rows cases into build failures, which is what I was after.
• try/finally around everything after the load, with the comment stating why read-only mode needs it.

The asymmetry is the right call and worth having written down: keeping the delimited lane's documented empty-content → [] behaviour (it predates this PR and now has a test pinning it) while the new xlsx lane refuses an empty declared sheet. A declared sheet with no rows really is the same authoring error as a missing sheet, and the new lane has no back-compat to preserve. Five tests covering both refusals in both lanes plus the preserved promise is the right coverage.

Thanks for filing #169 rather than letting the dispatch point evaporate — wave 4 before the chain grows again is exactly the right moment.

Nothing further — clear from my side.