Calibration loss under-penalises overshoots and silently drops targets

[vahid-ahmadi]

Two defects in the loss construction, found while investigating why several targets sit 1.5-3x above their values (#452, #454, PolicyEngine/policyengine-uk#1818).

First, a negative result worth recording: the loss is relative, not absolute. sre is scale-free, so the natural hypothesis — that a GBP 321m target gets swamped by GBP 89bn ones — is false. The missed targets are not de-weighted. dwp/benefit_cap_total_reduction (3.1x), dwp/benefit_capped_households (2.0x) and housing/rent_social (1.55x) together carry roughly 30% of the entire national half of the loss. The optimiser has every incentive to fix them and cannot.

Defect 1: torch.min makes the loss anti-convergent beyond 2x

utils/calibrate.py:326-329:

def sre(x, y):
    one_way = ((1 + x) / (1 + y) - 1) ** 2
    other_way = ((1 + y) / (1 + x) - 1) ** 2
    return torch.min(one_way, other_way)

Taking the min measures error against the larger of (model, target), so an overshoot of factor k is penalised as if it were a shortfall of 1/k. The loss is bounded by 1, and the gradient magnitude x·dL/dx peaks at ratio 2.0 and declines thereafter:

  ratio     loss   x*dL/dx
   1.04   0.0015    0.0726
   1.55   0.1243    0.4536
   2.00   0.2475    0.4975   <- peak
   3.10   0.4560    0.4372
   5.00   0.6374    0.3213
  10.00   0.8084    0.1813

Past 2x, the further a target overshoots the weaker the pull back. Every target in the three linked issues is an overshoot; housing/rent_private, the only one near 1.0, is the only one that converges.

Suggested fix — a symmetric, unbounded log-ratio loss:

def sre(x, y):
    return torch.log((1 + x) / (1 + y)) ** 2

Near the target it is indistinguishable (gradient 0.078 vs 0.074 at 1.04x) but at 3.1x it gives 2.26 against 0.437 — about 5x the pull — and it increases monotonically.

A demonstrable stalemate this creates

housing/rent_social sits at 1.55x. To hit it, the optimiser must cut social-renter weight by ~35%, which drives ons/tenure_england_social_rent to 0.645x. But:

sre(1.55x) = 0.124301
sre(0.645x) = 0.123542

The loss is essentially indifferent between the two outcomes. Which one wins is decided by unrelated targets. That is why the target sits stubbornly at 1.55x despite carrying ~5% of the national loss. A log-ratio loss does not remove the conflict, but it does stop the two errors being interchangeable.

Defect 2: targets are dropped silently when the year does not resolve

targets/build_loss_matrix.py:140 returns None when the closest available year is later than the calibration year, and :112 does if val is None: continue with no log line — only the exception path at :120-121 warns.

Consequence today: all 15 dwp/uc/two_child_limit/* targets are silently dropped, because they carry values={2026: ...} while calibration_year=2025 (datasets/frs_release.py:57). That includes households_affected (453,600) and children_affected. These are the only 15 dropped targets of 651 candidates (636 columns are built).

This is also a latent landmine: had calibration_year been 2024, every values={2025: ...} target — including both benefit cap targets and all three housing targets — would have vanished with no signal.

Suggested fix: log at WARNING whenever _resolve_value returns None, and add a build-time assertion that named critical targets are present in the matrix.

Minor, found alongside

• Weights ship from epoch 510 of 512. calibrate.py:476,516 save and log only when epoch % 10 == 0, so the last two steps are discarded. Either use epochs=511 or save after the loop.
• A 10% inconsistency between the two halves of the loss. datasets/local_areas/constituencies/loss.py:97 and local_authorities/loss.py:110 scale local age targets by * 0.9 while the national half targets the full ons/uk_population of 69,502,341. There is no comment justifying the 0.9 (introduced in 35c30e2 "Fix bug"). The two halves pull against each other by 10%.

Note for whoever picks this up

A complete per-target final error table already exists and needs no rebuild: get_performance (datasets/local_areas/constituencies/calibrate.py:34-110) writes constituency_calibration_log.csv every 10 epochs with per-target estimate/target/error, national rows tagged name="UK", and CI uploads it as an artifact on every push (.github/workflows/push.yaml:63-73). Filtering that to the final epoch answers most open questions across #452, #454 and PolicyEngine/policyengine-uk#1818 directly.