Baseline integrity: does the certified world carry the law legislated into its future? (#99)

[vahid-ahmadi]

Closes #99. Branches from main — independent of the other UK PRs.

The repo has a baselines registry (#13) that describes worlds, and ingests that stamp which world a run executed. Nothing checked that the certified policyengine-uk world actually contains the measures already legislated into the years it computes at.

That gap is quiet by construction. A counterpart at 2027 or 2028 measured against a baseline missing an announced measure diverges for a reason no axis in #59's registry names — so it gets attributed to whatever axis the analyst reaches for, and the real cause never appears.

Five measures, probed against a real policyengine-uk 2.89.2

The result is mixed, which is exactly why the registry records verifications and not only failures:

verdict	measure
carried	two-child limit abolition	2 in 2025, unbounded from 2026
carried	income tax threshold freeze	flat 12,570 / 37,700 through 2028
consistent	fuel duty freeze extension	no parameter encodes it — checked another way
missing	High Value Council Tax Surcharge	April 2028, no representation
missing	property income tax rates	April 2027, no representation

The two-child result is a bonus finding: it confirms the registered pre_ab2025 world is a genuine counterfactual and not a restatement of the baseline — which #49's case battery and #67's decomposition both lean on.

The fuel duty one is the method working. No single parameter encodes "a freeze was extended", so rather than guess I checked it against OBR's own scored CPI profile from #75: negative in 2026-27 (−0.129pp), positive in 2027-28 (+0.076pp) — the signature of a freeze that ends and catches up. The certified rate path steps up in exactly that year. Consistent, evidenced, and recorded as a check rather than manufactured into a finding.

The exposure

Both missing measures are already law. Quantified from the built database:

MISSING ab2025__high_value_council_tax_surcharge: 581 UK claims at or beyond 2028
MISSING announced__property_income_tax_rates:     953 UK claims at or beyond 2027

This is an envelope to triage, not a defect count. Whether a given claim moves depends on the quantity — a council-tax surcharge does not touch a taxpayer count. The point is that the envelope is currently neither triaged nor visible. A test asserts the tool describes it that way, so nobody quotes 953 as "wrong claims".

What the validator enforces

The discipline, not the conclusion:

• A verdict without a probe is an opinion — rejected.
• consistent carries the heaviest evidence requirement, because it is the verdict that could be wishful thinking.
• missing must name which claims it affects — an unlocated gap cannot be caveated.
• --probe re-runs every reading against the engine, so a caveat that becomes removable is noticed rather than left standing indefinitely.

Verification

Suite 273 passed, ruff format --check clean, checker green with and without the engine.

Reviewers

@MaxGhenis @DTrim99 — the judgement I'd most like tested is the consistent verdict. It is the one that could let a real gap through, which is why it needs argued evidence; but if you think a measure with no encoding parameter should simply be missing until someone builds one, that is a defensible stricter line and I would take it.

[vahid-ahmadi]

Review request — @MaxGhenis @DTrim99.

This one is part of a batch; the whole queue, with a suggested merge order and what is blocked on whom, is in #104 so you can triage in one place rather than PR by PR.

[DTrim99]

Verdict: request changes. The headline claim — that this verifies the certified world contains the law legislated into its future — isn't enforced by anything that runs in CI. What CI actually executes checks the registry against itself.

Critical

• pipeline/check_baseline_integrity.py → reprobe() is the only function that queries the live policyengine-uk engine, but it's gated behind --probe, is never called by any test, and CI (pytest tests/ -q, no engine installed) never invokes it. Everything CI runs (validate + the tests) reads the registry's own recorded probe numbers and asserts them against hardcoded literals — a registry-against-itself tautology. The missing verdicts (the whole point) are author-asserted, never measured; probe: {2028: null} is not confirmed against the engine. To make the check real, wire a CI job that installs the pinned engine and runs --probe (e.g. on a schedule) — otherwise it can't catch the regression it exists to catch.
• data/uk/baseline_integrity.json + the PR body cite data/externals/obr-policy-effects.json (#75) as the evidence for the consistent fuel-duty verdict, but that file does not exist on main. The consistent verdict rests on an absent/unmerged artifact, and the test only asserts the string -0.129pp appears in prose — evidence is text, not a computed check.

Should

• baseline_integrity.json uses bare Infinity literals (lines ~20, ~29). Python's json.load accepts them, but the file is not valid JSON — any strict/JS consumer on the app/ side will fail to parse it.
• pipeline/ is a new top-level dir; all existing pipeline code lives in scorecard_db/. ci.yml lists pipeline in ruff targets, but the module isn't imported by build_db or fed into the no-drift build — effectively dead outside its own test.
• The reprobe() pin guard raises SystemExit if the installed engine ≠ 2.89.2; combined with never running in CI, the engine path will bit-rot silently once the pin advances.

🤖 review via Claude Code