Minimum wage coverage and bite (mode 1): the first external check on PE-UK's minimum-wage machinery (#88) - #93
Minimum wage coverage and bite (mode 1): the first external check on PE-UK's minimum-wage machinery (#88)#93vahid-ahmadi wants to merge 2 commits into
Conversation
minimum-wage machinery (#88) PolicyEngine-UK carries `minimum_wage`, `minimum_wage_category` and the whole `gov.hmrc.minimum_wage` parameter tree, and there was no external claim about the minimum wage anywhere in the 15,858 UK rows. 119 claims from the Low Pay Commission Report 2025 data annexes: minimum_wage_bite 66 per cent of median hourly pay minimum_wage_coverage 13 jobs at or below the rate minimum_wage_coverage_rate 40 per cent of jobs Two things introduced on purpose. A JOBS unit concept. Coverage counts JOBS — one person can hold two and one household several, and ASHE is an employer survey of jobs — so it is not interchangeable with persons, households, benefit units or families. Mapping it onto `persons` because that unit already existed would have misstated what the number is. uk_aliases records lpc:jobs DISTINCT from dwp_hbai:persons and uk_hmrc:individuals. Count and rate stay separate metrics for the same reason poverty_count and poverty_rate do. The bite denominator, on the claim. Bite is the rate over ASHE's median hourly wage of full-time workers; the certified PE-UK world is FRS-based, a household survey whose hourly-pay distribution is not ASHE's. So a PE-vs-LPC bite gap is a SURVEY-POPULATION difference before it is an engine question. That axis rides in conditions["denominator"] on all 66 bite claims, a bite claim without one is REFUSED at staging, and source.json states the axis must be sized before any engine defect is named. This is likely to be the family's most interesting finding rather than a problem with it. What is deliberately not read is tallied, not implied: 191 of the 193 main-annex sheets (the report's other figures, each its own population decision) and the 349-row local-authority coverage table. LA geography is finer than any UK geography this repo registers and the certified engine cannot resolve it either, so opening an LA vocabulary for one table would create identities nothing else can join to. Every claim also says WHICH rate it is about — the adult rate, an age-band rate, or all NMW/NLW rates together are three different questions. LPC's "East of England" is aliased once onto the repo's registered "East". Suite 282 passed, two builds agree on content_hash, no-drift clean, ruff format clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me
DTrim99
left a comment
There was a problem hiding this comment.
Reviewed on tip f934ddc — pytest tests/test_lpc_minimum_wage.py = 20 passed. Clean; same lane discipline as the rest of the stack.
Verified:
- Provenance — both .xlsx SHA-256 pinned and recompute-checked at build + in a test; edition pinned ("Low Pay Commission Report 2025"),
source.jsonuses versioned GOV.UK asset URLs (not "latest"). - Adapter — parses by sheet name + title-anchor (a reshuffled workbook raises), no cell-ref fragility. Coverage(jobs) vs bite(percent) modeled as distinct Metrics from distinct columns; spot-checked UK 2025 raw cells (2,020,000 / 6.6) stage verbatim with no re-derivation or %/count confusion.
- Accounting —
ingestassertscounts == _EXPECTEDandfinishraises on claim_id collision; 119 = 66 + 13 + 40 pinned, and the 191 unread sheets + 349 LA rows are declared drops with reasons (tallied and asserted), so the identity holds. - New metrics
MINIMUM_WAGE_BITE/_COVERAGE/_COVERAGE_RATE+UnitConcept.JOBSkeep count vs rate apart (poverty_count/rate precedent) and are non-duplicative. - Fail-loud on unmapped geography / age-band / rate_scope / unit, and — the strongest design choice — bite refuses to stage without its denominator. All tested.
- Identity —
lpc:jobsDISTINCT fromdwp_hbai:personsanduk_hmrc:individuals; rate_scope keeps adult-rate vs age-band vs regional claims from colliding. - held_out — all 119 → HELD_OUT with an ASHE-jobs-vs-FRS-survey basis;
never_calibratepermanent-holdout precedence is checked before the lpc branch. - Determinism/mirror — data/lanes.json byte-identical to the app/public mirror; the only rebuild "drift" was CRLF-vs-LF from running under Windows Python (CI produces LF), an environment artifact, not a lane defect.
One cosmetic nit: a uk_aliases comment says the region is "aliased here, once," but the "East of England"→"East" alias actually lives in the adapter REGIONS map (functionally correct — staged output never contains "East of England"; the comment is just slightly misleading). No partner test files touched. Approving.
The uk_aliases comment said the East-of-England alias lives 'here, once'; it actually lives in the harvest side's REGIONS map, so the staged rows never carry the LPC spelling and this registry only ever ACCEPTS the repo's own region names. Functionally identical, but the comment pointed at the wrong file. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me
|
Thank you — and the nit is a fair catch, fixed in the latest push. You're right that the alias doesn't live where the comment said. On your CRLF observation — that's the same thing David hit on #92, and there's now a Noted on the rest — particularly that you spot-checked the UK 2025 cells (2,020,000 / 6.6) staging verbatim. That pair is the one I'd most want independently eyeballed, since it's where a count-vs-rate confusion would be invisible. |
Found by re-checking every branch for the defect review caught on #92: DISTINCT is a plain frozenset of "source:value" strings, so nothing stops it naming a source with no registered vocabulary. A pair like that reads as if a distinction had been checked when there is nothing on the other side to check against — and because the pair is just a string, every test asserting it exists still passes. Mine did. This branch claimed DWP's age bands DISTINCT from LPC's minimum-wage age bands. The distinction is real, but the `lpc` vocabulary is registered on #93's branch and not here, so on this branch the assertion pointed at nothing. Deferred with a comment naming the prerequisite, and stated in prose in source.json where it is a sentence rather than a ledger entry. Two occurrences is a class, so it gets a guard rather than a third correction. tests/test_uk_aliases_ledger.py reads the registered sources out of uk_aliases itself — no hardcoded list — and refuses: - a DISTINCT pair naming a source with no registered vocabulary - the stricter form: a pair naming a VALUE that is registered nowhere, since "ukmod:q9" would be equally unfalsifiable plus a check that the guard is not passing trivially on a near-empty ledger. Verified by reintroducing the dangling pair and watching it fail. Suite 283 passed, two builds agree on content_hash, ruff format clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me
source.json cited the LPC lane as "(#93/PR #93)", conflating the issue with the pull request. The issue is #88; #93 is its PR. The uk_aliases comment and the test both say "#93's branch", which is correct — it is the branch that carries the vocabulary — so only the source.json note needed correcting. Small, but this note exists to tell a future reader where the other half of a deferred assertion lives, and a wrong pointer defeats the point of writing it down. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016HuXJFVme8HRbnke2Ey3Me
Closes #88. Branches from
main— independent of #91/#92.PolicyEngine-UK carries
minimum_wage,minimum_wage_categoryand the wholegov.hmrc.minimum_wageparameter tree, and there was no external claim about the minimum wage anywhere in the 15,858 UK rows. 119 claims from the Low Pay Commission Report 2025 data annexes:minimum_wage_biteminimum_wage_coverageminimum_wage_coverage_rateA JOBS unit concept
Coverage counts jobs. One person can hold two and one household several, and ASHE — the survey behind it — is an employer survey of jobs. Mapping that onto
personsbecause the unit already existed would have misstated what the number is.uk_aliasesrecordslpc:jobsDISTINCTfromdwp_hbai:personsanduk_hmrc:individuals.Count and rate stay separate metrics for the same reason
poverty_countandpoverty_ratedo.The bite denominator rides on the claim
Bite is the rate over ASHE's median hourly wage of full-time workers. The certified PE-UK world is FRS-based — a household survey whose hourly-pay distribution is not ASHE's. So a PE-vs-LPC bite gap is a survey-population difference before it is an engine question.
That axis is in
conditions["denominator"]on all 66 bite claims, a bite claim without one is refused at staging, andsource.jsonstates the axis must be sized before any engine defect is named. I suspect this is the family's most interesting finding rather than a problem with it.What is deliberately not read, tallied rather than implied
Both are declared drops with reasons and row counts, asserted by a test.
Every claim also says which rate it is about — adult rate, an age-band rate, or all NMW/NLW rates together are three different questions. LPC's "East of England" is aliased once onto the repo's registered "East".
Verification
content_hashruff format --checkcleanReviewers
@MaxGhenis @DTrim99 — two calls worth challenging. (1)
JOBSas a new unit concept: the alternative is to refuse coverage counts entirely until a jobs population exists elsewhere, but I think an honestly-labelled job count beats no claim. (2) Reading only 2 of 193 sheets: I'd rather land a small, fully-identified population than a large half-triaged one, but the split is a judgement and the tally is there so you can see exactly what I left.