Add main residence value to LA calibration

[vahid-ahmadi]

What this PR does

Adds a derived proxy LA-level main-residence-value calibration target to datasets/local_areas/local_authorities/loss.py. Per-LA target is a constructed product, not a directly observed total:

y_la = avg_house_price_la × ownership_share_la × n_households_la
matrix col = main_residence_value (per household, from policyengine-uk)

Same multiplicative shape as the existing private-rent target (median_rent × renter_share × n_households). LAs missing any input (Wales / Scotland / NI — EHS is England-only) fall through to the national_property × la_household_share fallback, identical to how the tenure target handles missing LAs.

Lineage caveat (flagged in review by @MaxGhenis)

This is a derived/proxy target, not a direct benchmark:

• Matrix col main_residence_value (policyengine-uk) is WAS-imputed stock wealth, regionally uprated via property-wealth intensity ratios.
• Target value uses HMLR UK HPI "Average Price" — a transaction-weighted geography-period price index, not an observed stock total of owner-occupied main residences.
• The product avg_price × ownership × n_households is a defensible identity ("if every owner-occupied dwelling were valued at the LA HPI average, the total would be £X") but the two sides of the calibration constraint reference different price concepts.

A separate policy question — whether derived/proxy targets like this should sit at full training weight alongside directly observed targets (HMRC SPI, ONS pop, DWP UC, VOA dwellings), or be soft-weighted — is being tracked separately and is not blocking this PR.

Closes #370.

Files

New

• policyengine_uk_data/storage/la_land_values.csv — 360 rows: code, name, households, avg_house_price. avg_house_price from HM Land Registry UK HPI Dec 2025 with name-based fallback for re-allocated codes (Sheffield E08000019 → E08000039), NI country-level fallback for missing LGD months, national-avg fallback for the Isles of Scilly.
• policyengine_uk_data/targets/sources/la_land.py — load_la_avg_prices(), _compute_la_targets() (observed-input product, no national-total apportionment), get_targets() returning Target objects named housing/main_residence_value/{code} with source=hmlr, geographic_level=LOCAL_AUTHORITY.
• 18 unit tests in tests/test_la_land_value_targets.py, 8 in tests/test_la_loss_land_value.py.
• changelog.d/370.md.

Modified

• datasets/local_areas/local_authorities/loss.py — adds the housing/main_residence_value column following the rent-block pattern: merge avg_house_price into tenure_merged, compute target inline, apply np.where(has_property, target, national * la_household_share) fallback. Same shape as the surrounding tenure / rent / ONS-income blocks.

Tests

26 new tests cover:

• CSV data quality — row count, schema, value bounds, no missing values, IoS regression fixture (households in [500, 5_000]).
• Target computation — every per-LA target equals avg_price × ownership_share × n_households exactly; all-positive; English LAs covered (Wales/Scotland/NI fall through to the loss.py national-share fallback by design — same behaviour as the existing tenure target, which only has EHS England data).
• Sanity ordering — Kensington & Chelsea total exceeds Blackpool; sum of London LA targets exceeds North-East total by ≥3×.
• Registry integration — 291 Target objects produced (one per English LA), all tagged local_authority and source=hmlr.
• Loss-matrix wiring — housing/main_residence_value column present in both matrix and y; per-LA y equals the observed-input product for covered LAs; matrix column equals sim.calculate("main_residence_value") (gated on enhanced-FRS fixture).

Full run including adjacent suites (regional land, target DB, target registry, release manifest): 72 passed, 15 skipped (FRS-fixture-gated), no regressions.

Sanity check

LA	avg HMLR price	ownership share	households (Census)	target
Kensington & Chelsea	£1,178k	~29%	~80k	£27.8bn
Blackpool	£130k	~62%	~65k	£5.1bn

Both ordering (K&C ≫ Blackpool) and absolute level (£10s of bn per LA) look right.

Sources (constructed inputs, not direct LA totals)

• HM Land Registry UK HPI — Dec 2025 — avg transaction prices.
• English Housing Survey via la_tenure.xlsx — ownership shares, England only.
• Census 2021 via la_count_households.xlsx — household counts.

Related

• Regional property wealth calibration: WAS-imputed main_residence_value doesn't match UK HPI by region #314 — regional intensity ratios (merged).
• Add regional household land value calibration targets #315 — regional household land value targets (uses the older national-apportionment shape; could be revisited as a follow-up if we want full consistency at regional level too).

[MaxGhenis]

Blocker: data bug in la_land_values.csv — Isles of Scilly (E06000053) has 2,492,115 households. The real figure is ~1,000 (pop ~2,000, 1,115 households per ONS mid-2023 estimate).

Impact: IoS alone absorbs 8.6 % of the national household share (2.49M / 29.0M), which the methodology then multiplies by the national household-land value — £65 bn of UK household land 'lives' in Scilly under this target, depressing every other LA's share by that amount. London LAs take the biggest hit because their share-of-average-price is highest.

Quick verification:

$ awk -F',' 'NR>1 {sum+=$3} END {print sum/1e6}' la_land_values.csv
31.5   # with IoS bug
$ awk -F',' 'NR>1 && $2 != "Isles of Scilly" {sum+=$3} END {print sum/1e6}' la_land_values.csv
29.0   # without — matches ONS ~29.4M

Looks like a UK-HPI 'national-total-as-fallback' path leaked into one LA row. Likely two lines to fix:

1. Correct la_land_values.csv row to E06000053,Isles of Scilly,1115,308582 (or whatever the canonical source gives).
2. Add a test that bounds households per LA, e.g. assert (df.households.between(500, 500_000)).all(). None of the current 18 tests catch a 1000x outlier.

Happy to approve once that's in. The methodology itself is sound — mirrors mhclg_regional_land.py::_compute_regional_shares correctly and the target shape aligns with the regional targets.

[vahid-ahmadi]

@MaxGhenis thanks — fixed in 3ed729c.

Data fix

• Patched E06000053,Isles of Scilly from households=2,492,115 to 1,115 (ONS mid-2023 estimate). avg_house_price=308,582 kept as-is.
• Post-fix UK household total across the CSV drops to 29.73M, in line with the ~29.4M ONS figure you quoted (was 31.5M pre-fix, matching your awk).

Quantified impact of the fix

• IoS share of national property-wealth share: 7.85% → 0.0038%.
• K&C 2024 target: £42.6bn → £46.2bn (+8.5%). Every non-IoS LA gets the same ~8.5% uplift; London LAs were the most suppressed.

Tests added

• test_households_within_plausible_range — bounds every LA to [500, 500_000] per your suggestion. A future fallback leak of this class fails immediately.
• test_isles_of_scilly_households_are_thousands_not_millions — explicit [500, 5,000] bound on E06000053 so the specific row that leaked has a named regression test.

Full suite: 20/20 pass locally via uv run pytest policyengine_uk_data/tests/test_la_land_value_targets.py.

Generation-path note: the 2,492,115 figure matches the South West regional household total, so the fallback that fired during CSV generation was a regional sum, not "national-avg" as the PR body suggested. I'll correct the PR description; worth flagging for whoever regenerates the CSV next.

[vahid-ahmadi]

@MaxGhenis — for context, here is the full set of LA-level targets the reweighter trains on after this PR (from datasets/local_areas/local_authorities/loss.py):

Family	Columns	Quantity	Source
HMRC income	hmrc/{var}/amount, hmrc/{var}/count for each variable in INCOME_VARIABLES (employment, self-emp, pension, dividends, …), filtered to in-SPI frame (income_tax > 0)	£ totals + counts	HMRC SPI table 3.15, scaled to national projections
Age	age/{lower}_{upper} — one column per age band	Population count	ONS mid-year estimates, scaled to 90% of UK total
Universal Credit	uc_households	Benunit count on UC	DWP Stat-Xplore
ONS small-area income	ons/equiv_net_income_bhc, ons/equiv_net_income_ahc, ons/equiv_housing_costs	£ aggregates of equivalised HBAI net income (BHC/AHC) and implied housing costs	ONS small area income estimates × Census household counts, with national-share fallback for missing LAs
Tenure mix	tenure/owned_outright, tenure/owned_mortgage, tenure/private_rent, tenure/social_rent	Household counts by tenure	English Housing Survey × Census household counts, with national-share fallback
Private rent £	rent/private_rent	Aggregate annual private rent	VOA/ONS median rent × EHS renter share × Census household counts
Main residence value (this PR)	housing/main_residence_value	£ aggregate main-residence value of owner-occupiers	HMLR UK HPI avg price × EHS ownership share × Census household counts

All targets follow the same shape — observed per-unit value × observed share × observed count, with a national × la_household_share fallback for LAs missing any input. Plus a country mask so e.g. an English household never contributes to a Welsh LA's targets.

[vahid-ahmadi]

The LA target now uses directly observed housing indicators (avg_house_price × ownership_share × n_households) instead of apportioning a national ONS land-value total. The shape now mirrors the existing private-rent block exactly:

y["rent/private_rent"]            = median_rent × renter_share × n_households
y["housing/main_residence_value"] = avg_price  × ownership_share × n_households   ← this PR

Both go through the same np.where(has_data, target, national_share_fallback) pattern. Wales/Scotland/NI fall through to the national-share fallback because the EHS only covers England — same behaviour as the existing tenure target.

Matrix variable changed from household_land_value (an imputed regional-intensity rescaling of property wealth) to main_residence_value (a direct FRS observable). Target name changed from ons/household_land_value/{code} to housing/main_residence_value/{code}.

72 passed, no regressions. Sanity check: K&C target £27.8bn vs Blackpool £5.1bn, ordering and level both look right.

[vahid-ahmadi]

Addressed the framing issue flagged the review (commit 717c9cd):

• PR description, la_land.py docstring, and loss.py comment now call this target a derived proxy, not "directly observed". I had been overselling it.

• Lineage caveat is now documented explicitly: PE main_residence_value is WAS-imputed stock wealth; HMLR HPI "Average Price" is a transaction-weighted geography-period price index, not an observed stock total of owner-occupied residences. The two sides of the constraint reference different price concepts. The product is a defensible identity, but it's a derived proxy.

No behaviour change in this commit — only documentation/labelling, since you said "#371 is not necessarily wrong, but it should be explicitly treated as a derived/proxy calibration target, not described as direct."

Open policy question that I'm not solving here: "If the standard is 'only calibrate direct official targets,' #371 should not be a hard training target as written." This applies repo-wide — tenure/*, rent/private_rent, ons/equiv_*, and now housing/main_residence_value are all derived proxies sitting at full training weight alongside direct VOA/HMRC/ONS/DWP targets. Worth a separate PR / decision. Happy to open a tracking issue if you want to formalise the policy.

[MaxGhenis]

Follow-up on the direct-target discussion: I pushed c330b44 to make housing/main_residence_value validation-only by default instead of a training target. The target is still useful as a proxy diagnostic, but HMLR average price x EHS ownership share x Census households is not a direct LA stock-value target and crosses source/concept boundaries. This also fixes the docstring so it no longer claims soft weighting unless the optimizer actually implements it.\n\nVerification: uv run pytest policyengine_uk_data/tests/test_la_land_value_targets.py policyengine_uk_data/tests/test_la_loss_land_value.py -q; ruff check/format on touched files.

[vahid-ahmadi]

@MaxGhenis — nit on the test side.

The new test_main_residence_value_is_validation_target_by_default only asserts that "housing/main_residence_value" is a member of the VALIDATION_TARGETS constant. That's tautological — it locks in the constant's value, not the calibrator's behaviour. Someone could later remove excluded_training_targets=LA_VALIDATION_TARGETS from create_datasets.py and the test would still pass.

A behaviourally meaningful test would run a small calibration (or use the toy calibrator already in test_calibrate_save.py) and assert the post-fit residual on housing/main_residence_value is not driven near zero — i.e., the cell really wasn't trained on. That's the property the constant is supposed to encode, and it's the property a future refactor could silently break.

Happy to take a swing at it as a tiny follow-up if you want. Not blocking.

[MaxGhenis]

Updated based on Max's target-standard call: nonconforming/proxy quantities should not live in the targets database or calibration target matrix, even as validation-only targets.

Pushed ecfd6c3, which removes the LA housing/main_residence_value target source, LA loss-matrix column, validation-target exclusion, CSV fixture, changelog, and target tests added by this PR. The PR now has no changed files relative to the base.

Recommendation: close this PR rather than merge it. If we want a property-value diagnostic later, it should live outside the calibration target registry/matrix and be labelled as diagnostics, not as a target.

[vahid-ahmadi]

@MaxGhenis you removed all the changes in this PR in ecfd6c3 — the diff against main is now empty. Do you want to close this PR?

[MaxGhenis]

Closing per target-standard decision: this PR now has an empty diff after removing the nonconforming/proxy LA property-value target from the target registry and calibration matrix. If we add property-value diagnostics later, they should live outside the targets database/matrix.