[juaristi22]

Fixes #880

Part of #728. This makes the executor serve policyengine.py 6.10.0 (PolicyEngine/policyengine.py#685), which makes the Microcosm national dataset the UK default and routes constituency and local-authority regions to the Microcosm local-area dataset. UK-wide and nation-level reports run on the national file alone. Their constituency and local-authority results come from a separate run, requested on demand with breakdown: "local_areas", on the local-area file alone.

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

After this PR, on a release built with .py 6.10.0:

• UK-wide and nation-level requests (region absent, uk or country/<nation>) load only microcosm_uk_2024_25 and build one baseline and reform. They compute no constituency or local-authority results: both fields are null, and the job status body omits them. This removes the second baseline and reform run on the local-area file from every such request (about 70-90 s and 12 GiB each), from every UK budget-window year and from the UK release sentinel.
• Local-area breakdowns. A UK-wide or nation-level request with breakdown: "local_areas" loads only microcosm_uk_2024_25_dense and builds one baseline and reform there with the region's filter. Both area levels then come from one disclosure-controlled release (Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860), and the job returns a LocalAreaBreakdownOutput.
• Constituency and local-authority requests are unchanged: they run on the local-area file, as María's commits below set up.
• Older releases. Worker apps for older .py versions are never rebuilt, and they ignore fields they do not know. The gateway therefore sends a breakdown only to a worker whose routing bundle manifest declares the new capability, and refuses it with 400 for any other.

Contract

• SimulationRequest.breakdown: Literal["local_areas"] | None = None. Accepted only for UK requests whose region is absent, uk or country/<nation>. Refused with 422 for other countries, constituency, local-authority or bare-name regions, include_cliffs: true, segmented: true, a region_group or any other value. include_cliffs: false stays accepted, because the Public API always sends it. BudgetWindowBatchRequest does not take the field.
• LocalAreaBreakdownOutput (extra="forbid"): kind: "local_areas", model_version, policyengine_version, dataset, data_version, constituency_impact and local_authority_impact. Both lists are GeographicImpactOutput and may be empty.
• JobStatusResponse.result is SingleYearMacroOutput | LocalAreaBreakdownOutput | None. Both models forbid extra fields, so a payload validates as only one of them.
• Routing bundle manifest key local_area_breakdown, version 1, written by the routing publisher only for apps whose bundle routes UK area regions. The gateway refuses breakdown with 400, before spawning, for any route whose manifest lacks it, including routes from the legacy version dicts. The error names the .py version.
• The entry starts no Stage 12 comparison for a breakdown request (skip reason unsupported_options).

The gateway OpenAPI golden and the generated client are regenerated.

Commits

María's commits (unchanged):

• 5ad72b0f Build routed UK area results from the local-area dataset
• 6aad6520 Check the routed area dataset in the image smokes
• c2728bce Copy no area codes into Stage 12 for a routed UK national report
• 24690a27 Record the routed UK area changes in the changelog

Added on top:

SHA	Commit	Kind
a96c72c2	Merge origin/main into uk-728-area-breakdowns-microcosm	merge, no conflicts
7dad9206	Load the routed UK area pair through main's dataset and span helpers	red-green
acc1d2bf	Use disclosure-controlled UK geography outputs for constituencies and local authorities	cherry-pick of #860's 3c99e59d (Pavel), adapted for this PR
897e4b56	Withhold the inner areas that sum to a withheld UK area	cherry-pick of #860's 8fb0b788 (Pavel)
0e132423	Type the UK area pair counts that the containment helper reads	Pyright fix for the previous commit
73dfd895	chore(deps): update policyengine to 6.10.0	cherry-pick of #883's 2e98847d (github-actions)
0ec2f4d4	Require the disclosure columns on the routed area dataset before simulating	red-green
3f031fd2	Add breakdown=local_areas to the simulation request and its result model	red-green
f14cc7ba	Run a local-area breakdown on the local-area dataset alone	red-green
841fc732	Stop building the area pair in UK-wide and nation runs on routed bundles	red-green
362a904c	Declare the local-area breakdown capability in the routing manifest	red-green
ef6b7eca	Refuse breakdown requests for workers that do not declare serving them	red-green
ad9c35da	Check UK local-area breakdowns and area-free national results in beta	red-green (local helper tests)
3cdc8149	Document UK local-area breakdowns and record them in the changelog	docs
895d20dc	Remove the UK weight-matrix region fallback	red-green

The cherry-picks use git cherry-pick -x, so each keeps its author and names its source commit. acc1d2bf's message has an "Adapted for #881" paragraph. Its disclosure release reads the pair the area breakdowns use, #881's builder tests patch compute_uk_geography_disclosure_output, and the planning test expects the four household columns. #860 and #883 are still open; this PR now contains their commits.

Red-green

Each commit marked red-green was written tests first. The new tests were run and failed for the expected reason before the code was written, and the tests and the code were committed together. Each commit message records how its tests failed. For example:

• the request tests failed with extra_forbidden where they expect value_error or literal_error;
• gateway polling of a breakdown result returned 500;
• invalid breakdown requests reached the national path (DID NOT RAISE);
• a routed national run loaded both datasets;
• the routing manifest had no local_area_breakdown (KeyError);
• the gateway returned 200 and spawned the job for workers without the capability.

A few new tests pinned behaviour that already held, and passed before the change; their commit messages say which. Examples are the Stage 12 adapter skipping an unknown field, and the non-default-dataset rule for deterministic baseline ids.

A merge and cherry-picks cannot be written test-first. They were checked by running the full suites, as below.

Testing

On the final tree, with Python 3.13 and the pinned ruff 0.15.22:

• Unit tests, each in its own locked environment (uv run --frozen pytest tests/):

  • executor: 1826 passed, 22 skipped;
  • contract: 312 passed;
  • gateway: 267 passed;
  • entry: 668 passed;
  • observability: 35 passed;
  • Stage 12 persistence: 40 passed;
  • fastapi: 14 passed.

• make check passes. It regenerates the client, runs ruff format and check, and runs Pyright in every package. The executor's Pyright errors went from 77 to 68: Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860 removed 6 and the weight-matrix removal 3, and no new error is recorded.

• Other checks:

  • python scripts/run_quality_guards.py passes.
  • python -m scripts.rules render leaves no diff.
  • uv lock --check passes for the executor.

• Generated client: ./scripts/generate-clients.sh, then the policyengine-apis-integ tests that need no services (24 passed). test_ping needs the docker-compose services and was not run.

• On the real 6.10.0 bundle, without loading data, the breakdown path resolves uk, country/wales and country/northern_ireland to microcosm_uk_2024_25_dense with its revision microcosm-uk-2024-25-dense-20261019T000432Z-ee5c2dbe. It refuses constituency/Aldershot, and country/ni is an unsupported region.

• Not run locally:

  • the Docker builds, the Modal image smoke and the local integration job;
  • the beta integration tests: test_calculate_uk_local_area_breakdown is new, and test_calculate_uk_model gains an assertion;
  • an end-to-end breakdown on the private data.

  CI and the beta deploy run them.

Environment variables

None of the changed code reads a new environment variable.

• Routing publisher (update_version_registry.py, changed in 362a904c). The new key comes from uk_area_regions_routed(). That function reads the bundle manifest packaged inside the installed policyengine package (policyengine/data/bundle/manifest.json), with no network access and no environment variable.
  • The publisher's existing variables are unchanged. MODAL_TOKEN_ID and MODAL_TOKEN_SECRET are GitHub Actions repository secrets; both names are listed in the repository's secrets. simulation-deploy.yml passes them to simulation-deploy.reusable.yml with secrets: inherit, and the plan, record and publish steps set them in env:.
  • MODAL_ENVIRONMENT, APP_NAME, POLICYENGINE_VERSION, POLICYENGINE_US_VERSION, POLICYENGINE_UK_VERSION, FORCE_LATEST and RELEASE_ENVIRONMENT come from workflow inputs and job outputs.
  • GITHUB_WORKFLOW_REF, GITHUB_ACTIONS, GITHUB_REPOSITORY, GITHUB_RUN_ID and GITHUB_RUN_ATTEMPT are set by GitHub.
• Executor. A breakdown run downloads the private local-area file through the existing run_simulation Modal function. That function already attaches the pe-uk-private-hf-read-token Modal secret, with required key HUGGING_FACE_TOKEN, and gcp-credentials. Constituency and local-authority runs on 6.10.0 already use the same download. I confirmed the attachment in code only; I did not check that the Modal secret exists in Modal.
• Gateway and entry read nothing new.

Code-owner review

These commits change files under rules/, which need code-owner approval (sim.process.rule-machinery-code-owned):

• rules/baselines/pyright-executor.txt and its count in rules/registry.toml: 77 to 71 in acc1d2bf, then 71 to 68 in 895d20dc. Both only remove entries.
• rules/baselines/sim-microdata-weight-stripping.txt and its count: 1 to 0, from Use disclosure-controlled UK geography outputs for constituency and local authority impacts #860's commit.

Rollout

1. Merge and deploy this PR. That creates policyengine-simulation-py6-10-0 with the local_area_breakdown capability, and moves latest to 6.10.0. The Public API still pins 6.7.5, so its users see no change. Unpinned clients get national UK results without area data.
2. The app-v2 change can merge on its own. Outputs from 6.7.5 carry area data, so its Compute buttons do not appear yet.
3. The Public API change sends breakdown and moves to 6.10.0. It needs this PR deployed first.

Risks and follow-ups

• The 6.10.0 executor is frozen once main's lock moves past it, so later fixes to the breakdown code reach only newer versions.
• An entry rollback to an older Cloud Run revision would return 422 for breakdown.
• The local-area file (about 720 MB) is downloaded on every cold container. Measure this in beta, and consider baking it into the image.
• Disclosure control may withhold many areas on the local-area file, or publish nothing for a level. The cost of its 200 bootstrap replicates on that file has not been measured. The beta breakdown test fails if either level is empty.
• country/ni, the Public API's spelling, is not a .py region; .py names it country/northern_ireland. The Public API change maps it.
• data_version on single-area UK runs still names the national release (_selected_dataset_revision is US-only).
• No Stage 12 local-area report exists, so the shadow comparison does not cover breakdown numbers.
• The executor's synchronous FastAPI surface (compat_models.SimulationOptions, used by docker-compose) does not take breakdown.

🤖 Generated with Claude Code