Break equal-start uprating ties the way auto-carry-over does

[MaxGhenis]

#562 and #563 have merged, so this is now one PR on master (9a12a9a5). The head is db5c3de8: this PR's commit fcd4c4a0, rebased onto master, plus five follow-up commits that answer the reviews. Only fcd4c4a0 changes code under policyengine_core/.

Found by the final-commit review of #563 (finding 5). The tie is still on master.

Problem

When a variable with uprating has no formula result for a period, Simulation._calculate uprates the latest earlier input in its own unit. On master it picks that input with max(earlier_input_periods, key=lambda p: p.start).

Two earlier inputs can start on the same day. Holder.set_input calls a variable's set_input helper only for an input in another unit, so a yearly variable stores a year:2012:2 input as given, beside one for 2012. It does this whether or not the variable has a helper. Both inputs start on 2012-01-01, so max returned whichever came first in get_known_periods(): the one stored first, or the one in memory over the one on disk.

Executed on master 9a12a9a5, with an index rising 3.7% a year and inputs year:2012:2 = 1 and 2012 = 5:

Variable	2015, year:2012:2 set first	2015, 2012 set first
uprated_any_unit (no helper)	1.1151576	5.575788
uprated (set_input_divide_by_period)	1.1151576	5.575788
uprated_monthly, month:2012-01:3 vs 2012-01, for 2012-06	1.0	5.0

The results are the same with auto-carry-over on and off.

Fix

Use the order #562 already uses to choose the input auto-carry-over carries. Its sort key moves into one module function, _latest_input_key(period, definition_period), and both paths call it. The key ranks, in order:

1. the variable's own unit;
2. the latest start;
3. the latest end (_end_order, which also orders periods ending after 9999-12-31);
4. the larger unit;
5. the period's string form.

On a tie, uprating now starts from the input that ends last. That is the input auto-carry-over would carry, so uprating with a flat index returns exactly what auto-carry-over returns.

The production diff is that one function and the two max calls that use it. The carry-over path's behaviour is unchanged; only its inline key moved. Two things are deliberately unchanged:

• the uprating factor, which still runs from the source's start;
• the candidate filter: inputs only, in the variable's own unit, starting before the period.

Why "ends last"

• Parity with carry-over. The two paths now agree on which input is the latest one (invariant 3 below). The alternative, "shortest first", would make a variable that gains uprating start from a different input than auto-carry-over carries.
• Most recent information. year:2012:2 says what the variable is in 2013 as well. Auto-carry-over already reads it that way.
• It doesn't depend on the factor question. If the factor later runs from the source's last covered year (see below), "ends last" is still the right source.

Should uprating from a multi-period input be allowed at all?

The factor uses period.start. Uprate a year:2012:2 source to 2015, and two things go wrong:

• it picks up 2012→2013 growth that the input says did not happen;
• 2013 itself, which the input covers, comes out uprated by a year (1.037 from an input of 1).

Changing that would change results without any tie, which makes it a semantic choice rather than a determinism fix. It also sits with related multi-period behaviour that isn't about ties:

• the helper is skipped for a same-unit multi-period input;
• an input starting on the requested period is not an uprating source;
• month:2012-01:12 is stored as the year 2012, because storage keys are period strings.

They are written up in #583, each one executed, with options and a recommendation. This PR keeps multi-period inputs as sources and only makes the tie deterministic.

Exposure: none known.

• PE-US's eCPS h5 has period keys 2024 and 2025 only.
• PE-UK's loader calls set_input(variable, year, …) for each year in dataset.years.
• No YAML test in either repository uses a multi-period period:.
• Every year:…:N string in either repository's source and tests is a parameter update, not a variable input.
• These were checked on PolicyEngine/policyengine-us@fd693d66 and PolicyEngine/policyengine-uk@bfc5fea1.

Invariants

Over the generated domain:

• each variable's own unit, including multi-period inputs that start with another input (year:2012:2, year:2012:10, month:2012-01:3 and others);
• inputs set in any order, some of them on disk;
• earlier requests (plain calculations, ADD, DIVIDE);
• reads from the simulation, a branch or a nested branch, which may set its own input;
• auto-carry-over on and off.

1. Order independence, byte for byte. The uprated value depends only on which inputs are stored. It does not depend on the order they were set in, or on which of them are in memory and which on disk. Tested by test_uprating_does_not_depend_on_the_order_inputs_were_set.
2. Total order. _latest_input_key never ties on two distinct periods: unit, start and end fix the period. So max never falls back to iteration order.
3. Differential: uprating agrees with auto-carry-over. Take an index that never moves. Given the same inputs, an uprated variable returns, byte for byte, what auto-carry-over returns for a twin variable without uprating. Tested by test_flat_uprating_reads_the_input_auto_carry_over_reads.
   • The one stated exception: an input in the variable's own unit, other than the period itself, that starts on the period's first day. Auto-carry-over carries that input, but uprating takes only inputs that start before the period, as before.
4. Reference rule. The result equals the reference rule in tests/fixtures/uprating_order.py, which is written independently of _latest_input_key and breaks ties the same way on both paths.
   • The rule's last_day numbers each period's last day without _end_order: for days, the start's ordinal plus the days after it; for months and years, Period.stop, moved back by whole 400-year Gregorian cycles.
   • A differential property checks that last_day + 1 == _end_order for days, months and years up to 5,000,000 units long, many of them ending after 9999-12-31.
5. Unchanged without a tie. Where one earlier own-unit input starts last, the key's first two components already decide, so the source is the one master picks. The carry-over key is the same tuple as before, and every existing Carry over only inputs, the latest at or before the requested period #562 and Uprate only from inputs, so uprated values don't depend on calculation order #563 test passes unchanged.

Tests

tests/core/test_uprating_order.py goes from 97 to 141 cases, still with no Hypothesis dependency. The new cases cover:

• both set orders, for a variable with a helper and one without;
• monthly inputs;
• 2012, year:2012:2 and year:2012:10, where the string order disagrees with the end;
• memory against disk, in both directions;
• a branch input that starts with a parent input;
• flat uprating against carry-over;
• the unchanged rule that an input starting on the period is not a source;
• inputs ending after 9999-12-31: one such input, two of them (day:9999-12-30:10 and day:9999-12-30:3), and a carry-over between month:9999-01:13 and day:9999-01-01:370. Each is compared with the reference rule.

tests/core/test_uprating_order_property.py goes from 2 to 5 derandomized properties:

• the existing property, with its domain widened to multi-period inputs and 3 new @example rows;
• set-order and disk independence, 300 examples. Each has a 2- or 3-way same-start group, all inputs set in a drawn permutation, and a drawn subset of them on disk. It also has one explicit example that puts the winning input on disk and the losing one in memory;
• the flat-index differential, 300 examples;
• master's helper property, unchanged;
• the last_day differential against _end_order, 500 examples.

tests/fixtures/uprating_order.py gains:

• the reference rule's tie-break, through last_day;
• uprated_flat and uprated_daily_flat, uprated by an index that never moves, and carried_any_unit, their twin without uprating;
• the Windows guard below.

Windows (#526). OnDiskStorage names a value's file after its period. On Windows, numpy.save raises OSError for default_year:2012:2.npy, a name with two colons; CI showed this on the four Windows test jobs.

• At import, the fixture tries to store year:2012:2 in a temporary directory. DISK_STORES_ANY_PERIOD records whether that worked.
• Where it didn't:
  • the property keeps every drawn period whose string form has a colon in memory;
  • the "longer" memory-or-disk regression is skipped;
  • the explicit disk-winner example is a strict xfail(raises=OSError).
• Once Storage keys need a structured (branch, period) scheme: str(period) is lossy and separators collide with branch names #526 lets OnDiskStorage store such periods, the probe succeeds and all of these run on disk again.
• example(...).xfail needs Hypothesis 6.62.0 or later, so the dev extra now requires hypothesis>=6.62,<7. uv.lock's specifier matches. The lock's own project version was already stale on master (3.32.12 against 3.32.17), and this PR leaves it.

Against master's code, 22 of the 447 tests in the five uprating and carry-over files fail: 19 regression cases and 3 properties.

Mutation check on db5c3de8:

• each mutant runs in a scratch tree;
• POSIX runs the five uprating and carry-over files plus test_storage_input_index.py: 465 tests, 0 failing unmutated;
• winsim runs the three uprating files under a plugin that makes numpy.save and tempfile.mkstemp reject a file name containing :, so the probe fails as it does on Windows: 377 pass and 2 are skipped, unmutated.

All are killed. Failing tests:

Mutant	POSIX	winsim
Uprating by start only (master's rule)	22	20
Same, with the candidates reversed	24	22
Uprating prefers the shorter input	37	35
Key without the end order	14	10
String form before the end order	14	10
Key without the own-unit preference	5	3
Carry-over by own unit and start only	8	3

On the reference rule itself, 3 more mutants, run on the overflow regressions and the last_day property:

• the old overflow handling fails 6 tests;
• dropping the 400-year shift fails 3;
• an off-by-one last_day fails the property.

How often the set-order property reaches a tie. On the 300 examples it runs, a tie decides the source in 76. In the draws it splits between memory and disk, the winner is always in memory, which is why the explicit disk-winner example exists.

Local runs:

• On db5c3de8:
  • the five uprating and carry-over files: 447 passed;
  • ruff format --check .: clean.
• On f081b7c4, which differs from db5c3de8 only in the changelog fragment:
  • the full tests folder: 1532 passed, 4 skipped, 1 xfailed;
  • country-template YAML: 39 passed;
  • the three uprating files under the Windows simulation: 377 passed and 2 skipped with the guard, and the expected 3 failures without it;
  • the standalone repro: both set orders give 1.1151576, with auto-carry-over on and off. On master 9a12a9a5 it gives 5.575788 when 2012 is set first.

CI on db5c3de8 passed: all 18 checks in run 37395979986.

Review

• GPT-6.1 Sol, on f051cc28: REQUEST CHANGES.
  • P2. The shared key evaluated period.stop, which raises OverflowError for a period ending after 9999-12-31. Fixed: the key uses Carry over only inputs, the latest at or before the requested period #562's _end_order.
  • P3. The flat differential skipped more than the one diverging case. Fixed.
• GPT-6.1 Sol, on e12993c5: APPROVE.
• Opus 5.5, twice after the rebase onto master (6148851d, then 3f222e1f): APPROVE WITH NITS each time, no code defect. Between them they found that:
  • the guard keyed on sys.platform; it is now a probe;
  • no example put the winner on disk; the explicit example now does;
  • comments and commit messages overclaimed what Windows rejects;
  • example.xfail needs Hypothesis 6.62. All are fixed.
• Opus 5.5 and GPT-6.1 Sol, on 578911f2: both APPROVE WITH NITS, no production defect.
  • Sol found that the reference rule misordered two inputs ending after 9999-12-31, which fell to string order, and an overflowing day input against a month input that ends later. The simulation was right. This is fixed by last_day, with regressions and the differential property.
  • The rest were commit-message and docstring wording, now fixed.
• On f081b7c4: two independent GPT-6.1 Sol reviews, one APPROVE and one APPROVE WITH NITS. The nit was that the changelog fragment did not say the tied inputs are in the variable's own unit. Fixed.
• On b7713e4d: two independent GPT-6.1 Sol reviews, one APPROVE and one APPROVE WITH NITS. The nit was that a commit-message example needed "with no set_input helper". db5c3de8 rewords that message; its tree is identical to b7713e4d's.

The rewordings kept every reviewed tree: the reviewers checked b43c3562, 7e2aa238 and 49ed6a4a, and git diff --quiet b7713e4d db5c3de8 is clean. The commit messages form the squash body, so they were reviewed as well.

Not in scope

• Multi-period input semantics, as above (Multi-period inputs in a variable's own unit: helper skipped, uprating base, storage aliasing #583).
• Lossy storage keys and Windows file names (Storage keys need a structured (branch, period) scheme: str(period) is lossy and separators collide with branch names #526).

axiom: n/a: core engine semantics, no policy encoded.

🤖 Generated with Claude Code