Evaluate recursions over periods in full when they reach an input or the start of a formula

[MaxGhenis]

Finite anchored period recursions are evaluated in full regardless of previously calculated periods, tracing, or persistent cache policy. Unanchored recursions still use the existing depth limit, and values reached by a cut are discarded when the calculation ends.

Repair and invariants

• Deeper periods are scheduled iteratively. Once a deferred key starts evaluation, recursion cannot defer it again; its pending calculation is retried after deeper dependencies finish. For a fixed registered branch graph with finitely many reachable calculation keys, scheduling makes progress without a cache-dependent 10,000-deferral budget.
• Only actual readable inputs of the variable's own period unit, or formula boundaries in the direction of recursion, authorize deeper evaluation. Deleted inputs and derived replacements cannot remain anchors through stale registry entries.
• Outstanding scheduling requests are recorded before they are raised and checked outside formula exception handlers before publishing a result.
• Deferred periods retain the caller scope required by requires_computation_after.
• A temporary cache holds completed, cut-free deferred results independently of tracing, _do_not_store, and cache blacklisting.
• Spiral cleanup removes exact derived entries from their owner; independent descendant inputs and contained periods survive. Clones drop copied cut values before becoming usable. Existing explicit invalidation behavior remains separate.
• Flat traces prefer accepted completed calculations over abandoned or provisional attempts, retaining their dependencies and parameter accesses ahead of later cache reads.

Round-2 findings

All eight review findings are addressed: descendant input deletion; swallowed scheduling exceptions; lost prerequisite caller; invalid clone snapshots; carry-over from calculated values (settled by master's input-only provenance); None retry traces; tracing/cache-policy differences; and cache-dependent deferral exhaustion. The inherited memory sharing, input/derived metadata, disk directory lifetime, clone aliases, system entities and reform baselines from master are preserved.

The original 200-example recurrence order property is retained. Added differential properties compare finite forward/backward recurrences with an independent iterative NumPy evaluator across tracing, storage policies, guarded formulas and caller scope; additional properties cover cut copies, descendant inputs and deleted input provenance. Exact deletion is compared across copied memory, shared memory and disk snapshots.

Validation

Targeted, serial single-file checks: 245 passed in total.

File under tests/core/	Result
test_spiral_order.py	109 passed; includes a cold/warm 10,002-month anchored chain
test_spiral_order_property.py	1 passed, original 200 generated examples retained
test_spiral_scheduler_property.py	4 passed, 60/40/24/24 generated examples plus explicit examples
test_spiral_storage.py	12 passed
test_spiral_trace_retries.py	6 passed
test_cycles.py	5 passed
test_fast_cache_guards.py	5 passed
test_clone_invalidated_caches.py	14 passed
test_carry_over_order.py	4 focused provenance cases passed, 60 deselected
test_docs_deployment.py	5 passed
test_simulation_copy_pickle.py	78 passed after the final master merge
test_simulation_copy_pickle_property.py	2 passed after the final master merge, 60/40 generated examples

The final merge leaves the repaired scheduler/storage/tracer source and spiral tests unchanged. The eight deleted-input cases also passed again through the offline uv run launcher before the final clean master merge. Ruff format/check passed for all 19 repaired or incoming Python files, and git diff --check passed. Final copy checks emitted only the existing NumPy generic-timedelta deprecation warning.

Repaired head: f34b54c569c6d700ca18046e8ff106c0379ce5e3.

Baseline checks on the history-preserving merge 2ed6c34db0f5e6eae1b9dee6397c5c034938bb93 reproduced 32 failures, with 42 passing and 27 deselected cases. Carry-over cases passed there because master's input-only provenance already repairs them. The additional deleted-input regression failed in all eight cases before the provenance fix.

The fresh dependency install failed on PyPI DNS. Successful checks use the existing read-only Python 3.13 environment, with this checkout first on PYTHONPATH; the final focused provenance and copy checks launch it through offline, frozen, no-sync uv run. The first run of the added recurrence differential file hit Hypothesis's input-generation timing health check (one failed, three passed); only too_slow was suppressed for its two bounded generators, with every example count and assertion retained, and all four tests then passed. No full suite, country tests, partner baselines, documentation build or new microsimulation was run. Earlier full-suite and impact results are historical evidence only.

Composition and status

This remains draft and held for Max's core-semantics merge decision. Current master 0946ba3d826b9a9bfece08b28008b52a37b554c4, including its copy/pickle repairs, is merged without rewriting history. Independent source review of the exact repaired head found no introduced P1/P2; current-head green CI and downstream impact remain required before landing. #576 is absent from this master; the second lander must connect this PR's new Holder.delete_array to #576's _evict_fast_cache, preserving input protection and exact spiral cleanup. #566 is planned to fold into #576. No decision ID is associated with this bounded repair.

Conditional formula-only base cases and defined_for masks still use the existing cut behavior. Mixed-direction finite recurrence graphs have less differential coverage than same-direction anchored graphs; their existing regressions and the original order property are preserved.

axiom: n/a: core engine repair, no policy encoding.

Historical downstream evidence

The following measurements belong to the original PR head and its original scratch merge. They have not been repeated on f34b54c569c6d700ca18046e8ff106c0379ce5e3; current-head impact and partner gates remain pending.

Real runs on master b78b0ba and on all three of #571/#572/#573 merged together (scratch merge 37aadada), each output compared array by array (compare.py). The PE-US 2026 run was also repeated on each branch alone, and the 2035 run on #572 alone; all identical too.

Run	Outputs compared	Result
policyengine-us 6c5170fd, eCPS 2024, 3,000-household subsample, 2026 (income tax, itemizing and SALT branches, state taxes, CTC, EITC, SNAP, household and SPM net income, weights, marginal tax rates)	25 arrays	bitwise identical
same, 2035	25 arrays	bitwise identical
policyengine-uk 7b9fc379, enhanced FRS 2024-25, full sample, 2026 (income tax, NI, UC and legacy benefits, Pension Credit, HB, council tax, HBAI income, poverty, marginal tax rates)	36 arrays	bitwise identical

A probe of the PE-US 2026 and 2035 and PE-UK 2026 and 2042 runs on master found one recursion cut by the limit: PE-US weeks_worked in 2035 (below).

PE-US weeks_worked (formula_2025: last year's value) is the one recursion the probe saw reach the limit on real data: a new 2035 simulation cuts it at 2025. The eCPS file carries no weeks_worked, so every year is the default 0 and the cut changes nothing above. Where the input exists it does. A real policyengine-us run on a household with one worker (40 weeks entered for 2024):

2035	New simulation	After calculating 2030
master: weeks_worked, spm_unit_work_expenses	0, $0	40, $1,800.14
this branch	40, $1,800.14	40, $1,800.14

So household calculations from 2035 on (ten years after the 2025 formula start) change, to the value calculating year by year gives.

Generated with Claude Code.