Give each disk-backed holder storage its own directory

[MaxGhenis]

Problem and change

After adopting #585, independently recreated holders for the same variable still used the same temporary subdirectory. Collecting an old holder could remove files needed by its replacement or a clone. A missing caller-supplied storage directory also failed on first use, and overwriting an enum with a plain array retained stale enum metadata.

This PR's runtime changes relative to its merged master base are:

• Holder.create_disk_storage() uses mkdtemp(prefix=f"{variable}_") for each independently created temporary storage. Explicit directory arguments and preserve=True retain the directory/variable dump layout.
• Simulation.data_storage_dir creates missing caller-supplied directories and inherited parent directories on first use.
• OnDiskStorage.put() removes enum metadata when the replacement is a plain array.

The implementation retains #585's TemporaryStorageDirectory ownership and clone/copy lifetime model. Cloned storages share the source directory and copy their mappings. Writes to shared files use mkstemp in replaced/; later writes reuse the writer's own unshared replacement. restore() reads direct key files only, ignores replaced/, and does not select the latest write across independently constructed storages. Independent writers using the same explicit directory have no snapshot-isolation guarantee. Files restored after cloning are also outside that guarantee; clone or copy after restoring to protect the restored values.

The API page describes these contracts and warns that exits skipping finalizers, such as os._exit in multiprocessing workers, can leave uniquely named holder subdirectories accumulating in a caller-owned directory.

Invariants and regression coverage

Existing property and differential coverage is preserved. No load-bearing runtime logic changed in this review follow-up.

• Replacing holders leaves earlier holders' and clones' files readable; each independently created temporary holder owns a distinct directory.
• Clone/copy protects the files shared when it is created from later family writes. Restoring additional files after cloning and independent same-directory writers are outside this guarantee.
• Explicit and preserved storages retain the dump layout; caller-owned directories outside simulation-created temporary folders remain after normal collection. Temporary folders and nested caller-selected directories are removed when their owning references are released, subject to preservation and the documented fork lifetime limit.
• Plain overwrites of enum values read back as plain arrays.
• The disk/memory differential compares returned values or exception class names after generated operations, checks every live mapped file exists, and checks final cleanup without finalizer errors. It runs 75 generated examples of up to 30 operations plus two explicit holder-replacement examples. The directory-lifetime property retains its 150 generated examples.
• Restore tests pin direct-file reads, including ignoring replacements after cloning. They provide no latest-writer invariant.

test_disk_storage_ownership.py contains 26 test functions and 33 parametrized cases. Most cover contracts inherited from #585; no claim is made that every case fails on the base. This review's untracked description, validation-provenance and cleanup-documentation probes failed on the saved original body/page and pass on the corrected versions. These are documentation/evidence regressions; the review identified no runtime P1/P2 defect.

Validation

Local results below are from c616364cd65d8a79ef10bbe275e94b91ef6167b4 on macOS with Python 3.13.9. Runtime and test files are identical to reviewed head 85c3e787be26e38966970dae2ba60362fbe26b30; the follow-up commit adds only the cleanup sentence. Dependency setup in a new environment was interrupted before tests ran; local runs use an existing read-only environment with Hypothesis 6.141.1 in a separate overlay, rather than claiming a fresh lockfile installation. Both property modules run without skipping.

Recorded local dependency versions: NumPy 2.5.3, pytest 9.1.1, h5py 3.16.0 and numexpr 2.14.2.

Each file ran separately in the foreground using uv run --python 3.13 --active --frozen --no-sync --with hypothesis==6.141.1 python -m pytest -q tests/core/<file> -W error::pytest.PytestUnraisableExceptionWarning. The environment selects the current checkout through PYTHONPATH and disables bytecode writes to keep external dependencies read-only.

File	Result
test_disk_storage_ownership.py	33 passed
test_disk_storage_differential.py	1 passed
test_data_storage_dir.py	40 passed, 2 expected xfails
test_data_storage_dir_property.py	1 passed
test_on_disk_storage_clones.py	27 passed
test_branch_scoped_delete_arrays.py	6 passed
test_dump_restore.py	1 passed
test_holders.py	22 passed

Total: 131 passed, 2 expected xfails.

The two existing strict xfails in test_data_storage_dir.py document a forked child reading after its parent removes the inherited directory. They remain unchanged.

For the reviewed head, CI run 37847856790 has 19 successful checks, including core tests and country-package smoke checks on Linux and Windows with Python 3.11–3.14, lint, and changelog validation. CI checked out synthetic merge 501a2c27379612f5b123754b8a9b1470af312309, containing the reviewed head; it did not test the PR-only checkout. Its Ubuntu Python 3.13 core job reported 1791 passed, 4 skipped and 3 xfailed, plus 39 country-template cases; package and documentation builds also succeeded. These CI results precede the documentation follow-up and are recorded separately from the local results.

Full suites, Windows tests, country-package tests, package builds and make documentation were not rerun locally. ruff format --check and ruff check on the 10 Python files in the PR diff, plus git diff --check, passed.

Impact

Measured: focused synthetic storage/holder/dump tests, disk-versus-memory differential comparisons, directory-lifetime properties, and the existing CI checks described above. This is core disk-storage infrastructure used through MemoryConfig, with no policy formula change.

Microsimulations were not run. Country model-output differences, aggregate fiscal effects, partner baseline differences, performance and disk-space impact were not measured. CI's country smoke checks are initialization checks; microsimulation checks were skipped. The hub owns US/UK microsimulation and partner-impact measurement. No partner test files were edited.

Overlap, ordering and HOLD

• Remove a simulation's on-disk storage folder once nothing reads it #585 supplies the current ownership and replacement-file architecture. #560's sequence/input metadata overlap must be revalidated against that architecture; the earlier resolution for the superseded layout is not a verified composition with this head.
• #552 touches nearby OnDiskStorage.delete and key parsing. Underscore branch names and contained-period deletion remain outside this PR's scope; the differential uses branch names without underscores and whole-variable deletion.
• Existing limitations also include mutating returned arrays and : branch names differing between memory and disk, and deleted mappings leaving files until directory cleanup.
• Current d899 order: Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 → Keep the record of set_input values in step with storage #561 → Cache ADD and DIVIDE results only where a plain read returns them #571 → Count only inputs as already set when a set_input helper splits a longer period #581, with Give each disk-backed holder storage its own directory #558 alongside. The older hub sequence ended Keep branch period reads, dumps, and disk deletion consistent #552 → Rebuild a subsampled simulation from its inputs, not its calculated values #573 → Evaluate recursions over periods in full when they reach an input or the start of a formula #572 → Give each disk-backed holder storage its own directory #558; it does not authorize lifting any hold.
• HOLD: retain Max's inherited core-semantics merge gate and the recorded Resolve parameter nodes at instant on first read #570/Evaluate recursions over periods in full when they reach an input or the start of a formula #572 holds. This PR remains draft and requires the hub's merge decision; this repair does not lift those gates.
• Max's 2026-10-07 leaf-input principle, d1024 Proposal A (fresh parameter lookups reflect updates), and d1026 Proposal A (each weight column retains its own entity total under subsampling) remain in force.

Documentation review: durable storage contracts match the implementation; the review follow-up adds the caller-directory cleanup caveat. Impact is low and confidence is high for the documented storage behavior; cross-process lifetime limits and unmeasured country impact remain explicit.

[MaxGhenis]

US + core hub merge audit, core#558 at c616364cd65d8a79ef10bbe275e94b91ef6167b4 (squash)

• Max's ruling d899 (2026-10-08): merge the core order Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 → Keep the record of set_input values in step with storage #561 → Cache ADD and DIVIDE results only where a plain read returns them #571 → Count only inputs as already set when a set_input helper splits a longer period #581, with Give each disk-backed holder storage its own directory #558 before or alongside, each on the hub's gates. Restore inputs as inputs, evict the fast cache on holder writes, reject non-numeric uprating and defined_for #576 has landed.
• Change: disk-storage branch ownership, on top of Remove a simulation's on-disk storage folder once nothing reads it #585's architecture.
  • Independently created temporary holders each get their own mkdtemp directory.
  • Explicit and preserved storages keep the directory/variable dump layout.
  • Replacing holders leaves earlier holders' and clones' files readable.
• Reviews: the r1 delta review requested changes (P2.1 body accuracy, P2.2 test evidence). The fix round addressed both, and delta review r2 is APPROVE at this head, with no P1 or P2.
• Tests (fix-round evidence): 131 passed across eight files, plus 2 expected xfails; both Hypothesis properties ran.
• Gates checked live at merge:
  • gh pr checks exits 0;
  • MERGEABLE;
  • not a draft;
  • no CHANGES_REQUESTED review;
  • head pinned.
• Impact: storage layout only; no formula or parameter changes.