Add Plant IoPort for partitioned Ray ingress-pre-process/integration-sub-process/egress-post-process, Invoice Seed, & UnixFS CAR reads

[JEJodesty]

Summary

Goal: Restore distributed pre/post I/O (Bacalhau CoD–style ingress/egress) using Plant [SaaS] Ray, without putting Ray or Kubo into Process source. Align hotF (RayComputePort / integrated_subproc) with the same partition layout so ingress → integration → egress share stable partition identity (minimize shuffle). Default CATS_IO_PARTITIONS=1 keeps today’s TransportPort.migrate demo path.

Follow-on to #202 (signed BOM / AddressStore / LDP / Solid).

flowchart LR
  inv[Invoice input CID]
  ing[ingress IoPort Ray]
  layout[partition dir CID]
  cache[integration_cache TransportPort]
  hotF[hotF RayComputePort]
  egr[egress IoPort Ray]
  out[Invoice data_cid]
  inv --> ing --> layout --> cache --> hotF --> egr --> out

Loading

What landed toward that goal

• Plant IoPort: Function-owned IoPort protocol; when CATS_IO_PARTITIONS>1, Process ingress/egress use RayIoPort (part-*.car UnixFS directory layout via ContentMesh / AddressStore). Invoice still records one root CID per stage.
• hotF alignment: ComputePort.num_partitions / RayComputePort consume/produce the same part-* count — no repartition(n) when already n. Naming: Higher-Order Transfer Function (hotF) replaces former tHOF wording for integrated_subproc.
• Process import hygiene: function_* batch maps typed as Dict[str, Any] — no numpy import in Process callables at import time.
• Prerequisites: pure-Python UnixFS CID verify + CARv1 / directory extract; gateway-first get / dag_export.

Also in this PR (supporting / adjacent)

• Invoice Seed (Populate Invoice's Seed field #187): Executor mints Process replay dict {seed, rng_seed, num_partitions} into invoice.seed_cid before invoice_cid; flatten_bom resolves invoice.seed; provenance asserts non-null, unique Seeds across CAT0/CAT1. num_partitions on Seed records the observed I/O + hotF alignment n for the run (env-selected today; Seed is the control-plane home going forward).
• Intra-run PROV: signed ExecutionBom stageLineage (prov:Entity + prov:wasDerivedFrom along Invoice stage CIDs) and #executorRun Activity — envelope stays address-only.
• TF lock heal: remove orphaned .terraform.tfstate.lock.info when no live holder before destroy/apply heal.
• Make / deps: deps-uv via pip, deps-uv-sync, Terraform into .venv/bin; content-store-init / ensure / shutdown; committed Structure .terraform.lock.hcl files.
• Docs: INTEROP / PLANTs / ControlFeedbackLoop (IoPort + hotF), IPFS / STORAGE (UnixFS + partition CAR), BOM / Lineage / W3C (stageLineage + Seed), DEPS / ENV / README.

Design choices

Choice	Decision
AQ ownership	Process stays Plant-agnostic; Function-owned IoPort; Plant ships RayIoPort under plant_cid (mirrors ComputePort)
n == 1	Keep transport.migrate — iris demo unchanged
n > 1	Plant Ray I/O via IoPort; stable part-00000.car … names as shuffle keys
TransportPort	Still used for integration_cache and for ingress/egress when n=1
Seed	Replay dictionary on Invoice (seed_cid); stage CIDs remain interim feedback
W3C / CFL	Invoice records root stage CIDs only; BOM remains address-only

Non-goals: URI-as-address (2b), cross-CAT Order registry, second Plant adapters beyond Ray IoPort, raw-leaves write pins, Executor reading n from Seed instead of env (follow-up).

Data-plane read path (enables partition CAR I/O)

• W3C Integration: signed BOM, AddressStore, LDP cache, Solid dual-write #202: gateway-first cat + Kubo only-hash verify → RPC fallback
• Add Plant IoPort for partitioned Ray ingress-pre-process/integration-sub-process/egress-post-process, Invoice Seed, & UnixFS CAR reads #204: gateway-first cat / get / getCar
  • pure UnixFS verify (file layouts)
  • Kubo only-hash fallback (exotic layouts)
  • directory get via CAR + UnixFS extract (RPC fallback)

Control-plane / provenance (address-only envelope)

• W3C Integration: signed BOM, AddressStore, LDP cache, Solid dual-write #202: signed ExecutionBom → invoice_cid / log_cid / node_did / proof → BomLdpStore → optional Solid dual-write
• Add Plant IoPort for partitioned Ray ingress-pre-process/integration-sub-process/egress-post-process, Invoice Seed, & UnixFS CAR reads #204: same publish path, plus:
  • stageLineage Entities (ipfs:// stage CIDs); wasGeneratedBy #executorRun; wasDerivedFrom along data ← integration ← ingress ← input
  • structure_as_executed: generated-by only (observation; not on payload chain)
  • invoice.seed_cid → Seed {seed, rng_seed, num_partitions} (no longer null)
• When CATS_IO_PARTITIONS>1, Invoice stage CIDs may be UnixFS directory roots of part-XXXXX.car shards; AddressStore verify/extract apply per part; the Invoice still holds one root CID per stage

Test plan

Plant IoPort (primary)

• uv run pytest tests/test_ray_io_partitions.py -q
  • CATS_IO_PARTITIONS=1 (default): ingress/egress stay on TransportPort.migrate
  • CATS_IO_PARTITIONS>1: ingress/egress use RayIoPort part-*.car layout; Invoice stage fields remain single root CIDs
  • Process callables remain Plant-agnostic (TYPE_CHECKING-only data.* / IoPort imports; no numpy at import)
• uv run pytest tests/test_compute_plant_ports.py -q
  • ComputePort.num_partitions aligned with IoPort; no shuffle-heavy repartition when already n

AddressStore (UnixFS + CAR prerequisite)

• uv run pytest tests/test_unixfs_cid_chunked.py -q
  • Pure UnixFS verify for single- and multi-block file layouts; tamper fails
• uv run pytest tests/test_unixfs_car_extract.py -q
  • Directory get via CAR + UnixFS extract
• uv run pytest tests/test_address_store_gateway.py -q
  • Exotic layouts fall back to Kubo only-hash
  • Gateway-first get / getCar with RPC fallback
  • ContentMesh.get / getCar wired through AddressStore

stageLineage PROV + Seed

• uv run pytest tests/test_stage_lineage_prov.py -q
  • Signed bom includes stageLineage; wasDerivedFrom follows Invoice stage CIDs
• uv run pytest tests/test_network_seams.py -q
  • #executorRun Activity on signed envelope
• uv run pytest tests/test_provenance.py -q
  • invoice.seed_cid / flattened seed populated; keys {seed, rng_seed, num_partitions}; CAT0/CAT1 Seeds differ
• Cross-CAT reverse registry remains out of scope

TF state lock heal

• uv run pytest tests/test_terraform_state_lock_heal.py -q
  • Orphaned lock with no live holder removed; live holder left alone

Docs / review

• docs/INTEROP.md / PLANTs.md / ControlFeedbackLoop.md — IoPort vs TransportPort; hotF
• docs/IPFS.md / STORAGE.md — UnixFS verify, CAR get, partition CAR
• docs/BOM.md / W3C.md — stageLineage + Seed
• docs/DEPS.md / ENV.md — Make deps / content-store targets