feat(working-memory): measure working memory in admin calibration runs

[Pfannkuchensack]

Stacked on #9896. Until that merges, the diff also shows its commit; this PR's change is 470db713f5.

Summary

Working-memory estimates can now be checked against what an operation really uses, on the machine that runs it. This PR only records measurements; no estimate changes.

• Probe (invokeai/backend/working_memory/probe.py)
  • During a calibration run, every model_on_device(working_memory=...) window records its demand next to its estimate.
  • Demand is max(own-process driver delta, torch reserved-peak delta, allocated-peak delta), measured from a released baseline.
  • The driver delta comes from D3DKMT on Windows (NVIDIA and AMD), NVML or fdinfo on Linux, and the driver counter on MPS.
  • It locks with headroom, so a too-low estimate cannot spill while it is being measured.
  • It handles co-locked models and nested windows.
  • It marks a window invalid on a foreign lock on the same device, OOM, allocator retries, persistent spill to shared memory, exceptions, or probe errors.
  • Partial residency keeps a window but excludes it from fitting.
  • Without a calibration scope the cost is one ContextVar read per estimated lock.
• Runs (invokeai/app/services/working_memory_calibration/)
  • An admin enqueues a calibration batch through POST /api/v2/models/i/{key}/working_memory/calibrate.
  • Its items carry a reserved origin that enqueue_batch refuses for everyone.
  • At session start the runner re-admits each item: active admin owner, exactly one GPU generation device, run present for this device class.
  • It opens the probe scope per node and stores the windows in a JSON file per device class under databases/working_memory/.
  • The file is written atomically and under a file lock; files it cannot read are kept and never overwritten.
  • Status, reset and export routes are admin-only.

Related Issues / Discussions

Builds on #9896.

QA Instructions

• Tests: affected backend and app suites (working_memory, util, model_manager/load, the calibration service, runner and router tests, session queue/processor): 2079 passed, 115 skipped. Each new file also passes alone.
  • Mutation checks: co-lock offset, nested suspend, foreign lock at open, RAM-pin and unlock bookkeeping, admission refusals, retry lineage and the store's OS lock are each caught.
• Hardware: pytest -m slow tests/backend/working_memory/test_probe.py passed on an RTX 4090 (CUDA 13.0) and an RX 9060 XT (ROCm 10, Windows): the window is valid and covers the reserved growth.
• End to end on the RTX 4090 (server from this branch, cudaMallocAsync):
  • A FLUX VAE encode→decode at 1024² through the calibrate endpoint completed and recorded two valid, verified windows.
    • flux_vae_encode: reserved 2200 MiB, measured 1840 MiB.
    • flux_vae_decode: reserved 4400 MiB, measured 2912 MiB.
  • The status call did not change the process's VRAM.
• API: regenerated openapi.json/schema.ts; pnpm lint and pnpm test in invokeai/frontend/api pass.

Review

Resolved findings

• Runs could report "running" forever after Clear Queue, a crash or a prune.
  • Fix: the outcome is recorded in finally, and an item that is gone counts as canceled.
• Probe errors could leave windows valid with corrupted figures.
  • Fix: every hook, observer or close failure marks the affected windows.
• A foreign lock could slip in between the lock count and window registration at open.
  • Fix: the window is registered before the count.
• calibrate was not safe to retry after a lost response.
  • Fix: it takes an idempotency key.
• Windows os.replace failed transiently while another process held the file.
  • Fix: bounded retry.
• The ROCm attention and conv3d switches were missing from the environment fingerprint.
• Status available and the declared error codes now match what the routes do.

Remaining limits

• The Linux NVML, Linux fdinfo, MPS and XPU readers follow vendor documentation, untested on hardware. They are marked verified=false.
• Direct torch.cuda.empty_cache() calls and background offloads bypass the segment boundary. The torch peaks still bound the demand.
• Multi-GPU calibration is refused by design.

Compatibility / Rollout

• New admin-only routes; regenerated API artifacts.
• No database migration. Calibration data lives in <db_dir>/working_memory/<device_class>.json.
• Nothing reads it for estimates yet.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}