imagecache: eviction engine — root set, EvictUnused, startup orphan recovery

[igooch]

Phase 2 of #463 (image-cache GC + observability): this is the eviction engine itself — the root set, the eviction pass, and startup orphan recovery — assembled from the foundations merged in #650 (sizes/last-use), #656 (spec digest), #658 (retirement primitives), and #717 (record-first pull). What remains after this is the watermark loop + flags that turn it on, then metrics and e2e.

Inert in production: nothing calls EvictUnused yet — the watermark GC loop and its flags come in the next PR. The only behavior New gains is the startup orphan scan.

What

• Store.InUse() — the root set: scans bundle overlay specs under the actors dir (opt-in via WithActorsDir) for image digests, layer dirs, and exact layer-set signatures. The exact-set rule roots the multi-arch twin record and digestless (pre-imageDigest) specs' records without making every subset image unevictable.
• Store.EvictUnused(ctx, targetBytes, dryRun) — LRU eviction of unprotected images and the layers their removal leaves unreferenced. Min-age vetoes; per-victim re-checks under hitMu held exclusive against the cache-hit touch; two-phase deletion via retireLayer, with the slow RemoveAll outside all locks. If any layer of a deleted record must be kept (still referenced, rooted, fresh, or failed to retire), the record is atomically restored from bytes captured at listing time — a kept layer is never left unreachable.
• Store.RecoverOrphans, called once from New — reclaims layers no record references. Startup is the one moment this scan is race-free (no pull can be in flight); a failure never fails New.
• Everything fails toward retention. Both passes skip entirely when the record enumeration is incomplete (unreadable/undecodable record): refcounts from partial data would make referenced layers look like garbage. The skip is an ERROR naming the records; an undecodable record is never itself evicted.
• Per-item log lines are Debug behind an Enabled gate (level control via serverboot: dynamic --log-level flag for the server binaries #677); pass summaries, evictions, and restores are Info.

Notes for reviewers

• The gc tests reuse the helpers merged with imagecache: record layer sizes at unpack and image last-use on cache hit #650 (backdate, mustEnsure) and imagecache: write the image record before unpacking layers #717's gated-registry harness — no duplicated symbols. The slow-pull gated test gains the eviction-veto assertion deferred from imagecache: write the image record before unpacking layers #717.
• Dry-run mutates nothing, including the size-file backfill (layerSize split into read/walk/backfill parts; dry-run uses the read-only path).
• A Store built without WithActorsDir has an empty root set (deliberate for tools/tests); the wiring PR must pass it in atelet.

Testing

• go test -race ./internal/imagecache/: every veto, shared-layer survival, both restore-on-keep shapes, orphan reclaimed at startup and ignored by the periodic pass, enumeration gate (unreadable + undecodable), dry-run purity, 25-iteration ensure-vs-evict race loop.
• Kind, branch build deployed: full gVisor e2e suite green; a planted backdated orphan layer on the node was reclaimed at atelet restart with the expected log lines.

[igooch]

For reviewers:

How a pass decides, and specifically how running actors are protected. Protection is based on use, not age: an image is protected while any actor on this node references it (i.e., while that actor's bundle spec exists on disk). mtime only orders images that nothing references.

flowchart TD
    subgraph inuse ["Store.InUse() — LOOP over every container bundle of every actor on this node"]
        SPEC["read actors/<uid>/bundles/<container>/rootfs-overlay.json"]
        SPEC --> MAPS["ImageDigests: spec.imageDigest<br/>LayerHexes: each layer dir the spec names<br/>LayerSets: exact layer-set signature"]
    end

    inuse --> listing

    subgraph listing ["Store.listEviction() — LOOP over every image record (image-cache/manifests/sha256/*.json)"]
        REC["read + decode record;<br/>refcount++ each of its layers<br/>(rooted and fresh records count too)"]
        REC --> G1{"this record rooted?<br/>ImageDigests[digest] OR<br/>LayerSets[signature(its diffIDs)]"}
        G1 -- "yes — running actors exit here;<br/>mtime never consulted" --> NXT1["next record ↩"]
        G1 -- "no" --> G2{"record mtime younger<br/>than min-age?"}
        G2 -- "yes" --> NXT1
        G2 -- "no" --> CAND["→ candidate list"]
    end

    listing --> COMPLETE{"was every record read + decoded?"}
    COMPLETE -- "no" --> SKIP["⛔ skip entire pass (ERROR) —<br/>partial refcounts would retire layers<br/>unread records still name"]
    COMPLETE -- "yes" --> SORT["sort candidates by mtime, oldest first<br/>(order among evictables — never permission)"]

    SORT --> evict

    subgraph evict ["Store.EvictUnused() — LOOP over candidates until targetBytes freed"]
        RM["removeStaleRecord: re-check under hitMu<br/>(gone? fresh again?) then delete record"]
        RM --> layers

        subgraph layers ["retireCandidateLayers — LOOP over the deleted record's layers"]
            L1{"refcount-- still &gt; 0?<br/>(another image uses it)"}
            L1 -- "yes" --> KEEP["layer kept"]
            L1 -- "no" --> L2{"LayerHexes[hex]?<br/>(a bundle spec names this layer)"}
            L2 -- "yes" --> KEEP
            L2 -- "no" --> L3{"layer dir mtime fresh?"}
            L3 -- "yes" --> KEEP
            L3 -- "no" --> MARK["retireLayer: rename → .rm-* (the mark)"]
        end

        layers --> KEPT{"anything kept?"}
        KEPT -- "yes" --> RESTORE["restoreRecord — record rewritten,<br/>image NOT evicted; next candidate ↩"]
        KEPT -- "no" --> EVICTED["image evicted ✓; next candidate ↩"]
    end

    evict --> RRD["removeRetiredDirs — slow RemoveAll of .rm-* dirs,<br/>outside all locks; crash debris swept at next startup"]

Loading

• A referenced image never reaches the mtime sort — its layers can be months old with
  zero risk; protection ends only when its bundle is removed (post-checkpoint), never by aging.
• "Marked for deletion" = renamed to .rm-*, possible only for a layer no record
  references and no bundle names.
• Every failure path lands on retention (an unreadable bundle spec is the one WARN-and-root-nothing exception).