🤖 fix: stop background status fetch from converting repos into partial clones

[ibetitsmike]

Summary

The background git-status fetch (GIT_FETCH_SCRIPT) fetched with --filter=blob:none, which permanently converts the repo into a promisor/partial clone. Every commit fetched by that loop then lacks blobs, so a later git worktree add (workspace creation) must lazy-fetch blobs from upstream mid-checkout — and any transient network drop aborts creation with fatal: could not fetch <oid> from promisor remote. This PR switches the status fetch to --no-filter, adds an explicit one-time heal (backfill via git fetch --refetch, then unset the promisor config) for already-poisoned repos, strips promisor config keys in the SSH warm path, and classifies promisor lazy-fetch failures as repairable missing-object checkout failures.

Background

Root-cause chain:

1. 🤖 refactor: clean git status system and optimize performance #108 added --filter=blob:none to the background status fetch for speed. Git registers remote.origin.promisor=true + remote.origin.partialclonefilter=blob:none on the first filtered fetch, and the persisted filter then applies to every subsequent plain git fetch in that repo.
2. On SSH runtimes the shared base repo accumulates blobless commits from this loop. When a new workspace is created, git worktree add checks out one of those commits and lazy-fetches the missing blobs from the remote mid-checkout.
3. A transient SSH/network drop during that lazy fetch fails workspace creation outright with could not fetch <oid> from promisor remote — a failure mode that did not route to the existing repair-from-local/bundle-ref fallback.

#3290 already stripped promisor keys in ensureBaseRepo(), but that never stuck: sibling worktrees running the status fetch re-poisoned the shared base repo immediately, and the warm (fused preamble) path never stripped the keys at all.

Implementation

• GIT_FETCH_SCRIPT (src/common/utils/git/gitStatus.ts): --filter=blob:none → --no-filter stops poisoning healthy repos, but it does not remove persisted promisor config or backfill blobs that earlier filtered fetches omitted. So the script now also runs an explicit one-time heal, placed before any ls-remote/primary-branch gating (a stale origin/HEAD after an upstream default-branch rename would otherwise skip it forever): when remote.origin.partialclonefilter is exactly the blob:none this script used to write, it backfills in two stages and unsets the promisor config keys only after verifying completeness. Stage 1 enumerates missing objects (git rev-list --objects --missing=print --all --reflog) and batch-fetches exactly those OIDs via git fetch --stdin — downloads only the gaps, works on git ≥ 2.30 (e.g. Ubuntu 22.04's 2.34, which lacks --refetch), and uses the same protocol-v2 OID-want capability the repo's lazy fetch already depends on. Stage 2, feature-detected and only if gaps remain, runs git fetch --refetch (git ≥ 2.36) for servers that refuse OID wants. The completeness check includes --reflog because an upstream force-push strands the displaced blobless commit in the remote-tracking reflog — unsetting the config would break git reset --hard origin/main@{1} recovery. If gaps remain after both stages: with a reachable server, the promisor config is kept (preserving the lazy-fetch fallback) and a xum.promisorHealIncompleteAt marker throttles retries to daily; if every fetch failed (offline/auth), the mkdir lock is left in place so its 1h staleness window paces retries. The lock (timestamp file inside — portable, no find -mmin/stat mtime probing) also prevents sibling worktrees from concurrent backfills.
• Fetch timeout (src/browser/stores/GitStatusStore.ts): background fetch timeout raised 30s → 300s, since unfiltered transfers (and the one-time refetch) can be much larger than the old blob-filtered ones; killing a slow-but-progressing fetch wastes the transfer and leaves ahead/behind stale behind retry backoff.
• SSH warm path (src/node/runtime/SSHRuntime.ts): the fused preamble now heals the shared base repo the same way — when it carries the blob:none partial-clone config, it batch-fetches exactly the missing OIDs (after the origin URL is configured) and unsets BASE_REPO_PROMISOR_CONFIG_KEYS only when enumeration proves the object store complete. Stripping unconditionally would remove the lazy-fetch fallback while blobs are still missing and let git worktree add silently fall back to the stale bundle ref despite origin being reachable.
• Enumeration robustness: rev-list --missing=print can exit 128 with no output when a local ref names a missing commit; both heal sites treat enumeration failure as "not proven complete" (sentinel value) so the promisor config is never unset on bad data.
• Failure classification: could not fetch … from promisor remote is now matched by isMissingObjectCheckoutFailure and by the warm-path wt_reason=missing-objects case, so it routes to the existing repair-from-local/bundle-ref fallback instead of failing workspace creation.

Validation

• Sandbox e2e repro (git 2.49): confirmed a single --filter=blob:none fetch converts a plain clone into a promisor clone and that subsequent plain fetches stay blobless; confirmed --no-filter avoids conversion.
• New behavioral unit tests (gitStatus.fetch.test.ts): (1) poisons a real repo with a filtered fetch, then asserts the script heals it even on the LOCAL_SHA = REMOTE_SHA early-exit path — promisor config removed, previously missing blobs backfilled, heal one-shot on a second run; (2) pins a blobless commit with a local branch, deletes its branch upstream + GCs the server, and asserts the heal keeps the promisor config, sets the daily-retry marker, and skips the backfill on the next run; (3) force-pushes over a bloblessly-fetched commit + GCs the server, and asserts the reflog-aware completeness check keeps the promisor config even though no plain ref reaches the displaced commit anymore; (4) points a local ref at a nonexistent object so enumeration itself fails, and asserts the heal refuses to unset the promisor config.
• Extended the SSH runtime warm-heal integration test to assert promisor keys are stripped from the shared base repo (tests/runtime/runtime.test.ts); Docker-only, runs in CI.

Risks

Touches the background fetch and workspace-creation hot paths. Severity is low: --no-filter is a strict superset fetch (more data, same semantics), and the promisor-key strip is best-effort (|| true). One-time costs per poisoned repo: the heal's stage-1 backfill downloads exactly the missing blobs; only when the server refuses OID wants does the stage-2 --refetch download the full object set (roughly a clone-sized transfer, temporarily duplicating pack data on disk until git's auto-gc repacks). Known collateral: a repo the user intentionally created as a blobless (--filter=blob:none) clone is indistinguishable from one we poisoned and will be converted to a full clone once; other filters (e.g. tree:0) are left untouched. Repos on git 2.30–2.35 heal through the OID backfill stage; older than that, heal attempts fail and keep today's behavior. Repos with blobless objects that upstream no longer serves (deleted/force-pushed + GC'd) keep their promisor config permanently — also today's behavior — at the cost of one backfill attempt per day.

---

Generated with xum • Model: coder:anthropic-wif/claude-mythos-5 • Thinking: xhigh • Cost: $249.39

[ibetitsmike]

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6210c403b3

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ibetitsmike]

Addressed both review findings in 71eeb39:

P1 (persisted promisor state / no backfill / early exit): Correct — --no-filter alone neither removes the persisted config nor backfills previously omitted blobs. GIT_FETCH_SCRIPT now runs an explicit one-time heal before the LOCAL_SHA = REMOTE_SHA early exit: when remote.origin.partialclonefilter is exactly the blob:none this script used to write, it runs git fetch --refetch (negotiates as if the repo had nothing, so previously filtered-out blobs are re-sent) and unsets the promisor config keys only after the refetch succeeds — stripping first would leave missing blobs with no lazy-fetch fallback. A mkdir lock in the shared git dir prevents sibling worktrees from concurrent refetches; stale locks expire after 1h, which also rate-limits retries. Added a behavioral test that poisons a real repo, hits the early-exit path, and asserts config removal + blob backfill + one-shot behavior. Known collateral (documented in the PR body): intentional blob:none clones are indistinguishable from poisoned ones and get converted to full clones once; other filters (e.g. tree:0) are left untouched; git < 2.36 (no --refetch) skips healing.

P2 (30s timeout too small for unfiltered transfers): Raised the background fetch timeout in GitStatusStore.executeWorkspaceFetch() from 30s to 300s (FETCH_TIMEOUT_SECS), budgeting for full-object transfers and the one-time refetch; killing a slow-but-progressing fetch wastes the whole transfer and leaves ahead/behind stale behind retry backoff.

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 71eeb39293

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ibetitsmike]

Addressed all three round-2 findings in d452138:

P1 (refetch completeness): Correct — --refetch only re-sends objects reachable from the remote's current refs, so blobs pinned solely by local refs into upstream-deleted branches can stay missing. The heal now verifies completeness with git rev-list --objects --missing=print --all after the refetch and unsets the promisor config only when nothing is missing. When gaps remain it keeps the config (preserving the lazy-fetch fallback, i.e. today's behavior) and sets a xum.promisorHealIncompleteAt marker that throttles further refetch attempts to daily. Added a behavioral test: pins a blobless commit with a local branch, deletes the branch upstream, asserts config retention + daily-retry marker + no refetch on the next run.

P2 (heal unreachable behind ls-remote gating): The heal now runs at the top of the script, before primary-branch detection and the ls-remote early exits, so a stale origin/HEAD (renamed/deleted upstream default branch) can no longer leave a repo poisoned forever.

P3 (find portability): Dropped find entirely. Staleness now comes from a started timestamp file written inside the mkdir lock and compared with date +%s arithmetic — portable across BSD/GNU/busybox userlands. A failed refetch now intentionally keeps the lock so retries wait out the 1h staleness window instead of re-fetching every poll (this also fixes an inaccuracy in the previous revision, which claimed that throttling but removed the lock unconditionally).

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d452138228

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

💡 Codex Security Review

Here are some automated security review suggestions for this pull request.

Reviewed commit: d452138228

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

• You comment "@codex security review"
• A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

[ibetitsmike]

Addressed both round-3 findings in 6216de3:

P2 (git < 2.36 support): The heal now backfills in two stages. Stage 1 enumerates missing objects and batch-fetches exactly those OIDs via git fetch --stdin (git ≥ 2.30, so Ubuntu 22.04's 2.34 heals) — this is effectively a batched lazy fetch, riding the same protocol-v2 OID-want capability the poisoned repo's lazy fetch already depends on, and it only downloads the gaps instead of a clone-sized refetch. Stage 2 falls back to --refetch only when gaps remain and only after feature-detecting the flag (git fetch -h), covering servers that refuse OID wants. Hosts older than 2.30 keep today's behavior.

P2 (reflog reachability): The completeness enumeration (used both to build the stage-1 OID list and to decide whether unsetting is safe) now includes --reflog, so a commit displaced by an upstream force-push — reachable only via the remote-tracking reflog — keeps the lazy-fetch fallback. Added a behavioral test: force-pushes over a bloblessly-fetched commit, GCs the server, and asserts the heal moves origin/main to the replacement history while keeping the promisor config because the displaced commit's blob is reflog-reachable and unrecoverable.

@codex review

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 6216de39de

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[ibetitsmike]

Addressed round-4 findings in 79b6abd:

P2 (enumeration failure treated as complete): Fixed. xum_missing_objects now captures rev-list's exit status via command substitution (POSIX pipelines discard it) and prints an enumeration-failed sentinel on failure. Stage 1 skips the sentinel (no OID list to fetch), stage 2 still runs (a full refetch can restore a missing commit object itself), and the unset branch requires an empty result, so the promisor config is never removed on bad data. Behavioral test added: points a local ref at a nonexistent object (rev-list exits 128 with no output) and asserts the heal refuses to unset.

P2 (SSH warm path strips before backfilling): Fixed. The warm-path strip now mirrors the fetch script's heal: gated on partialclonefilter=blob:none, it batch-fetches exactly the missing OIDs (moved after originPreamble so the freshly configured origin URL is usable) and unsets the keys only when enumeration proves the object store complete — enumeration failure counts as incomplete. When the keys stay, lazy fetch plus the promisor-remote failure classification cover worktree materialization as before. The non-poisoned warm path adds a single git config --get probe, keeping the hot path cheap.

Security P2 (bound unfiltered background fetches): Responding with rationale rather than a code change, as the suggested quota would reintroduce the bug class this PR fixes:

1. The passive fetch downloads exactly what the user's own git fetch/git pull of that tracked remote would download, and what workspace creation (git worktree add of those commits) requires anyway. Blob-metadata-only fetching is precisely the --filter=blob:none strategy that poisoned repos into partial clones and broke workspace creation (the P0 this PR fixes); deferring blobs does not avoid the download, it only moves it to checkout time with worse failure modes.
2. Git provides no client-side byte/disk quota for fetch, and an aborted transfer wastes the bytes already sent while keeping status stale behind retry backoff — repeated aborted attempts cost more bandwidth than one completed transfer.
3. Threat-model-wise, an attacker who can push large objects to the primary branch of a remote you track can inflate any clone of that repo, xum or not; that capability already implies full control over the repo contents the workspace builds and executes.
4. Each attempt is bounded by the 300s bash timeout and exponential retry backoff, which caps sustained passive throughput.

Happy to revisit if a concrete bounded strategy exists that doesn't resurrect the partial-clone poisoning.

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Nice work!

Reviewed commit: 79b6abd364

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".