Unknown-provider PR polling still floods server logs; closed PR #5831 had a tested fix

[mikesulsenti]

What happened

A headless Linux T3 service kept running, but its service log became dominated by repeated source-control warnings. In one ~21-hour boot-service.log, 32,931 warnings were:

PR lookup failed; keeping last known PR state.
provider: unknown
providerOperation: listChangeRequests
errorDetail: No unknown source control provider is registered.

The same condition also produced 1,372 automatic thread settlement skipped warnings with a full cause/stack. Real warnings became difficult to find among the repeated entries.

The affected project uses a self-hosted GitLab remote over SSH with a custom hostname that does not itself identify the hosting provider. T3 resolved the provider as unknown.

The affected path is still present in stable 0.0.38, nightlies 0.0.39-nightly.20260902.1252 and 0.0.39-nightly.20260902.1253, and public main at b520120cf169ce63a5606447a292451e97622c9a.

Diagnosis

This does not appear to be a missing-backoff bug in the provider request itself.

This behavior had a direct proposed fix in #5831, fix(server): skip PR lookup for unknown providers. That PR introduced a typed ProviderUnknown result, avoided calling listChangeRequests for unresolved providers, preserved the last-known PR, and retained retry/backoff so provider refinement could recover. It included focused tests but was closed without merging on Aug 29. Current releases and current main still use the pre-fix error path.

The affected releases already implement per-PR-cache-key exponential failure backoff: 20 seconds, 40 seconds, 80 seconds, and so on, capped at 15 minutes. However, lookupStatusPr calls Cache.get(...) and attaches PR lookup failed; keeping last known PR state. in a catch outside that cache read. A cached failure therefore appears to be logged again for every caller even when the cache prevents a new provider request.

Separately, ThreadSettlementReactor runs every minute. When the same lookup failure reaches it, it emits automatic thread settlement skipped with Cause.pretty(cause). This creates a second repeated warning family for the same durable condition.

Relevant source, pinned to the current reviewed main commit:

• Failure TTL/backoff:

  t3code/apps/server/src/git/GitManager.ts

  Lines 123 to 143 in b520120

  	const PR_LOOKUP_CACHE_TTL = Duration.minutes(2);
  	const PR_LOOKUP_FAILURE_BASE_TTL = Duration.seconds(20);
  	const PR_LOOKUP_FAILURE_MAX_TTL = Duration.minutes(15);
  	const PR_LOOKUP_CACHE_CAPACITY = 2_048;
  	const isSourceControlProviderError = Schema.is(SourceControlProviderError);
  	/**
  	* How long a failed PR lookup is cached, given the number of consecutive
  	* failures for that branch.
  	*
  	* A hosting provider rejects a throttled request immediately, so caching every
  	* failure for a flat 20s made a rate-limited poller re-ask *faster* than a
  	* healthy one does (which waits PR_LOOKUP_CACHE_TTL), turning a transient 429
  	* into sustained pressure. Backing off per branch keeps the retry rate below
  	* the healthy rate once a branch has failed more than a couple of times.
  	*/
  	export function prLookupFailureTtl(consecutiveFailures: number): Duration.Duration {
  	const exponent = Math.max(0, consecutiveFailures - 1);
  	const backoffMs = Duration.toMillis(PR_LOOKUP_FAILURE_BASE_TTL) * Math.pow(2, exponent);
  	return Duration.min(Duration.millis(backoffMs), PR_LOOKUP_FAILURE_MAX_TTL);
  	}

• Backoff test:

  t3code/apps/server/src/git/GitManager.test.ts

  Lines 1503 to 1510 in b520120

  	// The point of the backoff: by the third retry a failing branch must not be
  	// asking more often than a healthy one, which refreshes every 2 minutes.
  	expect(Duration.toMillis(GitManager.prLookupFailureTtl(4))).toBeGreaterThan(120_000);
  	expect(Duration.toMillis(GitManager.prLookupFailureTtl(20))).toBe(900_000);
  	});
  	it.effect(
  	"status ignores unrelated fork PRs when the current branch tracks the same repository",

• Failure-aware cache:

  t3code/apps/server/src/git/GitManager.ts

  Lines 971 to 1044 in b520120

  	// Consecutive failures per cache key, so a branch that keeps failing waits
  	// longer before the next attempt. Cleared as soon as a lookup succeeds.
  	const prLookupFailureStreakByKey = new Map<string, number>();
  	const nextPrLookupFailureTtl = (key: string) => {
  	if (
  	!prLookupFailureStreakByKey.has(key) &&
  	prLookupFailureStreakByKey.size >= PR_LOOKUP_CACHE_CAPACITY
  	) {
  	const oldestKey = prLookupFailureStreakByKey.keys().next().value;
  	if (oldestKey !== undefined) {
  	prLookupFailureStreakByKey.delete(oldestKey);
  	}
  	}
  	const streak = (prLookupFailureStreakByKey.get(key) ?? 0) + 1;
  	prLookupFailureStreakByKey.set(key, streak);
  	return prLookupFailureTtl(streak);
  	};
  	const prLookupCache = yield* Cache.makeWith(
  	(key: string) => {
  	const [
  	cwd = "",
  	branch = "",
  	upstreamRef = "",
  	defaultBranch = "",
  	branchExists = "1",
  	remoteName = "",
  	] = key.split("\u0000");
  	const details = {
  	branch,
  	upstreamRef: upstreamRef.length > 0 ? upstreamRef : null,
  	defaultBranch: defaultBranch.length > 0 ? defaultBranch : null,
  	localBranchExists: branchExists !== "0",
  	...(remoteName.length > 0 ? { remoteName } : {}),
  	};
  	return Effect.gen(function* () {
  	const headContext = yield* resolveBranchHeadContext(cwd, details);
  	const upstreamHeadIsDefault =
  	headContext.headBranch === details.defaultBranch ||
  	(details.defaultBranch === null &&
  	(headContext.headBranch === "main" || headContext.headBranch === "master"));
  	// `git worktree add -b feature origin/main` makes the new local branch
  	// track origin/main. That upstream is the branch's base, not its
  	// published PR head. Looking up PRs for it can attach an old reverse
  	// merge from main and auto-settle an unrelated feature thread.
  	if (
  	headContext.headBranch !== details.branch &&
  	upstreamHeadIsDefault &&
  	!headContext.isCrossRepository
  	) {
  	return { latest: null, headContext };
  	}
  	// Only skip when the branch is untracked as well: anything carrying an
  	// upstream keeps the old behaviour.
  	if (
  	details.localBranchExists &&
  	details.upstreamRef === null &&
  	(yield* isUnpublishedBranch(cwd, headContext))
  	) {
  	return { latest: null, headContext };
  	}
  	const latest = yield* findLatestPrForHeadContext(cwd, headContext);
  	return { latest, headContext };
  	});
  	},
  	{
  	capacity: PR_LOOKUP_CACHE_CAPACITY,
  	timeToLive: (exit, key) => {
  	if (Exit.isSuccess(exit)) {
  	prLookupFailureStreakByKey.delete(key);
  	return PR_LOOKUP_CACHE_TTL;
  	}
  	return nextPrLookupFailureTtl(key);
  	},
  	},

• Per-caller warning after Cache.get:

  t3code/apps/server/src/git/GitManager.ts

  Lines 1104 to 1167 in b520120

  	const lookupStatusPr = Effect.fn("lookupStatusPr")(function* (
  	cwd: string,
  	details: {
  	branch: string;
  	upstreamRef: string | null;
  	defaultBranch: string | null;
  	isDefaultBranch: boolean;
  	},
  	) {
  	// Keyed by (cwd, branch) only: the upstream ref changing (e.g. a first
  	// `push -u`) must not orphan the fallback value for the same branch.
  	const branchKey = `${cwd}\u0000${details.branch}`;
  	return yield* Cache.get(prLookupCache, prLookupCacheKey(cwd, details)).pipe(
  	Effect.map(({ latest, headContext }) => {
  	if (!latest) return { pr: null, headContext };
  	// On the default branch, only surface open PRs.
  	// Merged/closed matches are usually reverse-merge history, not the thread's PR context.
  	if (details.isDefaultBranch && latest.state !== "open") {
  	return { pr: null, headContext };
  	}
  	return { pr: toStatusPr(latest), headContext };
  	}),
  	Effect.tap(({ pr, headContext }) =>
  	Effect.sync(() =>
  	rememberLastKnownPr(branchKey, {
  	pr,
  	upstreamRef: details.upstreamRef,
  	headBranch: headContext.headBranch,
  	remoteName: headContext.remoteName,
  	headRemoteUrlKey: headContext.headRemoteUrlKey,
  	}),
  	),
  	),
  	Effect.map(({ pr }) => pr),
  	Effect.catch((error) =>
  	Effect.logWarning("PR lookup failed; keeping last known PR state.").pipe(
  	Effect.annotateLogs({
  	operation: "lookupStatusPr",
  	branch: details.branch,
  	errorTag:
  	typeof error === "object" && error !== null && "_tag" in error
  	? String(error._tag)
  	: typeof error,
  	...(isSourceControlProviderError(error)
  	? {
  	provider: error.provider,
  	providerOperation: error.operation,
  	providerCommand: error.command ?? "unknown",
  	errorDetail: error.detail,
  	}
  	: {}),
  	}),
  	Effect.andThen(resolveBranchHeadContext(cwd, details)),
  	Effect.map((headContext) =>
  	resolveLastKnownPr(branchKey, {
  	upstreamRef: details.upstreamRef,
  	headBranch: headContext.headBranch,
  	remoteName: headContext.remoteName,
  	headRemoteUrlKey: headContext.headRemoteUrlKey,
  	}),
  	),
  	),
  	),
  	);

• One-minute settlement sweep and warning:

  t3code/apps/server/src/orchestration/ThreadSettlementReactor.ts

  Lines 41 to 165 in b520120

  	const sweep = Effect.fn("ThreadSettlementReactor.sweep")(function* () {
  	const snapshot = yield* snapshots.getShellSnapshot();
  	const now = DateTime.formatIso(yield* DateTime.now);
  	const projects = new Map(snapshot.projects.map((project) => [project.id, project]));
  	const candidates = snapshot.threads.filter((thread) => isAutoSettlementCandidate(thread, now));
  	const lookupKey = (thread: (typeof candidates)[number]) => {
  	if (thread.linkedPullRequest != null) {
  	return JSON.stringify([
  	"linked",
  	thread.linkedPullRequest.projectId,
  	thread.linkedPullRequest.repository,
  	thread.linkedPullRequest.number,
  	]);
  	}
  	if (thread.branch === null) return JSON.stringify(["none", thread.id]);
  	const project = projects.get(thread.projectId);
  	return JSON.stringify(
  	project === undefined
  	? ["missing-project", thread.id]
  	: ["branch", project.workspaceRoot, thread.branch],
  	);
  	};
  	const groups = Map.groupBy(candidates, lookupKey);
  	const pullRequestFor = Effect.fn("ThreadSettlementReactor.pullRequestFor")(function* (
  	thread: (typeof candidates)[number],
  	) {
  	if (thread.linkedPullRequest != null) {
  	if (!projects.has(thread.linkedPullRequest.projectId)) {
  	return yield* Effect.die(new Error("linked pull request project not found"));
  	}
  	const detail = yield* pullRequests.detail({
  	projectId: thread.linkedPullRequest.projectId,
  	repository: thread.linkedPullRequest.repository,
  	number: thread.linkedPullRequest.number,
  	});
  	return { state: detail.state, updatedAt: detail.updatedAt } satisfies SettlementPullRequest;
  	}
  	if (thread.branch === null) return null;
  	const project = projects.get(thread.projectId);
  	if (project === undefined) {
  	return yield* Effect.die(new Error("thread project not found"));
  	}
  	return yield* git.branchPullRequest({ cwd: project.workspaceRoot, branch: thread.branch });
  	});
  	yield* Effect.forEach(
  	groups.values(),
  	(group) =>
  	Effect.gen(function* () {
  	const pullRequest = yield* pullRequestFor(group[0]!);
  	yield* Effect.forEach(
  	group,
  	(thread) =>
  	Effect.gen(function* () {
  	const settings = yield* settingsService.getSettings;
  	const decisionNow = DateTime.formatIso(yield* DateTime.now);
  	if (
  	!shouldAutoSettleThread({
  	thread,
  	pullRequest,
  	now: decisionNow,
  	autoSettleAfterDays: settings.sidebarAutoSettleAfterDays,
  	autoSettleOnMerge: settings.sidebarAutoSettleOnMerge,
  	})
  	) {
  	return;
  	}
  	const uuid = yield* crypto.randomUUIDv4;
  	yield* engine.dispatch({
  	type: "thread.auto-settle",
  	commandId: CommandId.make(`server:auto-settle:${thread.id}:${uuid}`),
  	threadId: thread.id,
  	snapshotSequence: snapshot.snapshotSequence,
  	});
  	}).pipe(
  	Effect.catchCause((cause) =>
  	Cause.hasInterruptsOnly(cause)
  	? Effect.failCause(cause)
  	: Effect.logWarning("automatic thread settlement skipped", {
  	threadId: thread.id,
  	cause: Cause.pretty(cause),
  	}),
  	),
  	),
  	{ discard: true },
  	);
  	}).pipe(
  	Effect.catchCause((cause) =>
  	Cause.hasInterruptsOnly(cause)
  	? Effect.failCause(cause)
  	: Effect.logWarning("automatic thread settlement skipped", {
  	threadIds: group.map((thread) => thread.id),
  	cause: Cause.pretty(cause),
  	}),
  	),
  	),
  	{ concurrency: 8, discard: true },
  	);
  	});
  	const worker = yield* makeDrainableWorker(() =>
  	sweep().pipe(
  	Effect.catchCause((cause) =>
  	Cause.hasInterruptsOnly(cause)
  	? Effect.failCause(cause)
  	: Effect.logWarning("automatic thread settlement sweep failed", {
  	cause: Cause.pretty(cause),
  	}),
  	),
  	),
  	);
  	const start: ThreadSettlementReactor["Service"]["start"] = Effect.fn(
  	"ThreadSettlementReactor.start",
  	)(function* () {
  	const settingsChanges = yield* settingsService.subscribeChanges;
  	const initialSettings = yield* settingsService.getSettings.pipe(Effect.orDie);
  	let lastAfterDays = initialSettings.sidebarAutoSettleAfterDays;
  	let lastOnMerge = initialSettings.sidebarAutoSettleOnMerge;
  	yield* forkParked(
  	Effect.gen(function* () {
  	yield* worker.enqueue(undefined);
  	yield* worker.drain;
  	}).pipe(Effect.repeat(Schedule.spaced("1 minute")), Effect.asVoid),

• Closed unmerged fix: fix(server): skip PR lookup for unknown providers #5831

An unknown provider may be expected for a neutral custom hostname when no source-control CLI reports authentication for that exact host. Regardless of why detection remains unknown, background status and settlement work should treat that durable unsupported-provider state cheaply and should not emit tens of thousands of duplicate warnings.

Possible fix directions:

• Revive or adapt fix(server): skip PR lookup for unknown providers #5831's tested ProviderUnknown outcome so background PR lookup does not treat an unresolved provider as an operational failure.
• Emit the PR-lookup warning only when the cache loader performs a real retry, rather than whenever a caller receives a cached failure.
• Treat provider: unknown as a typed unsupported/no-PR result for background polling, while preserving an actionable error for explicit user operations such as creating a PR.
• Deduplicate or rate-limit automatic thread settlement skipped by project/branch/cause.

Steps to reproduce

1. Configure a repository remote with a neutral self-hosted hostname, for example git@scm.example:group/repo.git, and ensure no provider CLI reports an authenticated provider for that exact host. T3 should resolve the provider as unknown.
2. Run t3 serve and connect a client so VCS status is polled.
3. Have one or more active, unarchived, unsettled threads with branches in that project; leave automatic settlement enabled.
4. Leave the service running for at least ten minutes.
5. Count PR lookup failed; keeping last known PR state. and automatic thread settlement skipped in the service log.

Expected: an unsupported provider is skipped without repeated background warnings, or warnings occur only when the existing backoff performs a real retry.

Actual: repeated callers log the same cached SourceControlProviderError, and the one-minute settlement sweep adds another repeated full-cause warning.

Version

0.0.38-nightly.20260901.1250

Environment

Linux x64 (CachyOS, kernel 7.2.2), Node 26, t3 serve under a systemd user service

Evidence

Aggregate from one ~21-hour boot-service.log (mixed 0.0.33/nightly interval):

32,931  WARN  PR lookup failed; keeping last known PR state.
 1,372  WARN  automatic thread settlement skipped

Representative nightly entry:

[23:55:39.535] WARN: automatic thread settlement skipped
  threadIds: [ '<redacted>' ]
  cause: SourceControlProviderError: Source control provider unknown failed in
    listChangeRequests: No unknown source control provider is registered.
    at Object.listChangeRequests (.../t3/dist/bin.mjs:86505:34)
    at findLatestPrForHeadContext (.../t3/dist/bin.mjs:87300:20)
    at branchPullRequest (.../t3/dist/bin.mjs:87791:28)
    at ThreadSettlementReactor.pullRequestFor (.../t3/dist/bin.mjs:179752:31)
    at ThreadSettlementReactor.sweep (.../t3/dist/bin.mjs:179782:50)


The trace files rotated through about 105 MB during the same day, but this report does not claim that all or most of that volume came from these warnings without an event-type breakdown.

Related issues

• PR fix(server): skip PR lookup for unknown providers #5831 is the exact attempted behavioral fix: it made an unknown provider a typed non-error outcome, skipped listChangeRequests, retained provider-refinement retry/backoff, and preserved last-known PR state. It was closed without merging, so this report supplies current production evidence and asks that the fix be revived or adapted.
• PR fix(server): stop PR status lookups amplifying GitHub rate limits #5673 introduced the existing provider-request failure backoff. It reduces actual retry frequency but does not make an unresolved provider a non-error result and therefore does not prevent the repeated warning paths described here.
• Enterprise GitLab remotes with non-git SSH user (e.g. gitlab@) not detected — provider stays unknown #3648 has the exact No unknown source control provider is registered error, but its trigger was an SCP-style remote parser that only accepted the git@ user. That parser was fixed. This report is about repeated logging/caller behavior after a provider remains legitimately or otherwise unresolved.
• [Bug]: Source-control provider detection misclassifies unrelated hosts containing github, gitlab, or bitbucket #5932 fixed false-positive hostname classification. It is the inverse detection case and does not cover repeated background logging for unknown.
• [Bug]: Closed-PR threads auto-settle again after every completed turn #6417 and Auto-settlement marks old threads fresh and sends completion pushes on first launch #9057 concern settlement semantics and stale notifications, not cached provider-failure warning volume.
• PR fix(server): surface pull requests sooner without extra host requests #9125 may share more PR-cache entries between the sidebar and settlement paths, but it does not move/suppress the warning emitted when callers receive a cached failure.

Fix applied or workaround

No application files or state were modified.

If the custom host is GitLab, configuring and authenticating glab for the exact remote hostname may let T3 refine unknown to gitlab and avoid this particular error. That is an environment workaround, not a fix for repeated warning emission when a provider is unsupported.

Filed by

by Codex GPT-5.6 Sol reviewing a GLM-5.3-Flash report