Fix repeated wake delivery and desktop resume recovery#4726
Open
KyleAMathews wants to merge 122 commits into
Open
Fix repeated wake delivery and desktop resume recovery#4726KyleAMathews wants to merge 122 commits into
KyleAMathews wants to merge 122 commits into
Conversation
Foundation for context compaction: measure and surface how full the model's context window is, with no behavior change yet. - pi-adapter: capture cache-INCLUSIVE prompt size (input + cacheRead + cacheWrite) and the model context window per step. The existing input_tokens deliberately excludes cache reads for budget accounting, but cached tokens still occupy the window, so a fullness gauge needs the inclusive total. - persist context_input_tokens + context_window on the step row (optional/additive) via the outbound bridge. - token-accountant: single source of truth for usage ratio, severity level, and the compaction thresholds (85% background / 95% ceiling). - UI: ContextUsageIndicator renders "NN% used" in the composer footer from the same helper, coloured at the 85/95 thresholds. Observational only — nothing compacts yet; this validates the token accounting before later phases act on it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ndow in pi-adapter The bridge persistence and the usage-ratio helper were covered, but the adapter's cache-INCLUSIVE total (input + cacheRead + cacheWrite) — the accuracy premise of the context gauge — was not. Assert it equals 1350 where the uncached input_tokens is 150, and that context_window is emitted. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Once context usage reaches 25%, inject a <token_budget> notice into the model's messages stating remaining tokens + percent, so the model can pace itself. Recomputed each call from the latest step's persisted usage, so it is always current. - token-accountant: selectLatestContextUsage (latest step with usage), shouldSurfaceContextBudget (gate at 25%), formatContextBudgetNotice, and withContextBudgetNotice (inject just before the final message). - context-factory/runAgent: synthesize and inject before the model call. Synthesized (not persisted) on purpose: a self-superseding context row would leave load_context_history tombstones, which are misleading for an ephemeral budget hint. Synthesis stays deterministic (pure function of persisted steps) so replay/fork reproduce it, with no row churn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drive a real agent run with a capturing streamFn and assert the <token_budget> notice reaches the model's context, gated on a seeded step's usage: present at 80% usage (with correct "20k tokens (20%) remaining" wording), absent at 10%. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…se 2) Cap any single tool_result at ~10k tokens, replacing the body with a visible "[Output truncated: ...]" placeholder before the model call, so one giant output can't fill the context window on its own. Preserves toolCallId/isError so tool-call pairing stays valid. Mirrors Codex's per-message truncation. Truncation is always explicit, never silent. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Foundation for context compaction: a compaction checkpoint is a context_inserted row tagged attrs.kind="compaction". timelineMessages now treats the newest such checkpoint's order as a watermark — items before it are dropped (summarized away) and the checkpoint renders the summary in their place. No checkpoint -> watermark is -Infinity -> a strict no-op, so this is inert until the summarizer (next step) writes one. Adds compaction.ts with the checkpoint constants and the Codex summarization prompt + summary prefix (reused verbatim) for the summarization step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
summarizeMessages sends the full history + Codex's summarization prompt to the conversation's own model (a cheap small-window model would overflow a near-full context) and prefixes the result with Codex's summary preamble. The model call is injected via a `complete` seam (defaults to pi-ai completeSimple) so it is unit-testable without a network call or API key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Wires the compaction engine into the model-call path. Before a turn, if the last step left context at/over 95% of the window (and the history is actually large), summarize it, persist a compaction checkpoint (context_inserted kind=compaction), and send only the summary this turn — the current ask still arrives via runInput, and future turns reconstruct from the checkpoint watermark. Failure degrades gracefully (logs and proceeds uncompacted). - AgentConfig.summarizeComplete: model-call seam for the summarizer (defaults to the conversation model; injected by tests). - guard on estimated history size avoids re-compacting an already compacted (small) history while the last step's usage reads stale-high. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CONTEXT_USAGE_HARD_CEILING 0.95 -> 0.90, matching Codex's auto-compaction threshold. Drives the synchronous compaction trigger and the UI gauge's "critical" colour. Background start stays at 85%. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drives the real runtime end-to-end and makes a real Anthropic summarization call, asserting the returned summary carries the Codex prefix and retains key conversation facts. Skipped unless RUN_LIVE_COMPACTION=1 + LIVE_ANTHROPIC_API_KEY are set, so it never runs (or costs) in CI. Verified passing with claude-haiku-4-5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Its purpose (confirming a real model summarizes correctly through the compaction path) is done. Ongoing coverage lives in the deterministic stubbed tests (compaction-trigger, compaction-summarize, timeline-compaction); a live test pinning a model id + needing a paid key would only rot. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e/failed) Foundation for surfacing synchronous compaction in the UI. The trigger now writes a `running` checkpoint before summarizing and updates it to `complete` (with the summary) or `failed` after. Only a `complete` checkpoint acts as the reconstruction watermark, so an in-flight or crashed compaction never hides history; running/failed checkpoints are UI-only markers, skipped from the model context. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Shows a spinner + "Compacting context…" in the composer footer while a synchronous compaction is in flight, reading the latest compaction checkpoint row and showing it while attrs.status is "running" (clears on complete). Tells the user why the turn paused and that their next prompt is being queued. Mirrors the ContextUsageIndicator pattern. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ELECTRIC_AGENTS_COMPACT_CEILING (0..1, default 0.9) and ELECTRIC_AGENTS_COMPACT_MIN_TOKENS (default contextWindow/2) let the synchronous compaction path be exercised without filling a real window. RFC §12 tunables; default behavior unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces a completed compaction checkpoint as a collapsed, expandable "Context compacted" marker in the message history, at the point compaction happened. Adds a compaction custom timeline source (mirroring the comments source) reading compaction_summary context_inserted rows, a compaction row kind across the timeline dispatch, and a CompactionTimelineRow card (InlineEventCard, expand to view the summary). Running/failed checkpoints are filtered out — those are shown by the live composer indicator instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Compaction now runs before EVERY model step (not just between turns), so a single turn that balloons across many tool calls can no longer exhaust the context window before the turn ends. Wired through pi-agent-core's transformContext hook. - compaction-midturn.ts: createMidTurnCompactor — folds older messages into a summary, returns [summary, ...recent tail], caches the summary for the rest of the turn (re-summarizing chained off the prior summary only if the tail grows back over the ceiling). - pi-adapter: Codex-style token signal — anchor on the last step's REAL cache-inclusive usage + estimate only the trailing items appended since, vs the model's real context window (not an estimate of the whole history). transformContext + initialContextTokens options added. - context-factory: build the compactor and pass it to the adapter; removed the per-turn pre-sampling compaction block it replaces. Reuses the summarizer, checkpoint lifecycle (running/complete/failed), and UI. summarizeAgentMessages summarizes the AgentMessage[] the hook receives without re-converting. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The keepTail boundary could split a tool_call/tool_result pair — folding the assistant tool_use into the summary while keeping the matching tool_result in the tail. Anthropic rejects the orphaned tool_result (400 invalid_request_error: "tool_result must have a corresponding tool_use block in the previous message"). Advance the fold boundary past any leading tool-result messages so the kept tail starts on a fresh user/assistant turn. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Non-blocking compaction so the user almost never waits for the 90% sync floor. Self-contained so it can be reverted as one commit. - Trigger (process-wake): after a turn whose usage ≥ 85% (env ELECTRIC_AGENTS_COMPACT_BG_CEILING), kick off a DETACHED summarization. Its checkpoint is applied at the NEXT turn's start, OR — if the summarize finishes while the entity is idle — immediately by waking the idle loop and writing the checkpoint WITHOUT running the agent (so the indicator never lingers past completion). The slow summarize never blocks; a fast follow-up prompt just runs un-compacted. - context-factory: maybeStartBackgroundCompaction / writeBackgroundCheckpoint / failBackgroundCheckpoint on the handler-context result. Snapshots the timeline head as the watermark; summarizes the whole reconstructed history; writes a background-flavored running→complete checkpoint. - Unique checkpoint id per generation (compaction-bg-<watermark>): context supersession keys on id alone, so with one shared id the NEXT background's `running` row silently superseded the PREVIOUS `complete` one — erasing the watermark and undoing every compaction (context never shrank, the indicator stuck on "running"). running→complete→failed of one generation share the id; the next generation can't clobber it. Mid-turn sync keeps the shared id on purpose (its re-summarization chain wants supersession). - Reconstruction (timeline-context): checkpoints carry a stored attrs.watermark and the summary is rendered AT that watermark — so a prompt+answer that arrived while a background summary ran (physically after the checkpoint, logically after the watermark) are kept verbatim AFTER the summary (RFC §8.5). Falls back to the row's order for sync. - UI: CompactionIndicator shows a subtle "Compacting in background…" for background checkpoints, distinct from the blocking sync indicator. - Summarize hardening (compaction-summarize): bound every summarize call with a 120s deadline — the anthropic provider only enforces a timeout/ abort when the caller passes timeoutMs/signal and never retries, so a stalled stream hung forever. Forward timeoutMs + an AbortSignal the provider honours AND race a hard timer; a timeout becomes an ordinary failure retried next turn-end. The mid-turn 90% sync floor stays as the safety net for single runaway turns. To drop the feature: revert this commit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The `minTokens` guard (default 2000, env ELECTRIC_AGENTS_COMPACT_MIN_TOKENS) never changed the outcome at any realistic ceiling — 90% of a real context window is always far above 2000 tokens, so the floor only ever mattered when testing with an artificially low ceiling. Codex has no equivalent floor (it triggers on a single token threshold). Remove the knob, its env override, and the now-unused positiveFromEnv helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4605] The mid-turn (sync floor) checkpoint was persisted with no attrs.watermark, so reconstruction fell back to the checkpoint row's own (latest) stream position and, on the next turn, dropped every item before it — including the verbatim tail the mid-turn summary deliberately excluded (keepTail). Those recent messages were in neither the summary nor the kept set, so they were silently lost on the 90% path. Fix: the mid-turn compactor now folds the WHOLE context into the summary (Codex-style — no verbatim pre-compaction tail), and the checkpoint is stamped with watermark = current timeline head. Summary and watermark now agree, so reconstruction folds exactly what was summarized and keeps everything appended afterward (the model's post-compaction output + the next prompt). This also removes the keepTail + tool-pair orphan-guard complexity. Recent context is still shown after a compaction via the sticky view (summary + messages appended since), so within-turn coherence is preserved. Also drop the stale "95%" hard-ceiling comments (it has been 90% since the ceiling was lowered to match Codex). Tests: rewrote compaction-midturn for the summarize-everything behavior; added a reconstruction test asserting post-compaction messages survive a mid-turn checkpoint. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…#4605] A summarize is bounded by a ~120s hard timeout after which a terminal (complete/failed) checkpoint is always written, so a `running` checkpoint that lingers well past that is orphaned — its process crashed before writing the terminal row, and (with watermark-unique background ids) nothing supersedes it, pinning the spinner forever. Treat a `running` checkpoint older than 150s as stale and stop showing it, with a self-clearing timer so the spinner disappears even when no further events arrive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…4605] `pendingRequestMessageCount` records the incoming (uncompacted) message count, not the compacted list the adapter may return. That's intentional: pi-agent passes transformContext the full conversation each step, so the count indexes that original array — the next step's trailing slice then measures exactly the messages appended since. Add a comment so it doesn't read as a bug. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
#4605] The mid-turn checkpoint computed its watermark (timeline head) when the `complete` row was written — after the summarize await. Any event that materialized into the StreamDB during that (slow) await would bump the head past what the summary actually covered, so reconstruction could drop an un-summarized item next turn (item.at <= watermark). Narrow (mid-turn blocks the agent, and pending inbox rows don't materialize) but timing-dependent. Snapshot the head when the `running` row is written instead — before the await — so coverage and watermark derive from the same instant, matching how background compaction already captures its head up front. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eview #4605] The orphan-clearing logic added to CompactionIndicator had no test, and a regression where rows stop carrying `timestamp` would silently revert to the lingering-spinner bug (NaN → not orphaned → spinner stays). This package has no React-render harness, so extract the decision into a pure `isRunningCheckpointOrphaned(timestamp, now)` helper (with STALE_RUNNING_MS) in lib/ and unit-test it: fresh → shown, just under the deadline → shown, at/past the deadline → hidden, and missing/unparseable timestamp → shown (documented). The component now imports the helper, so the tested logic is exactly what runs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The RFC isn't checked in, so comments can't reference it; phase numbers are internal sequencing, not something the code should narrate. Remove the dangling RFC/§/phase references and tighten the surrounding comments to be brief and only where they clarify something non-obvious. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the `for (;;)` + breaks with a `do…while (appliedBackgroundDuringIdle)` so the loop's actual condition (re-idle while background compactions keep settling) is explicit, and trim the over-long `pendingBackgroundCompaction` comment. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tighten the over-long comments in the compaction indicator helper + component and the reconstruction watermark block, dropping a redundant inline restatement. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…into test-agents-prs
# Conflicts: # pnpm-lock.yaml
Contributor
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #4726 +/- ##
===========================================
+ Coverage 43.18% 60.22% +17.04%
===========================================
Files 312 397 +85
Lines 25598 44042 +18444
Branches 8699 12676 +3977
===========================================
+ Hits 11054 26524 +15470
- Misses 14466 17436 +2970
- Partials 78 82 +4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
Electric Agents Mobile BuildLocal mobile checks ran for commit The EAS Android preview build was skipped because the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This hardens agent wake delivery across repeated child completions and laptop suspend/resume. Parents now receive every queued child wake, and the desktop reconnects stale pull-wake streams after resume without restarting the agent runtime or interrupting active work.
Root cause
Three gaps combined to make wake recovery unreliable:
BuiltinAgentsServer.start(). Earlier failures in configuration, credentials, imports, or construction escaped the reconnect path.Approach
PullWakeRunner.reconnect()operation that uses the existing XStateSTREAM_RESETpath.BuiltinAgentsServer.reconnectPullWake().powerMonitorsuspend/resume events and reconnect only connected, running local runtimes.Key invariants
BuiltinAgentsServeror abort active LLM/tool/wake work.Non-goals
@durable-streams/client; that remains appropriate upstream work.Trade-offs
The deferred queue can grow while a claim remains permanently stuck, but retaining notifications is preferable to silently losing child completions. Electron resume handling and heartbeat-gap detection are intentionally redundant; both converge on the same idempotent XState reset path.
Verification
Verified locally:
Files changed
packages/agents-runtime/src/pull-wake-runner.ts: lossless deferred wake queue and explicit stream reconnect API.packages/agents-runtime/test/pull-wake-runner.test.ts: repeated-generation, ordering, offset continuity, duplicate reset, and active-wake preservation regressions.packages/agents/src/server.ts: narrow built-in runtime reconnect facade.packages/agents/test/builtin-pull-wake-registration.test.ts: lifecycle-safe facade coverage.packages/agents-desktop/src/runtime/lifecycle.ts: full startup error recovery and resume-time runtime selection.packages/agents-desktop/src/app/power-monitor.ts: idempotent suspend/resume listener lifecycle.packages/agents-desktop/src/main.tsandsrc/app/controller.ts: startup/shutdown wiring.