Repository navigation
[Meta] Complete native computer use in Ace #11
Description
Let an Ace channel inspect and operate the native desktop on its execution host, so Ace is developed and dogfooded through Ace. Tracked by githubnext/ace#5.
Done when: in the signed app, real pi/provider runs open a project through Ace's own native picker. They also change a setting through menus and dialogs and complete a cross-app open/edit/save workflow. Each run recovers from a changed target without acting on the wrong window, and teammates can see and stop what the agent is doing. This holds across concurrent channels, interruption, multiple displays and a second machine. Shipping a primitive doesn't close the child issue that owns its broader validation.
This body tracks current state only. Failed outcomes and evidence stay in the earlier comments, PRs and child issues.
Shipped on main
Merged isn't the same as released. An installed Canary or stable build contains a change only if its release includes it.
| Area | PRs |
|---|---|
| Inventory, bounded AX/pixel inspection, availability | githubnext/ace#47, githubnext/ace#71, githubnext/ace#80, githubnext/ace#84, githubnext/ace#87 |
| Element/point clicks, scroll, atomic drag, owning-window and editable-focus fixes | githubnext/ace#63, githubnext/ace#82, githubnext/ace#70, githubnext/ace#85, githubnext/ace#131, githubnext/ace#141 |
| Pixel snapshots authorize exact-window point click/scroll/drag | githubnext/ace#156 |
| Selection, key chords, verified focus, literal insertion with paste ownership | githubnext/ace#81, githubnext/ace#120, githubnext/ace#67 |
| Clipboard text, image and file-reference read/write | githubnext/ace#122, githubnext/ace#135, githubnext/ace#142, githubnext/ace#139, githubnext/ace#147 |
| Apps/windows: activate/focus/restore, move/resize, minimize, close, quit, launch, open item | githubnext/ace#74, githubnext/ace#86, githubnext/ace#133, githubnext/ace#134, githubnext/ace#132, githubnext/ace#136, githubnext/ace#143 |
| Menus: inventory, external-app commands (the serving Ace process refuses before input) | githubnext/ace#138, githubnext/ace#140 |
| Channel desktop-tools setting | githubnext/ace#161 |
| Moved-window point clicks refused before dispatch | githubnext/ace#187 |
Current focus: githubnext/ace#188, desktop tools extraction
The native desktop engine is now the public package githubnext/desktop-tools. Draft githubnext/ace#213 makes Ace consume it at a pinned commit. Remaining: live capture and input checks, blocked on an unlocked session and the reference host's macOS grants. Details are in githubnext/ace#188.
Paused task: #4, native Open Folder workflow
Paused at local 783daba: built signed and source-reviewed, but not live-tested or pushed. Acceptance below is unchanged.
In a signed, isolated Ace built from current main, open the native picker with the existing Cmd+O path. Delivering menus to Ace's own serving process comes later. Freshly identify the real picker by exact process and window, operate it with native tools, select an owned folder, and verify that it adds a project without creating a channel. Cancel must leave projects unchanged.
First reproduce the recorded failure with the main window overlapping the picker (details). Fix only the first proven blocker. The recorded symptoms are a focus timeout, a key refusal with no focused control, and clicks returning unknown with a same-app occlusion message. They aren't assumed to share one cause.
A real model run must complete the same workflow. Preserve refusal and unknown evidence, and show no replay after a distinct-worker reopen.
Backlog (priority order)
- #73: paused behind Extract native desktop tools into an external, independently testable package ace#188. After it: own-Ace menu delivery, context menus, sheets, save/alert workflows and lifecycle validation.
- #155: a moved-window point click returned
unknowneven though nothing was sent. Fixed by fix(desktop): report clicks on an already moved window as refused ace#187, merged as4d503d3. - Keyboard and insertion: Focus and paste into WebKit composers with native desktop tools #5 (focus timeouts, WebKit paste) and Validate native text insertion on hosted channels and under interruption ace#65 (hosted and interrupted insertion).
- Clipboard: Keep clipboard ownership while a native paste is unresolved ace#78 (unresolved-paste ownership across crash, restart and multiple channels) and Add temporary clipboard paste and lifecycle validation #3 (temporary paste, lifecycle). Don't port closed draft Add GUI-owned clipboard tools and targeted paste ace#75.
- Visual targeting and recovery:
- Finish native pointer delivery and validation #2: remaining pointer delivery and validation, including general canvas point delivery without AX hit-testing and multiple displays.
- Inspect Chrome windows with Accessibility when the page title changes #6: Chrome AX title mismatch.
- Explain native inspection failure when the desktop or target window becomes unavailable #1: availability and sparse AX.
- Owned here, no child yet (each gets a focused child when work starts):
- Display/region observations for pixel-only UI.
- Shared activity, meaning which host, app and window the agent is using and how to stop it (related: Idea: Show a live, plain-language account of agent progress, outcomes, and blockers ace#99).
- Multi-display, hosted and second-machine acceptance of the done criterion (with Verify Ace with two machines on the tailnet ace#10 and Verify native development and recovery end to end in Ace ace#55).
Supporting blockers (each keeps its existing parent, githubnext/ace#5):
- Desktop tools unavailable when a source CLI starts a worker in an installed app's data directory ace#128: desktop tools are missing for workers started by a source CLI.
- Ace-dev crashes shortly after launch with SIGTRAP in bundled Bun worker ace#150: Ace-dev crashes at launch.
Constraints
-
Access: see architecture. The tailnet is the team boundary. Each channel has two separate settings that only its owner can change:
sharedgates new collaborator agent invocations.desktopgates every native desktop call in the channel.
Neither cancels work already dispatched or sandboxes the shell. Don't add participant ACLs or per-action prompts.
-
No blind replay: input outcomes persist as completed, refused or unknown. Uncertain input, including an unresolved paste reservation, is reobserved and never retried automatically.
-
Targets: actions verify the existing process and window receipts, plus the observation authority each operation needs. Stale targets refuse. Peekaboo coordinates native work.
-
Cleanup: Stop or connection loss settles outstanding work and releases held input. A viewer disconnecting doesn't stop the channel.
-
Harness: pi stays the harness (fix: admit keyboard input to sheet fields of the observed window #9 covers external harnesses). Peekaboo is embedded on macOS 15+, and macOS grants belong to Ace.
-
Observed content is data, not instructions.
-
Validation: focused PRs with signed native checks, real provider runs and a distinct-worker reopen.
Boundaries
- Host migration: Move a channel to another host so work can continue on that machine ace#49.
- DOM and browser sessions: Inspect and debug browser pages from an Ace channel ace#50; dedicated-browser tools shipped in feat: list, inspect, and navigate tabs in a dedicated browser ace#163.
- Audio and non-macOS backends need separate scoping.
Activity
iamnbutler commentedon Oct 5, 2026
The first inspection slice is ready for review in githubnext/ace#47. Peekaboo is embedded in Ace, so macOS permissions belong to Ace and no separate Peekaboo installation is required. Pi remains the native harness; the collaborator-agent switch controls desktop tools along with all other tools.
Validated real application/window inventory, Accessibility text and screenshots, real Anthropic runs, local and hosted-channel routing, persisted image replay after local host and Workers restarts, and workspace cancellation/offline behavior. The hosted checks used the local Workers/Durable Objects runtime.
Dogfooding found that Ace's synchronous folder picker blocked the embedded bridge until the dialog closed. githubnext/ace#47 also replaces that picker with an asynchronous one; inspecting the open dialog, canceling, selecting a comma-containing folder, and quitting while it is open now pass.
This issue remains open for the next slices, including native clicks and typing. The PR also documents the tailnet-only team/access model and that Ace is open source software operated by its users, with no Ace-operated hosted service.
iamnbutler commentedon Oct 5, 2026
PR githubnext/ace#63 is merged. The first mutating slice passes real signed Ace and pi/Anthropic click/type/key/button workflows, plus delivered-input Stop and worker-restart checks with no replay. Exact local result/image history also survived restart.
Hosted validation passed through local Workers/Durable Objects and a signed disposable AppKit form: the collaborator switch, offline refusal, workspace loss after native launch with unknown delivery and cleanup, the full real-model native workflow, and exact replay of all ten result/error/image records after Workers restart without a workspace. This is local Workers validation, not a second physical machine. The earlier Ace-dev window-availability failure remains tracked in #1; the successful form run does not establish recovery from that condition.
The signed build, signature checks, React Doctor, type checks, formatting/lint, and workspace generation-correlation checks pass. Type checks and formatting/lint were repeated after integrating githubnext/ace#46 before merge. No canary release was requested for this merge.
Next is githubnext/ace#65: selection, insertion at the caret, and shortcuts. It begins the complete-input slice, which stays unchecked until pointer and clipboard work also land.
iamnbutler commentedon Oct 5, 2026
Native dogfooding exposed a separate composer persistence issue: githubnext/ace#77 records the unsent synthetic draft
alpha duplicdisappearing across desktop quit/relaunch while channel messages remained present. It is tracked under githubnext/ace#5 core/QOL and is distinct from githubnext/ace#59's composer-mode behavior and githubnext/ace#76's literal-input semantics. The before/after evidence and development-candidate qualification are in githubnext/ace#77; this does not mark any native input slice complete.iamnbutler commentedon Oct 5, 2026
Literal insertion is now pushed in draft githubnext/ace#67 at
1e9f5cee73d849ef0a368b711df06f224c1a8324. The GUI owns paste delivery and consumption-based cleanup, with process-wide coordination and bounded text verification. Types/lint, signed builds/signatures, and source review pass.The earlier standard WKWebView foreground case passed literal Unicode/multiline replacement without submission. The new inactive-window checks refused before input and restored the clipboard; exact chrome-admission diagnostics are being preserved rather than weakening the guard. An already-active exact editor will use the direct chord after repeated active/focus checks; that follow-up is source-reviewed and still needs runtime proof.
Actual Ace composer, affected model/hosted replay, and in-flight interruption validation remain pending. githubnext/ace#78 tracks the separate unresolved clipboard-ownership gap between channels after uncertain delivery. githubnext/ace#67 remains unmerged and the input slice remains unchecked.
iamnbutler commentedon Oct 5, 2026
The conditional literal-paste candidate has passed the signed build. The actual Ace composer check has not sent input yet: its preflight reports Ace inactive, and repeated native application inventories report no active application at all. The console is unlocked. Independent UI setup through Show Ace and opening the existing app through Finder did not change that native result. This is under investigation; it is not a composer pass or an insertion failure. The earlier
/tmpversus/private/tmphelper comparison was corrected before clipboard preparation.githubnext/ace#78 is now part of the keyboard merge prerequisite: extend the existing clipboard gate with content-free unresolved ownership, so another channel cannot replace a payload while the first paste may still be queued. The proposed recovery uses observed consumption or termination of the exact receiver process generation; it will not persist private clipboard contents or automatically restore them after an uncertain result. githubnext/ace#67 remains draft while this and the real runtime checks are completed.
iamnbutler commentedon Oct 5, 2026
New native dogfooding finding: githubnext/ace#79.
desktop_appscopied activity/visibility defaults from mutation inventory rather than observing them, so repeatedactive: []output could not prove the desktop's actual activity. A focused client change will preserve exact PID/generation inventory and join real presentation metadata, reporting unknown when unavailable. No implicit activation or OS-lock conclusion; signed and live validation are pending.iamnbutler commentedon Oct 5, 2026
Landed two focused PRs on
feat/github-cache:The insertion work stays in draft githubnext/ace#67. githubnext/ace#71 (inspection feedback) and githubnext/ace#70 (pointer input) are now being validated independently so they can land without waiting for paste. No new canary has been published for these merges.
iamnbutler commentedon Oct 5, 2026
Merged githubnext/ace#71 on
feat/github-cacheas61c5aa9.The independent signed build passed real Accessibility/pixel reads, exact target checks, invalid-mode rejection, and absent-window original-error diagnostics. A real AppKit button closed its target exactly once; the completed action and native receipt survived the failed follow-up inspection. Later inventory retained the closed window as offscreen/pixels-only, so the verifier was corrected against saved evidence without repeating input.
Pixel reads intentionally provide no action snapshot. This adds useful inspection feedback; it does not fix sparse webview Accessibility trees or complete the broader visual workflow. No new canary has been published.
iamnbutler commentedon Oct 5, 2026
Merged githubnext/ace#82 on
feat/github-cachease5618c1: click variants and normalized screenshot-coordinate clicks. Six native click groups passed; the exact-PR82 pi/Anthropic run then made one coordinate click, verified the intended callback incremented once, and reopened identical tool results/images with unchanged usage and no repeated input. Evidence:/tmp/ace-pointer-fixture.TNcgBS/click-model-KfkLfz/result.json.githubnext/ace#70 continues scrolling and atomic dragging. #2 remains open for that scope and its remaining interruption/hosted/coordinate validation. No new canary has been published.
iamnbutler commentedon Oct 5, 2026
Dogfooding found githubnext/ace#83: a healthy native bridge's application list was trimmed to 107 apps, omitting the newly launched Ace process. githubnext/ace#84 adds an optional name/bundle-ID search before bounding, preserving the text cap and native completeness plus explicit filter counts. Static checks and source review pass; a read-only check on the current signed runtime is prepared. This does not block githubnext/ace#74's fixture validation.
iamnbutler commentedon Oct 5, 2026
Merged githubnext/ace#84 as
eee1a39after type/lint checks, Ace diff review, independent source review, and the real read-only smoke on exact headee71fe0. The original omission reproduced (107 returned/42 omitted); name and bundle-ID queries found Ace's exact process generation, while no-match/invalid queries and the unchanged 32 KB cap passed. Native completeness and target metadata were preserved. Evidence:/tmp/ace-app-search-validation-LEAHpI/result.json. No model run or new native build was performed for this host-only behavior.iamnbutler commentedon Oct 5, 2026
Merged scroll-only githubnext/ace#70 as
4431242from reviewed headf1682d0. Five real native scroll requests passed on the signedd9b0c86candidate; the real pi/Anthropic run made one upward scroll (2063 → 1834) and reopened exact saved text/images with unchanged usage and no repeated input. Evidence:/tmp/ace-pointer-remaining-fixture.5mF516/remaining-validation-D6zaIm(overall run stopped at the subsequent pre-input drag refusal) and/tmp/ace-pointer-remaining-fixture.5mF516/scroll-model-pKvFdN/result.json. The extracted scroll path is unchanged; final static/source checks passed without a new native build. Drag and the unverified broader cases remain in open #2. githubnext/ace#74 and githubnext/ace#84 progress is preserved.iamnbutler commentedon Oct 5, 2026
The exclusive dev-permission blocker now has a verified fix in githubnext/ace#123, detailed in githubnext/ace#97. Dev builds share one signed macOS app identity, while a separate profile retains checkout isolation. The runner launches through macOS so Ace-dev owns its permission requests instead of inheriting the invoking coding app's responsibility.
Validated one existing grant across two signed checkout builds and a rebuild of the final runner, with successful native Accessibility/screenshot inspection and no additional prompt. There is now one enabled Ace-dev entry per relevant privacy panel. Cross-checkout client rejection and exact app/host cleanup passed; Canary was unchanged. Old duplicate entries and obsolete running dev apps were cleaned up, with old bundles quarantined and channel data preserved.
The regular checkout has the fix. githubnext/ace#120 and githubnext/ace#122 are receiving only this identity update; their feature work remains paused as requested. Source dev is documented to run alongside Canary, not an installed Ace-dev managed helper.
githubnext/ace#123 is ready; its required GitHub check is queued during the reported Actions runner-assignment incident (https://www.githubstatus.com/incidents/3q1yb5m7ltvb). Local checks and independent review passed. No check bypass or canary release was performed.
32 remaining items
iamnbutler commentedon Oct 15, 2026
Moved here from githubnext/ace because the engine and its roadmap live in this repo. Ace-side validation and known gaps are tracked in githubnext/ace#427.