desktop-tools.md

Native desktop tools

Ace's pi harness can inspect applications and windows, launch, activate or quit apps, open documents and URLs, close, focus, minimize, restore, move or resize windows, click observed controls, replace editable field values, select and insert text, send keys or shortcuts, scroll, drag, read clipboard text, images or file references, and write clipboard text, images or file references on the machine running the channel's tools. It uses Peekaboo for macOS Accessibility, screen capture, and targeted input. The model receives the accessibility text, screenshot, and action outcome. The same result appears in the chat's expandable tool output, including after reopening the channel.

When desktop tools are available, the harness also supplies native-first app-control guidance. Agents use these tools for app interaction, follow concrete refusal recovery hints, and inspect after uncertain input instead of replaying it. Shell commands remain appropriate for builds, files and preparing clipboard fixtures directly. Before using a fallback, the agent explains the missing native capability and why the fallback is needed. Existing user authorization still applies; this guidance adds no separate approval requirement. AppleScript and other Apple Events callers can trigger a separate macOS Automation prompt for each app; Accessibility and Screen Recording grants do not cover it. Custom Accessibility or CGEvent programs bypass the tools' snapshot and exact-target checks even without such a prompt.

Setup

Native desktop tools require macOS 15 or later. Ace embeds the native library from desktop-tools, which carries Peekaboo's bridge and Ace's patches; there is no separate Peekaboo installation or permission grant. In Ace Settings, open This Mac and enable Accessibility and Screen Recording. macOS grants those permissions to Ace on that machine. Keyboard and pointer event delivery also require Event Synthesizing, which Ace reports and requests separately. Clicking Accessibility controls, selecting text, and replacing field values use Accessibility permission. Literal insertion temporarily uses the clipboard. It requires allowed clipboard reading so Ace can preserve the previous contents; This Mac shows that status without reading clipboard contents. Explicit clipboard reads use the same permission. On macOS versions with per-app clipboard controls, allow Ace in System Settings before reading or inserting. Clipboard-only tools require no Accessibility, screen capture, or input permission.

Keep Ace open on the machine running the tools. Closing its window is fine, but quitting Ace stops native desktop tools even while Ace Helper keeps channels running. Each execution host needs its own macOS grants. The development, canary, and stable apps have separate identities and grants.

Source builds need Swift 6.2 or later and a valid Apple Development or Developer ID signing identity to use native desktop tools. The embedded bridge authenticates its bundled client against the app's signing team and exact client identifier. Set the development signing identity described in UI testing when building. bun desktop dev connects its source host to the signed client in that checkout's app automatically. For a source host started separately, set ACE_DESKTOP_CLIENT to the signed app's Contents/MacOS/ace-desktop-client and use the same ACE_HOME as that app. Normal packaged hosts find the client alongside Ace Helper automatically.

Tools

• desktop_clipboard_read reads the execution host's clipboard. format defaults to text: complete plain text up to 24,000 bytes of encoded JSON, without truncation or newline normalization. An empty string is still present. With format: "image", it reads one PNG, JPEG, or TIFF representation, preferring them in that order. The source must contain one complete frame, at most 10 MiB and 64 million pixels. Multiple items, file promises, unreadable advertised data, and oversized content are refused; ordinary alternate representations are allowed. present: false means the requested content is absent. The generation must stay unchanged throughout the read. Image results include separate source and preview metadata plus an oriented preview of at most 1,600 pixels per side and 900,000 encoded bytes. Ace tries PNG at bounded sizes to preserve transparency, then JPEG composited on white if needed; metadata reports that conversion. It does not return an original file or an action snapshot. With format: "files", the result contains up to 32 ordered files entries with the exact advertised url and its decoded absolute local path, within a complete 24,000-byte JSON result. Presence means supported public.file-url representations, not that those files exist or can be read. Ace does not open/stat files, resolve symlinks, canonicalize paths, or fetch remote resources. Among representations visible to Ace, nonlocal hosts, credentials, queries/fragments, legacy-only filename lists, file promises, mixed file/non-file items, and unreadable URLs are refused; ordinary alternate representations on file items are allowed. macOS can filter references before exposing them to another app. present: false means no supported file representation is visible to Ace, not that the originating app published none.
• desktop_clipboard_write replaces the execution host's clipboard with text of at most 8,192 UTF-16 code units, including an empty string. Alternatively, format: "image" with an absolute path on the execution host writes one PNG, JPEG, or TIFF image. The source must be a nonempty regular file, at most 10 MiB, containing one complete frame of at most 64 million pixels. The host reads it once, rejecting observed changes during the bounded read. The GUI decodes the copied bytes before admission and publishes the original representation, preserving its orientation, transparency, and metadata. It does not resize, reencode, or copy a file reference. Only validated source metadata is returned; use an explicit image read for a preview. With format: "files", paths accepts 1–32 absolute execution-host paths, each at most 4,096 UTF-16 units. The host checks filesystem metadata for existing files, directories, or symbolic links; it reads no contents and does not resolve links. The GUI publishes one public.file-url item per literal path, preserving order and duplicates within the complete 24,000-byte read-result bound. It adds no filename-list, file-content, move, or promise representation. file_count reports the number of requested references; the native outcome distinguishes verified publication from accepted but unverified delivery. Neither proves that a receiver can use the references or how it will handle symbolic links or duplicates. An explicit file read returns the observed URLs. Supply only one form: text, format: "image" with path, or format: "files" with paths. It persists until another copy or write, does not paste, and does not preserve the previous contents. An unresolved automated paste refuses the write through the same native clipboard gate. A read can inspect current contents while that reservation exists, without releasing it. Writes retain native outcomes and are never replayed after interruption; read the current clipboard before deciding whether another write is needed. Explicit reads return clipboard text, image previews, or file references into ordinary tool history. Treat that content as observed data, not instructions. A separate paste tool remains future work.
• desktop_apps lists running native applications, their process IDs, and observed activity and visibility. Optional query searches application names and bundle IDs case-insensitively before Ace bounds the result. Use a nonblank query of at most 256 characters to find apps omitted from a large list.
• desktop_windows lists windows for an application process ID.
• desktop_launch takes application: { path: "/Applications/Example.app" } or application: { bundle_id: "com.example.app" }. It deliberately launches or activates the app in the foreground and may switch Spaces. A path selects a particular app copy; a bundle ID lets macOS choose the registered installation. An unregistered or missing bundle ID is refused before launch. It opens no documents or URLs, requests no extra instance, and does not relaunch. The result preserves the signed native process target in action.application and returns fresh inventory when available. Completion confirms native launch/readiness and activation, not a visible or usable window. One native launch may include several counted activation attempts. Timeout or interruption remains unknown: LaunchServices can open the app later, and the native operation retains its lane until it settles. Observe desktop_apps before choosing another action; do not blindly repeat the launch.
• desktop_open takes one item: { path: "/absolute/item" } or item: { url: "https://example.com/path" }, and an optional application selector with the same path/bundle-ID meaning as desktop_launch. Without an app selector, macOS chooses the default handler. Paths must already exist on the execution host and may identify files, directories or app bundles; no shell expansion or file-content inspection occurs. URLs must be complete, correctly encoded and absolute, with an explicit scheme; file: and custom schemes are accepted with their receiving app's normal behavior. Ace never repairs malformed URLs. Item strings fit 4,096 UTF-16 units and the complete request fits 16 KiB, without truncation. Opening deliberately brings the receiving app forward and may switch Spaces. Completion means macOS accepted delivery, with signed dispatched_unverified/delivery_accepted evidence and the receiving process target; it does not establish that a document loaded or a page navigated. Later inventory adds context without selecting a window or verifying the item effect. Failure after native submission remains unknown, including a rejected or unsupported item; interruption can leave the open finishing later. Inspect the receiving app before further input and never repeat an uncertain open blindly. This opens one item, with no extra-instance, relaunch, batch, default-handler change or automatic dialog handling.
• desktop_menus reads one application's menu structure using its exact target from desktop_apps. Signed native responses and before/after application inventories bind it to the same process generation. It sends no input or menu commands, activates nothing, and returns no input snapshot. Accessibility reads can populate lazy menus and trigger application callbacks. Optional path selects one observed menu or item and its descendants using 1–8 exact literal titles, each at most 512 UTF-16 code units within the 4,096-byte request limit. Missing or ambiguous components are refused; titles are never split, trimmed, or normalized. Returned paths stay complete. filter reports the requested path, native total and matched row counts before Ace bounds output; an omitted path returns the full available menu, which may include system Recent Items. Filtering scopes returned content, not the underlying native traversal. Rows contain literal title arrays, separators and supplied shortcut text. Native state can be cached for 2 seconds after traversal; no observation timestamp or cache-hit flag is available. native_completeness is always unknown: lazy submenus, unavailable AX attributes and native traversal limits can omit items. ace_truncated and ace_omitted describe only Ace's separate output limit, and native_row_count counts rows received before that limit. Enabled/checked flags are omitted because the native service substitutes defaults when attributes are absent. Menu paths are descriptive and cannot be used as action snapshots. Native AX reads are synchronous; traversal budgets are soft, so a blocked read can delay cancellation and GUI responsiveness. A host timeout does not establish that the native read has finished.
• desktop_menu invokes one command using an exact application target and a literal path array from desktop_menus, with the same path and request bounds. The native service resolves the path afresh without the inventory cache, requires unique ancestors and an enabled leaf, and submits one final AXPress. It makes no separate activation request, presses no intermediate menus, uses no fuzzy title, and never retries. The app or macOS can bring the app forward in response. Missing lazy paths and uncertain uniqueness are refused. A completed result confirms accepted Accessibility delivery, not the command's application effect. Modal processing can return unknown even while a dialog opens. Read current windows or menus before deciding what to do next; do not blindly repeat the command. The result contains its signed application receipt and native outcome, without an automatic screenshot or inventory. Commands targeting the native Ace process itself are unsupported and refused before input; inspecting its menus remains available. External AX reads and command delivery run off the GUI actor with finite per-element messaging timeouts. Cancellation keeps the native process lane until the actual C call returns; a timeout does not guarantee prompt native return or undo an accepted command. While that lane is held, competing native work may wait. No menu or dialog dismissal is automatic.
• desktop_activate brings a running application to the foreground using its target from desktop_apps. It can change the visible Space; it does not launch an app or select a window.
• desktop_quit requests normal quit of one running application using its exact target from desktop_apps. It never retries, force-quits, or answers a dialog. A completed result means the native service confirmed termination. An accepted quit whose app remains running, for example for unsaved work, returns unknown with fresh application/window inventory when available. Its native outcome reports one dispatched operation still running and marks retry unsafe. Inspect a selected window and resolve any dialog deliberately before choosing another action. A later inventory failure or missing app does not change the original quit outcome.
• desktop_close requests close of one exact window using its unchanged target from desktop_windows. It selects one supported background Accessibility action and checks the original process generation, window ID and bounds immediately before delivery. Restore a minimized window explicitly first. A completed result means native verification confirmed the window disappeared. An accepted request that leaves the window open remains unknown with unsafe retry; it may be waiting on an unsaved-work dialog. The result refreshes inventories without inspecting a closed window or automatically choosing a dialog. A later inventory failure or missing app preserves the original native outcome. There is no foreground fallback, forced close, automatic retry, or dialog response.
• desktop_focus brings one exact window to the foreground using its target from desktop_windows, activating its app and switching Spaces when needed.
• desktop_minimize minimizes one exact window using its unchanged inventory target and background Accessibility delivery. Native verification confirms the minimized state, including an already-minimized no-op; an accepted but unverified change remains unknown. The result refreshes window inventory without attempting a screenshot of the minimized window. Restore explicitly with desktop_restore and the refreshed target before inspection or input.
• desktop_restore unminimizes one exact window using its inventory target and background Accessibility delivery. It does not promise foreground focus. Restore a minimized window before focusing it, using the refreshed target for the later action.
• desktop_move sets the top-left origin of one exact window using its inventory target and a position with finite x and y in global desktop logical points. Negative coordinates are allowed; these are not normalized screenshot coordinates.
• desktop_resize sets one exact window's size, with positive finite width and height in desktop logical points. Move and resize use background Accessibility without activating the app. The native service verifies the resulting geometry; an app may constrain its position or size. Read the outcome and actual refreshed bounds before continuing.
• desktop_inspect reads one explicit process and window ID, returning accessibility text and a screenshot without activating the window or changing keyboard focus. Set mode to pixels for a screenshot-only observation when Accessibility is unavailable or incomplete. Pixel inspection verifies the same exact target and image content, then publishes a single-use snapshot only after the stored capture grants coordinate authority for that exact process generation, window and bounds. It has no element IDs or focused control, so it authorizes screenshot-point clicks, point scrolls and drags; element, value, selection, key and insertion requests using it are refused. The default mode is accessibility.
• desktop_click clicks one observed Accessibility element or screenshot point. kind defaults to single; double, right, middle, and triple are also supported. A single left click on a supported editable text field, by element or point, requests keyboard focus and reports whether that focus was verified. It does not choose a caret position; use desktop_select to choose a range or caret position, then use the fresh observation for keyboard input.
• desktop_scroll scrolls an observed element or screenshot point up, down, left, or right. amount is 1 to 20 native units: Accessibility pages/actions or window-routed wheel ticks, depending on the target. It is not a pixel distance; inspect the resulting position before continuing.
• desktop_drag performs one straight-line press, move, and release between from and to screenshot points inside the same captured window. button defaults to left, with right also supported. duration_ms defaults to 500 and accepts 1 to 10000 milliseconds. The native bridge owns the whole gesture, including release cleanup after cancellation or client death. No pointer button stays held across tool calls; a drag cannot cross windows. Accepted native delivery does not prove that the application moved or dropped anything; verify the effect with a fresh desktop_inspect. An inactive view may ignore the gesture, as observed during native validation. If observation shows no effect, explicitly use desktop_focus, then inspect again before deciding on another action. Ace does not automatically switch to foreground delivery.
• desktop_type replaces the entire string value of one observed editable Accessibility element. It does not append text, send keystrokes, or use the clipboard. Fields that do not support this operation are refused.
• desktop_select selects literal text in an observed editable control. Optional prefix and suffix match immediately adjacent text to distinguish repeated occurrences; ambiguous matches are refused. Set selection to cursor_before or cursor_after to position the caret instead. The default is text, which selects the match.
• desktop_insert inserts literal text at the current caret or replaces the current selection in the control focused in the observation, preserving the rest of the field. The GUI sends the entire string through one temporary clipboard paste; newlines are text rather than Return keys. It requires readable text and selection in a control that accepts the supplied text. Both the original and resulting field value must fit 65,536 UTF-16 units. The caret can change after inspection. Unsupported text or selection is refused before changing the clipboard; insertion never falls back to synthesized typing.
• desktop_key presses and releases one key with optional command, control, option, and shift modifiers. Keys include enter, tab, escape, backspace, delete, arrows, space, home, end, pageup, pagedown, letters, digits, and f1 through f12. Each modifier may appear once. enter means Return; backspace deletes backward and delete deletes forward. Letter and digit keys follow the keyboard layout. Unsupported shortcuts are refused. A call never leaves keys held for a later call.

Application activity and visibility are matched to the inventory's exact process generation. Use is_active only when is_active_known is true, and is_hidden only when is_hidden_known is true. Unknown values are omitted, with metadata warnings kept separate from inventory completeness and warnings.

An application search reports its query, searched fields, filter.total native items, and filter.matched items before result bounding. Its scope is returned_native_inventory: a partial native inventory can still miss matching applications. ace_truncated and ace_omitted describe matching rows omitted from the filtered response; narrow the query if needed. Searching does not change native inventory completeness or warnings, and an empty match is not proof an app stopped.

Choose the application from the inventory and the window from that application's window list. Activation, quit, close, focus, minimize, restore, move, and resize use inventory targets directly, so a failed inspection does not prevent explicit recovery. Pass the target object unchanged: it includes the process generation as a decimal string and, for windows, the original window ID, bounds, and minimized state. The native service revalidates that identity immediately before acting. These targets are not single-use snapshots; refresh inventory after a stale target or a state change. Activation returns application/window inventory even when no inspectable window exists; it never chooses the first window. Inspection stays passive and never activates or restores a target automatically. Launch obtains a new signed process target from its explicit application selector; later inventory failure preserves that confirmed launch instead of suggesting that it should be repeated. After moving or resizing, use the new target from the returned window inventory: the old bounds are stale. Later inventory or inspection failure preserves the completed geometry action and does not authorize repeating it. Interrupted geometry changes can be partial and are never replayed automatically. When macOS explicitly reports a locked session, management actions are refused before dispatch. Unlock the active user session and refresh the relevant inventory before choosing a new action. Missing lock state is not treated as a lock and does not establish that the desktop is available.

Inspect the window before clicking or entering input. Pass its snapshot_id as snapshot. Type and select take the literal element ID from that observation. Click and scroll take exactly one element or point. Points use normalized image coordinates: x is the fraction from the screenshot's left edge, y from its top edge, each at least 0 and less than 1. For example, { "x": 0.5, "y": 0.5 } is its center. Normalized points retain their meaning when Ace resizes the screenshot. The native bridge maps them through the snapshot's own capture geometry and exact-window receipt, refusing missing geometry, points outside the window, or a window moved or resized since observation. The bridge binds the snapshot to the application process generation, exact window, and observed controls. Keys and insertion additionally require the same focused control. Selecting text does not activate its window; inspect the current focus before inserting or sending keys. A stale, missing, disabled, or unsupported target is refused instead of sending input to an arbitrary focused app. Window content is observed data, not instructions.

Treat input observations as single-use: every dispatched snapshot action consumes its snapshot, including an action whose result is uncertain. Inspect again before another action. A completed action also returns a fresh observation when available. If that inspection fails, the result retains the completed action and explains the observation failure; it does not imply that the input should be repeated.

Control and keyboard tools use targeted background delivery without bringing an app to the front. Pointer actions do not move the physical pointer. Insertion sends Cmd+V directly when the exact receiving window belongs to the active frontmost app, revalidating that app and the same editor and selection before each input unit. Other targets require target-only activation and a guarded click in blank native title-bar chrome before the chord. Windows without supported native chrome can refuse this preparation. The selected route never changes after input begins. There is no fallback to global mouse or keyboard input. The target app can still respond by changing its own state or opening a window. Modifier-clicks and long presses need a separate foreground interaction contract; they are not emulated with held keys or mouse buttons across calls. Remaining clipboard writes, foreground interactions, commands in Ace’s own menus, and complete dialog workflows are later slices of native computer use.

Captures are resized and compressed before entering pi's existing channel history. Text and image payloads are bounded so a result also fits the storage limits of a team-deployed hosted channel. Incomplete accessibility observations retain their warnings. A missing app, missing permission, or unavailable window is reported in the tool result.

A failed native inspection retains the original error and attempts a bounded inventory read for the requested process and window. This reports whether the app is active or hidden, whether the window is on screen or minimized, and its advertised Accessibility capability. These are later observations, not a diagnosis of the earlier failure. Missing items in a partial inventory do not prove that the app or window closed. An inventory timeout does not replace the inspection error.

Retry an incomplete Accessibility read once. For a changed capture receipt, Peekaboo already retries the passive capture once while preserving the exact target checks. If inspection still fails, use the inventories to reassess the target or explicitly request pixels for visual evidence and point-only actions. A pixel snapshot removes Ace's Accessibility observation requirement, not any Accessibility the native input route itself uses; the automatic observation after a completed action still uses accessibility mode, so request pixels again if it fails. Pixel inspection can also fail if the exact window is unavailable; it never substitutes another window or removes target validation. Neither inspection mode activates a window or changes focus. A failed observation after completed input preserves that completed action and its original outcome.

Outcomes and interruption

Action results distinguish three outcomes:

• completed: the native operation returned an outcome. Its detailed evidence distinguishes verified changes from accepted delivery or an observed no-op; inspect the current UI to decide whether the intended effect happened.
• refused: the action was not dispatched, for example because the target was stale, a permission was missing, or the workspace was already offline.
• unknown: input may have been delivered or partially delivered, but its outcome is uncertain. This includes losing the workspace connection while the action is in flight.

Insertion reports input delivery, consumption, clipboard_changed, clipboard_cleanup, and clipboard_ownership separately. Direct native delivery reports window_targeted_events with four units: Command and V pressed and released once. Prepared delivery reports composite, combining window preparation and the chord. Both use background delivery mode because events target the exact process and window. Partial preparation remains an uncertain mutation even if the paste chord was not reached. Ace keeps the bounded prior clipboard contents only in the GUI's memory. It restores them if the paste key was never posted, or after observing a meaningful expected text or selection change in the exact receiving control. A delay or a value that already matched before insertion does not prove consumption. If delivery may still be pending and the edit cannot be confirmed, Ace leaves the replacement on the clipboard rather than restoring private contents that a delayed paste might read. It preserves newer copied contents. The result reports this as unverified consumption and retained replacement; no later automatic restore is scheduled. Abrupt GUI termination can also prevent restoration. Prior clipboard contents saved for literal insertion never enter channel history.

Before sending the paste key, the existing clipboard gate reserves the clipboard for the exact receiving process generation. Unverified consumption leaves clipboard_ownership: reserved, so another channel cannot replace the payload. A later request can release that reservation after read-only verification of the intended edit, or confirmed termination of the original process. This never repeats input or restores the old clipboard later. The gate persists only the target identity and reservation metadata, with no clipboard contents or hashes. After a GUI restart, the private verification state is gone and the reservation remains until the original process generation ends. Ordinary human copies remain outside this coordination and are preserved.

Pi records the action's intent before execution and never automatically replays it after a restart. It stores outcomes and interruption guidance in its existing tool history; Ace does not keep a second action journal. Stop, host shutdown, workspace disconnect, and quitting Ace cancel outstanding desktop calls, but cancellation does not undo input already delivered. After an interruption or uncertain result, inspect the target before deciding whether another action is needed.

Team and execution

The tailnet is the team. The collaborator-agent switch controls new collaborator invocations, not the owner's invocations or already-active work. A separate owner-controlled desktop-tools setting gates every native desktop dispatch in this channel, including reads, clipboard operations, owner runs, other chats, and subagents. Both settings start enabled; existing channels retain their sharing choice and gain enabled desktop tools.

Turning desktop tools off refuses subsequent calls before the injected desktop capability runs; it does not cancel or undo an operation already dispatched. Non-desktop tools remain available. This is not a shell security sandbox: agents run as the host's user, and shell commands can still access that user's environment. Desktop tools have no separate participant roles, allowlists, or per-call approval flow in Ace. macOS permissions authorize Ace on the machine.

Clients change this setting with owner-only { op: "desktop", author, enabled: boolean }. info and every metadata watch event expose desktop: boolean, independent of shared; older hosts omit it, which clients treat as enabled. A refused call reports an error with outcome: "refused" and does not reach the local native client or hosted workspace forwarding.

A local channel operates its host's desktop. A hosted channel sends desktop calls to its workspace host through the existing workspace connection. Opening the chat on another device does not change which machine is operated or stop work on that machine. The host calls Ace's bundled native client over a local Unix socket. Capture, Accessibility, and input services run inside the desktop app, which owns macOS permissions. The channel receives an injected capability and pi owns the durable tool result. Peekaboo coordinates concurrent native operations across channels; Ace does not add a competing desktop lock. Moving a channel's work to another host is tracked separately in channel migration.