You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b244c30
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/architecture/computer-use-control.md
+41-24Lines changed: 41 additions & 24 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -156,19 +156,35 @@ background input.
156
156
157
157
## Ownership and execution
158
158
159
-
Portable DTOs and `AppInputAction` live in `tool-contracts`; the host port lives in
160
-
Core. Desktop owns native resources and platform adapters. Assembly must not own
161
-
OS APIs, native focus workarounds or concrete capture implementations.
162
-
163
-
The shared action catalog and structural target validation live in
164
-
`tool-contracts`. Desktop's `control_session/admission.rs` is a deterministic
165
-
owner/mode/lease state machine; its facade alone performs native cleanup and
166
-
carries tokens across workers. `desktop_host/observation.rs` coordinates native
167
-
reads, while `observation_cache.rs` stores bounded references without owning input
168
-
permission. `computer_use_geometry` adapts the independent `computer-use-rust`
169
-
project's image/content/surface mapping for all three platforms. The reference
170
-
project's macOS-only capture/input and standalone MCP/session shell are not a
171
-
replacement for the existing Windows, Linux or product runtime providers.
159
+
This section details the split stated in
160
+
[Native provider ownership](#native-provider-ownership). Core does not depend on
161
+
`services/computer-use`; it reaches the provider only through the injected port.
162
+
163
+
| Owner | Owns | Must not |
164
+
|---|---|---|
165
+
|`tool-contracts` (`computer_use*` modules) | Portable DTOs, `AppInputAction` and its structural target validation, the shared action catalog (`ControlActionPolicy`), control and permission facts, `CaptureGeometry`, the settle decision and the `ComputerUseHost` port | Hold OS handles or make permission decisions |
166
+
|`services/computer-use`| Native resources and platform adapters for Windows, macOS and Linux: capture, AX/UIA/AT-SPI, input, OS permission probes and requests, remembered grants, the control session and `NativeComputerUseHost`; `host_runtime` holds CLI graphical-session admission and the macOS main run loop | Import Core, Tauri or the CLI, or discover Core config |
167
+
| Core (Assembly) | The ComputerUse tool and its permission policy, the `computer_use_program` executor, the `ai.computer_use_enabled`, missing-provider and remote-workspace gates, and release of a turn's control through the port; its `computer_use_host` module only re-exports that port | Own OS APIs, native focus workarounds or concrete capture implementations |
168
+
| Desktop | Provider injection and its diagnostic directory, the Tauri status, permission, sharing and preview commands, and the devtools-only full Tool fixtures | Reintroduce a native implementation |
169
+
| CLI | Provider injection and its diagnostic directory for chat/exec inside a graphical login session, the main-thread run-loop adapter and the `computer-use` command | Reintroduce a native implementation, or infer a provider for Dispatch, ACP or the stdio server |
`control_session/admission.rs` is a deterministic owner/mode/lease state machine
174
+
with no OS handles; its facade `control_session.rs` alone performs native
175
+
cleanup and carries tokens across workers. `desktop_host/` implements the port
176
+
for `NativeComputerUseHost`. The directory kept its name when it moved out of
177
+
the Desktop app; Desktop and CLI share it. `desktop_host/observation.rs`
178
+
coordinates native reads, while `observation_cache.rs` stores bounded references
179
+
without owning input permission. The
180
+
[Native Computer Use guide](../../src/crates/services/computer-use/AGENTS.md)
181
+
has the full module map.
182
+
183
+
`tool-contracts::computer_use_geometry` adapts the independent
184
+
`computer-use-rust` project's image/content/surface mapping for all three
185
+
platforms. The reference project's macOS-only capture/input and standalone
186
+
MCP/session shell are not a replacement for the existing Windows, Linux or
187
+
product runtime providers.
172
188
173
189
### Reference implementation migration
174
190
@@ -178,8 +194,8 @@ runtime, build, test or filesystem dependency of OpenBitFun.
178
194
179
195
| Area | Reference source | OpenBitFun owner and adaptation |
180
196
|---|---|---|
181
-
| Permission query/request |`src/native/permissions.rs`, native permission probes |`computer_use_permissions` contracts, Desktop `permissions.rs`, `ComputerUsePermissions.tsx`; four separate read-only probes and explicit request receipts; Windows capability/UIPI facts and Linux AT-SPI/session consent replace macOS-specific assumptions |
182
-
| App sharing and control grants |`src/sharing.rs`, `src/grants.rs`, sharing controls in `src/gui.rs`| Admission state machine plus Desktop`sharing.rs`, `shared_preview.rs`, `process_identity.rs`, `grants.rs`, `share_grants.rs`; user-selected native window, observe first, separate revocable grant, executable-bound remembered preference, manual preview gestures |
197
+
| Permission query/request |`src/native/permissions.rs`, native permission probes |`computer_use_permissions` contracts, native service `permissions.rs`, Web`ComputerUsePermissions.tsx`; four separate read-only probes and explicit request receipts; Windows capability/UIPI facts and Linux AT-SPI/session consent replace macOS-specific assumptions |
198
+
| App sharing and control grants |`src/sharing.rs`, `src/grants.rs`, sharing controls in `src/gui.rs`| Admission state machine plus native service`sharing.rs`, `shared_preview.rs`, `process_identity.rs`, `grants.rs`, `share_grants.rs`; user-selected native window, observe first, separate revocable grant, executable-bound remembered preference, manual preview gestures |
183
199
| Coordinate conversion |`CoordinateMap` in `src/sharing.rs`| Portable `CaptureGeometry` and Web `previewGeometry.ts`; image/content/global/preview round trips, aspect fit, negative origins, independent X/Y scales, strict padding rejection and immutable screenshot identity |
184
200
| Virtual mouse |`src/cursor.rs`, GUI cursor rendering | Web `VirtualCursor.ts` ports the deterministic physical model, rendered by `PreviewCursor.tsx`; 63 Rust-generated reference samples cover movement, retargeting, immediate placement, resize and visibility. Existing passive native overlays remain platform adapters |
185
201
| Background control | Native directed/semantic input and desktop-state comparison | Existing AX/UIA/AT-SPI and directed-input providers plus `background_guard.rs`; diagnostic focus/pointer/window-order evidence, explicit unsupported routes, no restore or replay. Concurrent human input does not revoke Agent control. The reference's implicit Command modifier is not copied because it changes the requested click |
@@ -407,15 +423,16 @@ until the next information boundary and reuses the returned observation. It does
407
423
not reduce the user's selected reasoning effort or impose step/retry-count caps.
408
424
409
425
`get_app_state`, `app_batch` and single app-input aliases accept optional
410
-
`include_image` (default `true`). With `false`, Desktop reads AX and refreshes
411
-
observation/node identities without retrieving or encoding a model screenshot.
412
-
This is suitable for sufficient semantic state or known keyboard flows. Canvas,
413
-
visual verification and selecting image coordinates still need an image-bearing
414
-
observation. Target preparation, control leases, cancellation and native consent
415
-
remain unchanged; AX-only is not a different input route. Legacy port providers
416
-
can retain their old capture path and omit the image from its result. Existing
417
-
callers/persisted tool calls omit the field and retain their previous behavior.
418
-
The option is local tool presentation, not a new remote GUI capability.
426
+
`include_image` (default `true`). With `false`, the native provider reads AX and
427
+
refreshes observation/node identities without retrieving or encoding a model
428
+
screenshot. This is suitable for sufficient semantic state or known keyboard
429
+
flows. Canvas, visual verification and selecting image coordinates still need an
430
+
image-bearing observation. Target preparation, control leases, cancellation and
431
+
native consent remain unchanged; AX-only is not a different input route. Legacy
432
+
port providers can retain their old capture path and omit the image from its
433
+
result. Existing callers/persisted tool calls omit the field and retain their
434
+
previous behavior. The option is local tool presentation, not a new remote GUI
435
+
capability.
419
436
420
437
Program receipts include `observation_elapsed_ms` alongside per-step and total
421
438
elapsed time. These measure host execution, not model reasoning or end-to-end
0 commit comments