Skip to content

feat(cp): OAB runtime CP client — [control_plane] config, worker serving, headless mode (PR 3/4) - #1471

Open
chaodu-agent wants to merge 5 commits into
feat/cp-observerfrom
feat/cp-runtime-client
Open

feat(cp): OAB runtime CP client — [control_plane] config, worker serving, headless mode (PR 3/4)#1471
chaodu-agent wants to merge 5 commits into
feat/cp-observerfrom
feat/cp-runtime-client

Conversation

@chaodu-agent

@chaodu-agent chaodu-agent commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

OAB runtime CP client — [control_plane] config, worker serving, headless mode (PR 3/4)

Stacked on #1470 (feat/cp-observer). PR 1/4 (#1469) is the CP hub; PR 2/4 (#1470) the observer/lobby protocol; this slice connects the OAB runtime itself to the CP. PR 4/4 (MCP facade tools: spawn_agent / check_delegation / list_agents / cancel_delegation + primary-side initiation) follows.

Implements ADR §3 (Registration, OAB side), §3-Headless, and the worker half of §4 — docs/adr/agent-control-plane.md.

Discord Discussion URL: https://discord.com/channels/1491295327620169908/1491365158868619404/1532377620241449040

Review Contract

Goal

An OAB runtime can join the control plane: a new optional [control_plane] config section makes the runtime dial the CP over WebSocket, register with its key-bound identity, heartbeat, and — when type = "worker" — serve incoming cp/delegate requests by running the prompt through its local ACP session pool and replying cp/delegate_result within the deadline. type = "worker" also unlocks headless mode: [agent] + [control_plane] with no platform adapters is a valid boot.

Non-goals

  • Primary-side initiation and the agent-facing MCP facade tools (spawn_agent, …) — PR 4/4 (ADR §6).
  • The Unix domain socket / openab agent CLI — PR 4/4.
  • Durable delegation state across runtime restarts: in-flight work dies with the process; the CP synthesizes target_disconnected (ADR §4 v1 contract).
  • Kiro-style session primitives (inbox, interrupt, broadcast) — explicitly deferred from v1 in the ADR.

Accepted Residual Risks

  • A worker that loses its CP connection abandons in-flight delegations without notifying anyone locally; the CP's target_disconnected synthesis is the single source of truth. Deliberate: per-connection ownership is the ADR's replica-safety rule.
  • The prompt seam refactor (stream_prompt_blocksPromptExecution) touches the shared platform delivery path; safety is argued by mechanical mapping + every pre-existing adapter/dispatch test passing unmodified, not by new platform-level tests.
  • Windows cross-check is UNVERIFIED (build host lacks the x86_64-pc-windows-gnu target); no platform-specific code was added and the crate compiles with --no-default-features.

Acceptance Criteria

  • Absent [control_plane] → zero behavior change (config parse test: None; no task spawned)
  • [agent] + [control_plane type="worker"], no adapters → boots headless; type="primary" alone without adapters still bails; [mcp]-only unchanged; [mcp] + [control_plane] runs facade AND client (run-mode test matrix)
  • Client registers against the REAL openab-cp server in-process (integration test, not a mock): register/ack, heartbeat keeps lease, delegate→result roundtrip, cancel mid-flight, local deadline → Timeout, lease-expiry close → reconnect + re-register succeeds
  • Reconnect: exponential backoff 1/2/4/8/16/30s, shutdown-aware; backoff resets only after a ≥60s session (no reconnect storm against an accept-then-close CP); one instance_id per process across reconnects
  • Executor: local concurrency cap enforced (over-cap / duplicate delegation_id → immediate Failed, never executed, CP capacity released); outcome mapping Completed/Failed/Timeout/Cancelled pinned by unit tests; cancel/deadline cleanup bounded (5s) and the pool session is discarded — no slot leaks
  • Completed result bodies are capped (512 KiB, UTF-8-safe marker) below the CP's max_frame_bytes transport limit — an oversized agent result can no longer drop the connection and kill co-inflight delegations
  • openab-cp gains a default server feature; openab-core depends on it default-features = falsecargo tree shows no axum edge into the runtime; the openab-cp binary still builds with default features; zero Dockerfile changes needed
  • CP auth_key never enters the agent child env (untouched env_clear discipline) and never appears in logs
  • Full matrix on the build host: openab-cp 94 passed; openab-core 712 passed (+35 new; 1 pre-existing macOS-only failure unrelated); root bin 25 passed; integration 7 passed; clippy and rustfmt at exact pre-change baselines

Follow-ups

  • PR 4/4: MCP facade tools + Unix socket + primary-side initiation (ADR §6).
  • ADR §3 example URL said wss://…/acp; the server mounts /cp — example corrected in this PR; revisit if an alias is preferred instead.
  • Consider surfacing delegation-serving activity in the runtime's own logs/metrics once the fleet runs this (observability today is CP-side via feat(cp): observer/lobby — 3-phase design + Phase 1 protocol scaffold (PR 2/4) #1470's lobby events).

At a Glance

[control_plane]                     openab (runtime)                    openab-cp (hub)
url / auth_key / namespace /   ┌──────────────────────────┐       ┌──────────────────────┐
name / type / labels /         │ control_plane::client    │──wss─►│ register → ack       │
max_delegated_sessions    ───► │  dial → register → serve │◄──────│ heartbeat lease      │
                               │  backoff 1..30s, 1 uuid  │       │                      │
absent section = no client,    │ control_plane::executor  │◄──────│ cp/delegate forward  │
existing deployments untouched │  cap check → session per │──────►│ cp/delegate_result   │
                               │  delegation → ACP pool   │       │ (→ lobby cp/event)   │
[agent]+[control_plane worker] │  cancel/deadline bounded │       └──────────────────────┘
= valid headless boot          └──────────────────────────┘

Prior Art & Industry Research

  • The reconnect/backoff loop mirrors the existing standalone gateway WS client (crates/openab-core/src/gateway.rs).
  • The headless run-mode gate extends the facade-only precedent introduced with [mcp] (feat(mcp): facade-only run mode — adapter-less [mcp] config is valid #1453).
  • The delegation executor reuses the prompt path cron jobs already exercise (AdapterRouter + SessionPool), rather than the ACP-over-WS acp_client synthesis path — one seam, no event fabrication.

Proposed Solution

  1. Feature split (crates/openab-cp): default server feature gates axum/registry/router/policy/events/server + the binary; proto (wire types) stays unconditional. The runtime consumes openab-cp with default-features = false — wire types only, no server deps, no new Dockerfile stubs.
  2. Config (crates/openab-core/src/config.rs): ControlPlaneConfig with deny_unknown_fields, CpAgentType (primary|workerobserver unrepresentable by construction), ${ENV} expansion free via the existing pass, validation for empty fields and max_delegated_sessions > 0.
  3. Client (crates/openab-core/src/control_plane/client.rs): process-lifetime instance_id; Bearer auth at upgrade; cp/register first frame; single select! loop over inbound / heartbeat / completion channel / shutdown; on disconnect cancels local work, drains (5s) then aborts, backs off shutdown-aware.
  4. Executor (crates/openab-core/src/control_plane/executor.rs): admission (cap, duplicate) decided before execution; fresh ACP session per delegation keyed control-plane:<sha256(instance_id + delegation_id)>; budget min(deadline − now, prompt_hard_timeout); PromptRunner trait seam so integration tests inject a fake runner while production uses the pool.
  5. Prompt seam (crates/openab-core/src/adapter.rs): stream_prompt_blocks returns PromptExecution { final_text, terminal_error, silent_failure }; platform callers map back mechanically — no observable change, pre-existing tests untouched.
  6. Lifecycle (src/main.rs): client spawned after router construction when the section is present; stopped (or aborted after 10s) before pool.shutdown(); headless run-mode matrix extended.

Alternatives Considered

  • New openab-cp-client crate: rejected — every one of the 19 Dockerfiles stubs workspace crates and would need edits; a module inside openab-core is Dockerfile-neutral and the client is small.
  • Reusing the ACP-over-WS acp_client event-synthesis path for delegated prompts: rejected — it fabricates gateway events to re-enter the dispatcher; the cron precedent calls the router/pool seam directly with less indirection and no fake sender identity.
  • A cargo feature for the client: rejected — the optional config section is already the opt-in; a feature would double the build matrix for no isolation gain (the client adds no heavy deps).
  • CP-side-only concurrency accounting: rejected — the runtime also enforces its cap locally so a CP bug cannot flood a worker; over-cap arrivals get a terminal Failed so CP capacity releases.

Validation

  • cargo test -p openab-cp — 94 passed (server unaffected by the feature split; binary still builds)
  • cargo test -p openab-core — 713 passed, +36 new (config/executor/client incl. the transport-cap regression); 1 failure is the known pre-existing macOS-only secrets::tests::resolve_exec_nonzero_exit, present at baseline
  • cargo test -p openab-core --test cp_client — 7 integration tests against the real in-process CP server: registration, heartbeat/lease, roundtrip, cancel, timeout, lease-expiry reconnect, shutdown deregistration
  • cargo test (root bin) — 25 passed incl. the headless run-mode matrix
  • cargo clippy --workspace --all-targets — warning count identical to baseline (12); cargo fmt --check diff count identical to baseline (217, pre-existing drift untouched)
  • cargo check --workspace --no-default-features — passes
  • cargo check --target x86_64-pc-windows-gnu — UNVERIFIED (target not installed on the build host); no platform-specific code added
  • Independent audit round: four blocking findings (facade-only foreclosing the client, backoff reset storm, unbounded cancel before discard, non-abortable shutdown) — all fixed and re-verified

@chaodu-obk

This comment has been minimized.

@chaodu-obk

This comment has been minimized.

@chaodu-obk

This comment has been minimized.

@chaodu-obk

This comment has been minimized.

@chaodu-obk

This comment has been minimized.

@chaodu-obk

This comment has been minimized.

…er delegation serving, headless mode (PR 3/4)

PR 3/4 of the Agent Control Plane stack: the OAB runtime side. The runtime
dials out to the CP, registers, serves `cp/delegate` through the same ACP
turn driver every chat platform uses, and reports each turn as
`cp/delegate_result`.

Config surface (`[control_plane]`, `crates/openab-core/src/config.rs`)
- `url`, `auth_key`, `namespace`, `name`, `type` (`primary` | `worker`),
  optional `labels`, `max_delegated_sessions` (default 1).
- Presence is the opt-in — no cargo feature — following the `[mcp]`
  precedent. `deny_unknown_fields`, so a mistyped key fails startup instead
  of defaulting silently. `${ENV}` expansion comes from the shared loader.
- `CpAgentType` has only Primary and Worker: `observer` is a read-only lobby
  role, so a runtime claiming it is unrepresentable rather than merely
  validated against.
- Validation: non-empty url/auth_key/namespace/name, and
  `max_delegated_sessions > 0`. Each of these would otherwise fail at
  registration and reconnect forever.

State machine (`control_plane/client.rs`)
- One process-lifetime instance id (uuid), reused across reconnects: it
  identifies the replica, not the connection.
- Connect ws/wss with `Authorization: Bearer` (never in the URL), send
  `cp/register` as the first frame, adopt `effective_max_delegated_sessions`
  from the ack, then one loop over inbound frames / heartbeat tick /
  delegation completions / shutdown watch.
- On close or error: cancel every local in-flight delegation (each stops its
  agent and drops its session; the results are deliberately dropped, since
  the CP synthesizes `target_disconnected`), then 1/2/4/8/16/30s
  shutdown-aware backoff, reconnect, re-register.
- CP-issued requests get JSON-RPC result acks; a delegation's outcome never
  rides in that ack.

Delegation serving (`control_plane/executor.rs`)
- Admission never executes: over-cap or duplicate `delegation_id` returns
  `status = failed` without touching the session pool.
- Bounded by the nearer of the CP deadline and
  `[pool].prompt_hard_timeout_secs`. Outcomes map to
  completed / failed (agent error, broker error, silent failure) /
  timeout / cancelled.
- One fresh session per delegation, keyed
  `control-plane:<sha256(instance_id + delegation_id)>`, discarded on every
  terminal outcome. Adds `SessionPool::discard_session` for the non-cancel
  teardown (`reset_session` would emit a spurious `session/cancel` and log an
  error for the benign already-gone case).

Prompt seam refactor safety
- `AdapterRouter::stream_prompt_blocks` now returns
  `Result<PromptExecution { final_text, terminal_error, silent_failure }>`.
  Nothing else changed inside it: two values are captured before the existing
  composition consumes them, and the delivery paths are byte-for-byte the
  same. `Err` still means what it meant (incomplete user-visible delivery).
- All platform callers map back with `.map(|_| ())`, and `DispatchTarget`
  stays at `Result<()>`, so Discord/Slack/gateway/ambient — and the existing
  dispatch mock — are untouched. Every pre-existing adapter test passes
  unmodified.
- The executor reuses that same driver with a private sink adapter (send-once,
  no placeholder, no reactions) and reads the reply from the return value.

Feature split (Dockerfile-neutral)
- `openab-cp` gains a default `server` feature gating config/registry/policy/
  router/events/server and the binary (`required-features`); `proto` — the
  wire contract — stays unconditional, as do serde/serde_json/chrono.
- `openab-core` depends on `openab-cp` with `default-features = false` for
  wire types only. Verified: `cargo tree -p openab-core -e normal` shows only
  chrono/serde/serde_json under openab-cp, an inverted axum query finds
  nothing, and `cargo build -p openab-cp` still produces the binary.
- No new crate, no new runtime cargo feature.

Headless mode (`src/main.rs`)
- `[agent]` + `[control_plane] type = "worker"` with no adapter is a valid
  boot; a `primary` alone is not (its prompts come from a chat platform).
  `[mcp]`-only is unchanged, and mcp + worker runs both.
- Extracted as `headless_run_mode` so the matrix is unit-tested.
- Client spawned after router construction, wired to the existing shutdown
  watch, and stopped before `pool.shutdown()` so the CP sees a clean
  disconnect rather than a lease timeout.

Security
- `auth_key` is used only for the upgrade header (marked sensitive), is never
  logged, and never enters the agent child env — `[agent].env` plumbing is
  untouched.

Tests
- Config: absent section, full section incl. `type` rename and `${ENV}`,
  defaults, observer refusal, unknown-key refusal, every validation failure.
- Run modes: the full headless matrix.
- Executor: cap enforcement, duplicate id, all outcome mappings, and
  cancel/deadline cleanup asserting the session is gone.
- `crates/openab-core/tests/cp_client.rs`: seven tests against the REAL
  in-process CP (ephemeral loopback port + `run_sweeper`), with a raw-WS
  primary as initiator and a scripted `PromptRunner` at the trait seam —
  register/ack, heartbeats holding the lease, delegate→result round trip,
  local deadline → `timeout`, cancel mid-flight, zero-lease sweep →
  reconnect + re-register (same instance id, and the new registration
  serves), and clean deregistration on shutdown.

Deviations from the design
1. `openab-cp`'s `config.rs` is behind `server` too. The design said "proto.rs
   (+ config types needed by proto) stays unconditional"; proto needs nothing
   from config (the dependency runs the other way), so gating it keeps anyhow
   and toml out of the runtime build.
2. Completed delegations report through an mpsc channel rather than a
   `JoinSet` polled in the `select!`. The inbound branch must spawn while the
   completion branch is armed, and one `select!` cannot lend the same
   `JoinSet` to both — the borrow checker rejects it. Same for the socket,
   which is split into sink/stream halves. Behaviour is unchanged; handles are
   still tracked for the drain-then-abort teardown.
3. `axum` added as a dev-dependency of `openab-core`: `openab_cp::server::app`
   returns an `axum::Router`, and the integration test needs `axum::serve` to
   bind it. Test-only — the runtime links no HTTP server.

Audit round-1 fixes (all four blocking findings):

1. [mcp] + [control_plane type=primary] now takes the full boot path so the
   facade and the CP client both run; facade-only had foreclosed the client.
   Run-mode test updated to pin both-run.
2. Reconnect backoff no longer resets on every clean close: a session must
   live STABLE_SESSION_SECS (60s) before a close resets backoff to 1s, so a
   CP that accepts-then-closes (lease misconfig, crash loop) faces
   1/2/4/8/16/30 escalation instead of a 1s reconnect storm.
3. The executor's best-effort session/cancel before discard is now bounded
   at 5s (the pool's own cleanup bound): a wedged agent stdin can no longer
   block discard and leak the delegation slot. Both sites (cancel, deadline).
4. Shutdown is storm-proof: the dial/register phase races the shutdown
   signal via select (a hung connect cannot stall shutdown), and main aborts
   the client task if it misses the 10s deadline instead of detaching it
   over pool teardown.
Self-review finding. The worker sent the agent's full final text in
cp/delegate_result (SinkAdapter deliberately does no chunking), but the CP
enforces max_frame_bytes (default 1 MiB) at the WS transport BEFORE parsing —
its router-level max_result_bytes truncation can never save an oversized
frame. A single >1 MiB result therefore dropped the whole connection, and
every in-flight delegation on that worker died as target_disconnected.

The executor now caps Completed results at 512 KiB (half the default frame
budget, leaving JSON-envelope headroom) with a UTF-8-boundary-safe truncation
marker; the CP still applies its own, typically smaller, max_result_bytes on
arrival. Regression test covers the oversized, multibyte-straddling, and
under-cap cases.
Self-review round-2 finding. serve() released its inflight slot with a plain
call after the execute().await — but the client ABORTS serving tasks that
outlive the 5s drain window on disconnect, and an aborted task never runs
code after its await point. The leaked entry then survives reconnection
(executor state is process-lifetime), permanently consuming capacity: with
the default max_delegated_sessions = 1, one abort turns the worker into a
zombie that refuses every delegation while heartbeating happily.

The window is realistic, not theoretical: the bounded (5s) session/cancel
added in the previous fix means a wedged agent stdin makes the task overrun
the equally-sized drain window by construction.

release() now lives in a Drop guard taken at admission, so the abort's
unwind frees the slot. Regression test aborts a serving task mid-run and
asserts active() == 0; mutation-verified (plain-release pattern fails it
5/5, the guard passes).
…ch it on cancel

Round-8 made the admission token protocol-visible on the result path;
round-9 extended it to cancellation. The runtime side of both:

- The executor carries forward.admission into all five terminal frame
  kinds (completed, failed, refused-at-admission, timeout, cancelled).
- The worker-side cancel handler matches (delegation_id, admission):
  a stale cancel naming a superseded admission of a reused id is ignored
  instead of aborting the live delegation — the worker-side half of the
  misdelivery the wire token closes. Regression pins both directions.
- The initiator-side integration test echoes the ack's admission in its
  cp/cancel, exercising the CP's required-field validation end-to-end.

The real-server integration suite is the first live client of the
token-enforcing contract on both the result and cancel paths.
@chaodu-obk

This comment has been minimized.

…teardown

Round-7 review fixes:

- F1: failed() now caps every error string (64 KiB, UTF-8-safe) inside the
  constructor so no call site can bypass it; oversized-error regression test
  mirrors the success-path one
- F34: delegation_session_key mixes in the CP admission token, so a
  re-admission of a reusable id can never resume an earlier admission's
  session (e.g. one orphaned by a drain-timeout abort); false reconnect
  claim in the doc corrected; key-property test pins admission scoping
- F22/F31: duplicated cancel+discard blocks extracted into bounded helpers;
  discard now shares the 5s bound instead of being unbounded on every
  terminal path
- F15: client WS now sets max_message_size/max_frame_size (1 MiB),
  mirroring the CP's accept-side max_frame_bytes instead of tungstenite's
  64 MiB default
- F5: the cp/register ack wait is bounded (10s, mirroring the CP's
  register_timeout_secs) so a stalled CP enters backoff instead of hanging
- F27: config/docs no longer claim a local min() the code does not
  implement — the runtime enforces the ack value
- F37: ADR section 7 facade label corrected from PR 3/4 to PR 4/4
@chaodu-obk

chaodu-obk Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

> [!IMPORTANT]
> CHANGES REQUESTED ⚠️ -- Round-8 group review at the new head: the single fix commit verifiably closes both round-7 carried criticals that it targeted (uncapped failure payloads; non-admission-scoped session keys) plus six carried important findings -- but the third round-7 critical, success-path JSON escape inflation past the CP's transport limit, remains open for the second consecutive round, and the fixes introduce three new important residuals.

What This PR Does

Connects the OAB runtime to the Agent Control Plane: a new optional [control_plane] config section makes the runtime dial the CP over WebSocket, register with its key-bound identity, heartbeat, and -- when type = "worker" -- serve cp/delegate requests through its local ACP session pool, replying cp/delegate_result within the deadline. type = "worker" also unlocks a headless boot with no chat adapters.

How It Works

  • openab-cp gains a default server feature; the runtime consumes the crate with default-features = false, so only wire types reach the runtime build -- no axum edge, zero Dockerfile changes.
  • control_plane::client owns the connection state machine: Bearer auth at upgrade, cp/register as first frame (ack wait now bounded at 10s), a single select! serve loop with a 1 MiB inbound frame ceiling, and shutdown-aware backoff (1/2/4/8/16/30s) that resets only after a 60s stable session.
  • control_plane::executor decides admission (capacity + duplicate) before executing, runs each delegation in a fresh pool session keyed by SHA-256 of (instance_id, delegation_id, admission), bounds it by the nearer of the CP deadline and the local prompt hard timeout, and frees slots via an RAII guard that survives task aborts.
  • Every terminal cp/delegate_result echoes the admission token; cp/cancel is matched on (delegation_id, admission); error strings are capped at 64 KiB inside the failed() constructor.

Round-8 Scope Note

Delta vs the round-7 head 655b4248: exactly one fix commit (d3d130b0, 6 files, +141/-49) addressing F1, F5, F15, F22/F31, F27, F34, and F37. Ten review lanes (architecture x2, performance, correctness, readability, safety, docs/UX, security/CI, integration, operability, simplicity) re-verified the claimed fixes and the carried findings line-by-line at this head.

One cross-lane dispute was resolved by coordinator source arbitration: whether the CP-side max_result_bytes (256 KiB) mitigates F46 under default config. It does not -- the server applies max_frame_bytes at the WS transport pre-parse (server.rs:235-238), while the router truncation (router.rs:903-906) runs post-parse and never sees an oversized inbound frame. F46 therefore has no default-config mitigation.

Verified fixed this round: F1 🔴 (error cap in failed(), 64 KiB, UTF-8-safe, regression test), F34 🔴 (admission-scoped session key, key-property test; one doc residual, see F54), F5 (register ack 10s bound), F15 (client WS 1 MiB config; confirmed no Nagle behavior change -- connect_async already passed false internally), F22 (teardown helpers extracted), F27 (capacity docs corrected), F31 (discard bounded; residual, see F52/F53), F37 (ADR facade label).

Findings

# Severity Finding Location
F46 🔴 Carried r7, re-verified by six lanes: success-path escape inflation. cap_result still compares raw UTF-8 text.len() against 512 KiB, but the frame is JSON-escaped before send: "/\ double, control chars inflate 6x (512 KiB of ESC chars -> ~3 MiB serialized). An escape-heavy result passes the cap, exceeds the CP's pre-parse 1 MiB max_frame_bytes, and drops the socket, killing every co-inflight delegation. Exploitability confirmed: final_text flows from raw agent chunks with no ANSI/control-char stripping. The error path is now safe (64 KiB x 6 worst case < 1 MiB); the success path is unchanged. No escape-inflation test exists executor.rs:389-409, client.rs:521
F51 🟡 New: the fixed 1 MiB MAX_INBOUND_FRAME_BYTES hardcodes the CP's default max_frame_bytes as if it were a protocol constant, but the CP exposes it as config (an 8 MiB configuration is tested hub-side). A legally configured larger CP frame is rejected at the worker transport, disconnecting the worker and converting valid delegations to target_disconnected. Same class: REGISTER_TIMEOUT mirrors a configurable server value. Root fix direction: carry the effective limits in RegisterAck (negotiation), or expose a worker-side configurable ceiling with documented compatibility requirements client.rs:52-60,205-217
F52 🟡 New: the teardown bound is doubled, not shared. cancel_and_discard awaits up to 5s for cancel then up to 5s for discard -- ~10s worst case against the client's single 5s DRAIN_TIMEOUT, so a wedged teardown is always aborted on disconnect, and on ordinary cancel/deadline paths a slot can be held ~10s while the CP believes capacity exists. The helper comments ("cannot burn the client's disconnect drain window", "stays under") claim the opposite. Fix: one shared 5s deadline across both steps (discard uses the remainder), plus a stalled-cancel+stalled-discard regression test executor.rs:346-369, client.rs:403-415
F53 🟡 New: bounded_discard runs on the success path before the result is returned -- a wedged discard delays a completed delegation's result by up to 5s while the initiator waits. Pre-existing ordering (the delta improved it from unbounded to 5s); with admission-scoped keys a deferred discard is safe. Fix: spawn the discard off the critical path or reorder to return-result-then-discard executor.rs:334
F54 🟡 New (F34 doc residual): the module-level invariant doc still describes the session key as (instance_id, delegation_id), contradicting the fixed three-parameter key. A maintainer relying on the module doc for session-isolation reasoning would miss the admission scoping executor.rs:9-12
F55 🟡 Promoted from a round-7 non-blocking note: the truncation marker embeds a non-ASCII ellipsis (U+2026) in a machine-truncated wire payload; ASCII ... is the least-surprise choice. (The CP-side marker has the same character -- base-branch scope) executor.rs:393
F2 🟡 Carried: delegation results inherit chat-platform rendering (tool-display prefixes, table conversion) via the fabricated SinkAdapter; platform() is "control-plane" so the platform_is_acp bypass misses executor.rs, adapter.rs
F3 🟡 Carried: narration_display = true leaks inter-tool narration into delegation results adapter.rs:747
F4 🟡 Carried: no worker-role guard -- handle_frame serves cp/delegate regardless of agent_type client.rs
F6 🟡 Carried: no URL scheme validation -- ws:// sends auth_key as a cleartext Bearer header with no warning or opt-in gate; docs present ws:// and wss:// as co-equal client.rs, config.rs
F7 🟡 Carried: a headless worker whose registration is permanently rejected retries forever with no fail-fast, no fatal/retryable distinction, and no health signal; the new register timeout bounds the wait, not the retries client.rs, src/main.rs
F8 🟡 Carried, strengthened: HeadlessMode::ControlPlaneWorker also names the [mcp] + type = "primary" full-boot path; the test a_primary_with_mcp_takes_the_full_boot_path_so_both_run asserts the contradiction concretely. Rename to a behavior-describing variant (FullBoot) -- module-private, zero-cost src/main.rs
F9 🟡 Carried, strengthened: config-reference headless table omits the [mcp] + primary row; the unqualified primary row implies a startup error where the code does a silent full boot docs/config-reference.md
F10 🟡 Carried: operational note understates that expiring either deadline (CP or local) yields Timeout docs/config-reference.md
F11 🟡 Carried: PromptOutcome and PromptExecution carry the same payload with different field names executor.rs, adapter.rs
F12 🟡 Carried: ADR/proto wording misalignments (heartbeat drift wording; "truncated head-first" vs head-keeping code; 512 KiB vs 256 KiB caps undistinguished) agent-control-plane.md, proto.rs
F14 🟡 Carried, narrowed: drain-abort skips cancel/discard, orphaning the pool session and agent child until the TTL sweep; with F34 fixed the orphan can no longer be resumed, but it still occupies pool capacity the CP believes is free client.rs, executor.rs
F16 🟡 Carried: rustls-tls-webpki-roots only, no custom-CA/pin option; private-CA deployments are pushed toward ws://, compounding F6 client.rs, Cargo.toml
F17 🟡 Carried: connect_async has no explicit connect timeout; DNS/TCP blackholes stall the reconnect loop for the OS default client.rs
F18 🟡 Carried: reconnect backoff has no jitter; a CP restart reconnects the fleet in lockstep client.rs
F19 🟡 Carried: outbound cp/register accepts unbounded operator-controlled labels with no serialized-frame budget guard client.rs
F20 🟡 Carried: no CP-client observability -- no connection-state gauge, delegation counters, reconnect counters, or readiness signal client.rs, src/main.rs
F21 🟡 Carried: resilience parameters (max backoff, stable-session threshold, drain timeout) are hardcoded client.rs
F23 🟡 Carried: register() comment says an unexpected pre-ack frame "is a protocol violation" but the code warns and continues client.rs
F24 🟡 Carried: the no-adapter bail message omits that type = "primary" + [mcp] is a valid boot src/main.rs
F25 🟡 Carried: the openab.delegation.v1 context schema is undocumented anywhere in docs/ (zero grep hits at this head) executor.rs
F26 🟡 Carried: openab.delegation.v1 is emitted inside the sender_context envelope, which the turn-boundary ADR promises stays openab.sender.v1; a different schema name in the same envelope is a semantic change, not an additive field executor.rs, turn-boundary-batching.md
F29 🟡 Carried residual: the admission echo is asserted only in the stale-cancel unit test; the real-server roundtrip never asserts params.admission and no happy-path terminal test pins the echo tests
F30 🟡 Carried: std::sync::Mutex + expect("inflight mutex") on all capacity operations including SlotGuard::drop -- a poison turns admit/release/cancel into process panics (panic-in-drop included) executor.rs
F32 🟡 Carried: the CP client JoinHandle is awaited only at shutdown; a panicked or early-returned client task leaves a healthy-looking process that serves nothing src/main.rs
F33 🟡 Carried: the client never validates ack.protocol_version; a version-mismatched CP is accepted silently client.rs
F35 🟡 Carried: cancel() returns false for two semantically distinct cases (not in flight vs stale token) and its doc describes only the first executor.rs
F36 🟡 Carried: ADR/proto say a stale cancel is "identifiable" at the worker but never state the worker-side MUST-match-and-ignore obligation agent-control-plane.md, proto.rs
F38 🟡 Carried: the worker-side 512 KiB result pre-cap and its truncation marker are undocumented; the ADR documents only the CP-side 256 KiB cap executor.rs, docs
F39 🟡 Carried: the 60s stable-session gate before backoff reset is undocumented client.rs, docs
F41 🟡 Carried (nit): timed_out() echoes the admission without the explanatory comment failed() carries executor.rs
F42 🟡 Carried: the config reference presents type = "primary" ("initiates delegations") as current behavior; initiation ships in PR 4/4 -- in this slice both types serve docs/config-reference.md
F43 🟡 Carried: delegation_context_json omits the admission token that the base branch's parent_admission contract (required for PR-4/4 sub-delegation) will need executor.rs
F44 🟡 Carried: ControlPlaneConfig derives Debug with auth_key unredacted -- a latent credential leak on any future {:?} config.rs
F45 🟡 Carried: no test pins the backoff progression, the 30s ceiling, or the 60s reset gate the acceptance criteria present as verified client.rs, tests
F47 🟡 Carried: SinkAdapter::message_limit() returns usize::MAX and the non-streaming path accumulates the full turn text before any cap runs -- a runaway agent can exhaust worker memory pre-cap executor.rs
F48 🟡 Carried: main.rs hand-rolls a CpAgentType -&gt; &amp;str match duplicating the Display impl this PR adds src/main.rs, config.rs
F49 🟡 Carried: the reply-helper ladder collapses a serde failure into a silent FrameAction::Ignore with no log client.rs
F50 🟡 Carried: serve-loop sink.send() calls are unbounded -- one wedged write stalls acks, heartbeats, and results together, surfacing only as CP-side lease expiry client.rs
F13 🟢 Praise: constructor-level error cap (call sites cannot bypass); admission-in-key as minimal reuse of an existing proto type; cap_text generalization and teardown-helper dedup with zero scope creep; exceptional doc density (invariants and threat models, not mechanics); behavioral test names; frame-limit/register-timeout/admission-key cross-system alignment; credential hygiene chain re-verified; F28 admission-scoped cancel still holds --
Finding Details

🔴 F46: Cap the serialized frame, not the raw string

cap_result caps the raw result string at MAX_RESULT_BYTES = 512 KiB, but the sent frame is serde_json::to_string(&amp;JsonRpcRequest {...}): " and \ double, control chars U+0000-U+001F become \uXXXX (6x). Worst cases: 512 KiB of " -> 1,048,576 bytes of field content alone plus envelope (over the CP's default 1 MiB max_frame_bytes); 512 KiB of ESC -> ~3 MiB. The CP rejects the frame at the WS transport pre-parse and closes the connection -- every co-inflight delegation on the worker dies as target_disconnected. This is exploitable with realistic agent output (terminal escapes, quote-heavy JSON code blocks, backslash-heavy shell text): final_text flows from raw agent message chunks with no control-character stripping. The CP-side max_result_bytes truncation cannot mitigate: it runs post-parse and never sees the oversized frame (verified at server.rs:235-238 vs router.rs:903-906).

The fix commit closed the error path (64 KiB budget leaves 6x headroom under 1 MiB) -- the same treatment must reach the success path.

Requested change (in preference order): (1) serialize-then-measure -- enforce the budget against the serialized frame at the send boundary, fixing every path in one place inside the shared cap_text helper; or (2) bound the raw input at a worst-case-safe budget (~170 KiB for 6x control-char inflation, or MAX_RESULT_BYTES / 2 if control characters are stripped first). Either way, add escape-heavy regression tests (quote-dominant and control-char-dominant payloads) asserting the final serialized frame stays under the transport ceiling. Longer term, F51's negotiation fix (carry max_frame_bytes in RegisterAck) gives the cap a contract link instead of a hardcoded guess.

🟡 F51: The frame ceiling is a config value, not a protocol constant

The CP's max_frame_bytes is operator-configurable and hub-side tests exercise 8 MiB, but the worker now hard-rejects anything over 1 MiB, turning a legal large-frame deployment into transport-level disconnects. REGISTER_TIMEOUT (10s) mirrors the CP's configurable register_timeout_secs the same way. The bounds themselves are correct defense (a bounded buffer and a bounded wait beat unbounded ones regardless of mismatch); the gap is the missing contract link. Requested change: advertise/negotiate effective limits in the register exchange, or expose a worker-side ceiling in [control_plane] with documented compatibility requirements; add an integration test with CP ceiling above 1 MiB and a delegate prompt between 1 MiB and that ceiling.

🟡 F52 + F53: Teardown sequencing

F52: cancel_and_discard serializes two independent 5s bounds; the worst case (~10s) exceeds the single 5s drain window the comments claim to stay under, and on ordinary cancel/deadline paths holds the slot ~10s while the CP sees free capacity. Share one deadline across both steps and fix the comments. F53: the success path awaits bounded_discard before returning the result; spawn it off the critical path or reorder -- with admission-scoped keys a deferred discard cannot collide.

🟡 F54 + F55: Small accuracy fixes from the round's own delta

F54: the module-level invariant doc still describes the two-tuple session key; update it to match the three-parameter reality (the fn-level doc is already correct). F55: replace U+2026 with ASCII ... in the truncation marker.

🟡 Carried findings F2-F50

Re-verified individually at this head by the assigned lanes; evidence unchanged from round-7 except as narrowed or strengthened in the table (F8/F9 evidence strengthened, F14 narrowed post-F34). Full explanations remain in the round-7 details and are accurate at this head.

Base-branch scope note

Two observations belong to the hub (PR #1470/#1469 scope), recorded here for cross-reference only: the CP truncates params.result but not params.error post-parse (no server-side error backstop if a non-Rust client omits the client cap), and the CP's own truncation marker also contains U+2026.

Non-blocking suggestions

  • release() removes by id alone -- asymmetric with cancel's admission pairing.
  • The registration loop silently drops Ping frames (no Pong), unlike the serve loop.
  • result_tx capacity is sized from the initial ack and not resized when the CP clamps downward.
  • No integration test covers the main.rs 10s shutdown timeout + abort path.
  • A standalone cargo check -p openab-cp --no-default-features CI job would make the feature-split contract self-documenting.
Baseline Check
  • PR opened: 2026-08-12; reviewed head d3d130b0094d9bed7d719e452c9e94cee67feb7e
  • Stacked PR: base is feat/cp-observer (not the default branch); merge-base 7473568d690ef5d544c2d11bbd81360dbe2ff6b8 equals GitHub's base.sha (clean stack)
  • Diff stat: 16 files, +3100/-34, reviewed locally against base...head only
  • Delta vs round-7 head 655b4248: one fix commit (d3d130b0, 6 files, +141/-49)
  • Net-new value: the runtime-side CP client, delegation executor, [control_plane] config, headless worker mode, and the openab-cp server/proto feature split
  • CI at review time: check (fmt/clippy/test) completed success on this exact SHA; conformance, validate, and validate-packaged-pins green; 14 docker smoke tests success, remainder in progress, none failed
  • Reviewer sandboxes lack a Rust toolchain this round; CI is the compile/test validation source, supplemented by line-level static verification of every claimed fix
What's Good (🟢)
  • The two criticals were fixed at the right layer: the error cap lives inside the failed() constructor so no call site can bypass it, and the admission token rides the existing key hash with no new abstraction -- both with regression tests.
  • Refactors reduce duplication instead of adding structure: cap_text(text, budget) replaces a second truncation copy; cancel_and_discard/bounded_discard replace three duplicated teardown blocks.
  • Cross-system alignment: client frame ceiling, register timeout, and admission-scoped keys now mirror the hub's accept-side contract point-for-point (see F51 for the configurability caveat).
  • No Nagle regression: the switch to connect_async_with_config(request, Some(cfg), false) matches connect_async's own internal default -- verified against the tungstenite source.
  • Documentation density remains exceptional: invariants, threat models, and why-comments at the point of need; behavioral test names read as invariants.
  • Credential hygiene chain re-verified end to end: header-only, set_sensitive(true), absent from the serialized register frame (pinned by test), never logged, agent child starts from env_clear().
  • F28 (admission-scoped cancel) still holds; SlotGuard RAII and the real-server integration suite remain in place.
  • Zero scope creep: the fix commit contains exactly the seven claimed fixes and their tests.

Addressing External Reviewer Feedback

No inline review comments, submitted reviews, or top-level comments from other reviewers existed on this PR at review time. The seven previous consolidated comments are our own rounds 1-7 at earlier heads and are superseded by this round.

5. Three Reasons We Might Not Need This PR

  1. The transport-safety contract is still landing in slices. Results gained a cap in round-6, errors in this round -- and the success-path cap remains measurably wrong (F46) while the ceiling it defends against is a hardcoded guess at a configurable value (F51). The pattern suggests the frame-budget contract should be designed once, at the registration boundary, rather than patched per payload type.
  2. SinkAdapter signals the turn-driving core is not yet factored out of the chat layer. F2, F3, F26, and F47 are symptoms of the same mismatch: a delegation is request/response RPC, not a chat turn without a UI. Merging entrenches the coupling.
  3. Headless workers still ship as observability black boxes. F7, F20, F30, F32, and F50 compound: no fail-fast on permanent rejection, no readiness signal, panic on a poisoned lock, an unobserved client task, and wedged writes that surface only as CP-side lease expiry.

Counterpoint from the same review: the ADR settled the model, the fixes this round were shaped correctly and landed with tests, and multiple lanes again judged the core design (outbound dial, one session per admission, admission before execution, RAII capacity) sound long-term. These are contract-completeness and operability arguments, not verdicts against the feature.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant