feat(cp): OAB runtime CP client — [control_plane] config, worker serving, headless mode (PR 3/4) - #1471
feat(cp): OAB runtime CP client — [control_plane] config, worker serving, headless mode (PR 3/4)#1471chaodu-agent wants to merge 5 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
2330bba to
3f3d181
Compare
3e246c0 to
9dc41e7
Compare
This comment has been minimized.
This comment has been minimized.
3f3d181 to
4a290d3
Compare
9dc41e7 to
c2030f2
Compare
This comment has been minimized.
This comment has been minimized.
4a290d3 to
b23b309
Compare
c2030f2 to
995b16c
Compare
This comment has been minimized.
This comment has been minimized.
b23b309 to
718a9f8
Compare
995b16c to
68dbe7e
Compare
This comment has been minimized.
This comment has been minimized.
718a9f8 to
08fd30c
Compare
68dbe7e to
a120ea6
Compare
This comment has been minimized.
This comment has been minimized.
08fd30c to
7473568
Compare
…er delegation serving, headless mode (PR 3/4)
PR 3/4 of the Agent Control Plane stack: the OAB runtime side. The runtime
dials out to the CP, registers, serves `cp/delegate` through the same ACP
turn driver every chat platform uses, and reports each turn as
`cp/delegate_result`.
Config surface (`[control_plane]`, `crates/openab-core/src/config.rs`)
- `url`, `auth_key`, `namespace`, `name`, `type` (`primary` | `worker`),
optional `labels`, `max_delegated_sessions` (default 1).
- Presence is the opt-in — no cargo feature — following the `[mcp]`
precedent. `deny_unknown_fields`, so a mistyped key fails startup instead
of defaulting silently. `${ENV}` expansion comes from the shared loader.
- `CpAgentType` has only Primary and Worker: `observer` is a read-only lobby
role, so a runtime claiming it is unrepresentable rather than merely
validated against.
- Validation: non-empty url/auth_key/namespace/name, and
`max_delegated_sessions > 0`. Each of these would otherwise fail at
registration and reconnect forever.
State machine (`control_plane/client.rs`)
- One process-lifetime instance id (uuid), reused across reconnects: it
identifies the replica, not the connection.
- Connect ws/wss with `Authorization: Bearer` (never in the URL), send
`cp/register` as the first frame, adopt `effective_max_delegated_sessions`
from the ack, then one loop over inbound frames / heartbeat tick /
delegation completions / shutdown watch.
- On close or error: cancel every local in-flight delegation (each stops its
agent and drops its session; the results are deliberately dropped, since
the CP synthesizes `target_disconnected`), then 1/2/4/8/16/30s
shutdown-aware backoff, reconnect, re-register.
- CP-issued requests get JSON-RPC result acks; a delegation's outcome never
rides in that ack.
Delegation serving (`control_plane/executor.rs`)
- Admission never executes: over-cap or duplicate `delegation_id` returns
`status = failed` without touching the session pool.
- Bounded by the nearer of the CP deadline and
`[pool].prompt_hard_timeout_secs`. Outcomes map to
completed / failed (agent error, broker error, silent failure) /
timeout / cancelled.
- One fresh session per delegation, keyed
`control-plane:<sha256(instance_id + delegation_id)>`, discarded on every
terminal outcome. Adds `SessionPool::discard_session` for the non-cancel
teardown (`reset_session` would emit a spurious `session/cancel` and log an
error for the benign already-gone case).
Prompt seam refactor safety
- `AdapterRouter::stream_prompt_blocks` now returns
`Result<PromptExecution { final_text, terminal_error, silent_failure }>`.
Nothing else changed inside it: two values are captured before the existing
composition consumes them, and the delivery paths are byte-for-byte the
same. `Err` still means what it meant (incomplete user-visible delivery).
- All platform callers map back with `.map(|_| ())`, and `DispatchTarget`
stays at `Result<()>`, so Discord/Slack/gateway/ambient — and the existing
dispatch mock — are untouched. Every pre-existing adapter test passes
unmodified.
- The executor reuses that same driver with a private sink adapter (send-once,
no placeholder, no reactions) and reads the reply from the return value.
Feature split (Dockerfile-neutral)
- `openab-cp` gains a default `server` feature gating config/registry/policy/
router/events/server and the binary (`required-features`); `proto` — the
wire contract — stays unconditional, as do serde/serde_json/chrono.
- `openab-core` depends on `openab-cp` with `default-features = false` for
wire types only. Verified: `cargo tree -p openab-core -e normal` shows only
chrono/serde/serde_json under openab-cp, an inverted axum query finds
nothing, and `cargo build -p openab-cp` still produces the binary.
- No new crate, no new runtime cargo feature.
Headless mode (`src/main.rs`)
- `[agent]` + `[control_plane] type = "worker"` with no adapter is a valid
boot; a `primary` alone is not (its prompts come from a chat platform).
`[mcp]`-only is unchanged, and mcp + worker runs both.
- Extracted as `headless_run_mode` so the matrix is unit-tested.
- Client spawned after router construction, wired to the existing shutdown
watch, and stopped before `pool.shutdown()` so the CP sees a clean
disconnect rather than a lease timeout.
Security
- `auth_key` is used only for the upgrade header (marked sensitive), is never
logged, and never enters the agent child env — `[agent].env` plumbing is
untouched.
Tests
- Config: absent section, full section incl. `type` rename and `${ENV}`,
defaults, observer refusal, unknown-key refusal, every validation failure.
- Run modes: the full headless matrix.
- Executor: cap enforcement, duplicate id, all outcome mappings, and
cancel/deadline cleanup asserting the session is gone.
- `crates/openab-core/tests/cp_client.rs`: seven tests against the REAL
in-process CP (ephemeral loopback port + `run_sweeper`), with a raw-WS
primary as initiator and a scripted `PromptRunner` at the trait seam —
register/ack, heartbeats holding the lease, delegate→result round trip,
local deadline → `timeout`, cancel mid-flight, zero-lease sweep →
reconnect + re-register (same instance id, and the new registration
serves), and clean deregistration on shutdown.
Deviations from the design
1. `openab-cp`'s `config.rs` is behind `server` too. The design said "proto.rs
(+ config types needed by proto) stays unconditional"; proto needs nothing
from config (the dependency runs the other way), so gating it keeps anyhow
and toml out of the runtime build.
2. Completed delegations report through an mpsc channel rather than a
`JoinSet` polled in the `select!`. The inbound branch must spawn while the
completion branch is armed, and one `select!` cannot lend the same
`JoinSet` to both — the borrow checker rejects it. Same for the socket,
which is split into sink/stream halves. Behaviour is unchanged; handles are
still tracked for the drain-then-abort teardown.
3. `axum` added as a dev-dependency of `openab-core`: `openab_cp::server::app`
returns an `axum::Router`, and the integration test needs `axum::serve` to
bind it. Test-only — the runtime links no HTTP server.
Audit round-1 fixes (all four blocking findings):
1. [mcp] + [control_plane type=primary] now takes the full boot path so the
facade and the CP client both run; facade-only had foreclosed the client.
Run-mode test updated to pin both-run.
2. Reconnect backoff no longer resets on every clean close: a session must
live STABLE_SESSION_SECS (60s) before a close resets backoff to 1s, so a
CP that accepts-then-closes (lease misconfig, crash loop) faces
1/2/4/8/16/30 escalation instead of a 1s reconnect storm.
3. The executor's best-effort session/cancel before discard is now bounded
at 5s (the pool's own cleanup bound): a wedged agent stdin can no longer
block discard and leak the delegation slot. Both sites (cancel, deadline).
4. Shutdown is storm-proof: the dial/register phase races the shutdown
signal via select (a hung connect cannot stall shutdown), and main aborts
the client task if it misses the 10s deadline instead of detaching it
over pool teardown.
Self-review finding. The worker sent the agent's full final text in cp/delegate_result (SinkAdapter deliberately does no chunking), but the CP enforces max_frame_bytes (default 1 MiB) at the WS transport BEFORE parsing — its router-level max_result_bytes truncation can never save an oversized frame. A single >1 MiB result therefore dropped the whole connection, and every in-flight delegation on that worker died as target_disconnected. The executor now caps Completed results at 512 KiB (half the default frame budget, leaving JSON-envelope headroom) with a UTF-8-boundary-safe truncation marker; the CP still applies its own, typically smaller, max_result_bytes on arrival. Regression test covers the oversized, multibyte-straddling, and under-cap cases.
Self-review round-2 finding. serve() released its inflight slot with a plain call after the execute().await — but the client ABORTS serving tasks that outlive the 5s drain window on disconnect, and an aborted task never runs code after its await point. The leaked entry then survives reconnection (executor state is process-lifetime), permanently consuming capacity: with the default max_delegated_sessions = 1, one abort turns the worker into a zombie that refuses every delegation while heartbeating happily. The window is realistic, not theoretical: the bounded (5s) session/cancel added in the previous fix means a wedged agent stdin makes the task overrun the equally-sized drain window by construction. release() now lives in a Drop guard taken at admission, so the abort's unwind frees the slot. Regression test aborts a serving task mid-run and asserts active() == 0; mutation-verified (plain-release pattern fails it 5/5, the guard passes).
…ch it on cancel Round-8 made the admission token protocol-visible on the result path; round-9 extended it to cancellation. The runtime side of both: - The executor carries forward.admission into all five terminal frame kinds (completed, failed, refused-at-admission, timeout, cancelled). - The worker-side cancel handler matches (delegation_id, admission): a stale cancel naming a superseded admission of a reused id is ignored instead of aborting the live delegation — the worker-side half of the misdelivery the wire token closes. Regression pins both directions. - The initiator-side integration test echoes the ack's admission in its cp/cancel, exercising the CP's required-field validation end-to-end. The real-server integration suite is the first live client of the token-enforcing contract on both the result and cancel paths.
a120ea6 to
655b424
Compare
This comment has been minimized.
This comment has been minimized.
…teardown Round-7 review fixes: - F1: failed() now caps every error string (64 KiB, UTF-8-safe) inside the constructor so no call site can bypass it; oversized-error regression test mirrors the success-path one - F34: delegation_session_key mixes in the CP admission token, so a re-admission of a reusable id can never resume an earlier admission's session (e.g. one orphaned by a drain-timeout abort); false reconnect claim in the doc corrected; key-property test pins admission scoping - F22/F31: duplicated cancel+discard blocks extracted into bounded helpers; discard now shares the 5s bound instead of being unbounded on every terminal path - F15: client WS now sets max_message_size/max_frame_size (1 MiB), mirroring the CP's accept-side max_frame_bytes instead of tungstenite's 64 MiB default - F5: the cp/register ack wait is bounded (10s, mirroring the CP's register_timeout_secs) so a stalled CP enters backoff instead of hanging - F27: config/docs no longer claim a local min() the code does not implement — the runtime enforces the ack value - F37: ADR section 7 facade label corrected from PR 3/4 to PR 4/4
|
> [!IMPORTANT] What This PR DoesConnects the OAB runtime to the Agent Control Plane: a new optional How It Works
Round-8 Scope NoteDelta vs the round-7 head One cross-lane dispute was resolved by coordinator source arbitration: whether the CP-side Verified fixed this round: F1 🔴 (error cap in Findings
Finding Details🔴 F46: Cap the serialized frame, not the raw string
The fix commit closed the error path (64 KiB budget leaves 6x headroom under 1 MiB) -- the same treatment must reach the success path. Requested change (in preference order): (1) serialize-then-measure -- enforce the budget against the serialized frame at the send boundary, fixing every path in one place inside the shared 🟡 F51: The frame ceiling is a config value, not a protocol constantThe CP's 🟡 F52 + F53: Teardown sequencingF52: 🟡 F54 + F55: Small accuracy fixes from the round's own deltaF54: the module-level invariant doc still describes the two-tuple session key; update it to match the three-parameter reality (the fn-level doc is already correct). F55: replace U+2026 with ASCII 🟡 Carried findings F2-F50Re-verified individually at this head by the assigned lanes; evidence unchanged from round-7 except as narrowed or strengthened in the table (F8/F9 evidence strengthened, F14 narrowed post-F34). Full explanations remain in the round-7 details and are accurate at this head. Base-branch scope noteTwo observations belong to the hub (PR #1470/#1469 scope), recorded here for cross-reference only: the CP truncates Non-blocking suggestions
Baseline Check
What's Good (🟢)
Addressing External Reviewer FeedbackNo inline review comments, submitted reviews, or top-level comments from other reviewers existed on this PR at review time. The seven previous consolidated comments are our own rounds 1-7 at earlier heads and are superseded by this round. 5. Three Reasons We Might Not Need This PR
Counterpoint from the same review: the ADR settled the model, the fixes this round were shaped correctly and landed with tests, and multiple lanes again judged the core design (outbound dial, one session per admission, admission before execution, RAII capacity) sound long-term. These are contract-completeness and operability arguments, not verdicts against the feature. |
OAB runtime CP client — [control_plane] config, worker serving, headless mode (PR 3/4)
Implements ADR §3 (Registration, OAB side), §3-Headless, and the worker half of §4 —
docs/adr/agent-control-plane.md.Discord Discussion URL: https://discord.com/channels/1491295327620169908/1491365158868619404/1532377620241449040
Review Contract
Goal
An OAB runtime can join the control plane: a new optional
[control_plane]config section makes the runtime dial the CP over WebSocket, register with its key-bound identity, heartbeat, and — whentype = "worker"— serve incomingcp/delegaterequests by running the prompt through its local ACP session pool and replyingcp/delegate_resultwithin the deadline.type = "worker"also unlocks headless mode:[agent]+[control_plane]with no platform adapters is a valid boot.Non-goals
spawn_agent, …) — PR 4/4 (ADR §6).openab agentCLI — PR 4/4.target_disconnected(ADR §4 v1 contract).Accepted Residual Risks
target_disconnectedsynthesis is the single source of truth. Deliberate: per-connection ownership is the ADR's replica-safety rule.stream_prompt_blocks→PromptExecution) touches the shared platform delivery path; safety is argued by mechanical mapping + every pre-existing adapter/dispatch test passing unmodified, not by new platform-level tests.x86_64-pc-windows-gnutarget); no platform-specific code was added and the crate compiles with--no-default-features.Acceptance Criteria
[control_plane]→ zero behavior change (config parse test:None; no task spawned)[agent]+[control_plane type="worker"], no adapters → boots headless;type="primary"alone without adapters still bails;[mcp]-only unchanged;[mcp]+[control_plane]runs facade AND client (run-mode test matrix)openab-cpserver in-process (integration test, not a mock): register/ack, heartbeat keeps lease, delegate→result roundtrip, cancel mid-flight, local deadline →Timeout, lease-expiry close → reconnect + re-register succeedsinstance_idper process across reconnectsdelegation_id→ immediateFailed, never executed, CP capacity released); outcome mapping Completed/Failed/Timeout/Cancelled pinned by unit tests; cancel/deadline cleanup bounded (5s) and the pool session is discarded — no slot leaksCompletedresult bodies are capped (512 KiB, UTF-8-safe marker) below the CP'smax_frame_bytestransport limit — an oversized agent result can no longer drop the connection and kill co-inflight delegationsopenab-cpgains a defaultserverfeature;openab-coredepends on itdefault-features = false—cargo treeshows no axum edge into the runtime; theopenab-cpbinary still builds with default features; zero Dockerfile changes neededauth_keynever enters the agent child env (untouchedenv_cleardiscipline) and never appears in logsFollow-ups
wss://…/acp; the server mounts/cp— example corrected in this PR; revisit if an alias is preferred instead.At a Glance
Prior Art & Industry Research
crates/openab-core/src/gateway.rs).[mcp](feat(mcp): facade-only run mode — adapter-less [mcp] config is valid #1453).AdapterRouter+SessionPool), rather than the ACP-over-WSacp_clientsynthesis path — one seam, no event fabrication.Proposed Solution
crates/openab-cp): defaultserverfeature gates axum/registry/router/policy/events/server + the binary;proto(wire types) stays unconditional. The runtime consumesopenab-cpwithdefault-features = false— wire types only, no server deps, no new Dockerfile stubs.crates/openab-core/src/config.rs):ControlPlaneConfigwithdeny_unknown_fields,CpAgentType(primary|worker—observerunrepresentable by construction),${ENV}expansion free via the existing pass, validation for empty fields andmax_delegated_sessions > 0.crates/openab-core/src/control_plane/client.rs): process-lifetimeinstance_id; Bearer auth at upgrade;cp/registerfirst frame; single select! loop over inbound / heartbeat / completion channel / shutdown; on disconnect cancels local work, drains (5s) then aborts, backs off shutdown-aware.crates/openab-core/src/control_plane/executor.rs): admission (cap, duplicate) decided before execution; fresh ACP session per delegation keyedcontrol-plane:<sha256(instance_id + delegation_id)>; budgetmin(deadline − now, prompt_hard_timeout);PromptRunnertrait seam so integration tests inject a fake runner while production uses the pool.crates/openab-core/src/adapter.rs):stream_prompt_blocksreturnsPromptExecution { final_text, terminal_error, silent_failure }; platform callers map back mechanically — no observable change, pre-existing tests untouched.src/main.rs): client spawned after router construction when the section is present; stopped (or aborted after 10s) beforepool.shutdown(); headless run-mode matrix extended.Alternatives Considered
openab-cp-clientcrate: rejected — every one of the 19 Dockerfiles stubs workspace crates and would need edits; a module insideopenab-coreis Dockerfile-neutral and the client is small.acp_clientevent-synthesis path for delegated prompts: rejected — it fabricates gateway events to re-enter the dispatcher; the cron precedent calls the router/pool seam directly with less indirection and no fake sender identity.Failedso CP capacity releases.Validation
cargo test -p openab-cp— 94 passed (server unaffected by the feature split; binary still builds)cargo test -p openab-core— 713 passed, +36 new (config/executor/client incl. the transport-cap regression); 1 failure is the known pre-existing macOS-onlysecrets::tests::resolve_exec_nonzero_exit, present at baselinecargo test -p openab-core --test cp_client— 7 integration tests against the real in-process CP server: registration, heartbeat/lease, roundtrip, cancel, timeout, lease-expiry reconnect, shutdown deregistrationcargo test(root bin) — 25 passed incl. the headless run-mode matrixcargo clippy --workspace --all-targets— warning count identical to baseline (12);cargo fmt --checkdiff count identical to baseline (217, pre-existing drift untouched)cargo check --workspace --no-default-features— passescargo check --target x86_64-pc-windows-gnu— UNVERIFIED (target not installed on the build host); no platform-specific code added