fix(web-security): align Caido surfaces on the 0.57 replay schema - #124
Merged
GangGreenTemperTatum merged 3 commits intoAug 20, 2026
Merged
Conversation
The vendored caido-mode skill already matches caido/skills@41697d8 (PR #22) byte-for-byte, so there was no content to sync. What PR #22 surfaced is that its central fix -- "restore Caido 0.57 compatibility (SDK 0.2.0 -> 0.4.0)" -- landed on only one of the four Caido surfaces this capability ships. Caido 0.57 changed the replay schema: ReplaySession became an interface, createReplaySession requires `kind: ReplaySessionKind!`, and sending moved to a task model. Bring the rest of the stack onto that side of the break. Version contract - caido-cli pin 0.45.0 -> 0.57.1 (both linux arches verified downloadable). The server was two years behind the client the skill runs. - caido-sdk-client floored at >=0.3.0 in capability.yaml, mcp/caido.py's PEP-723 header, and Dockerfile.runtime. A bare spec resolves 0.2.0, which hardcodes the pre-0.57 documents and fails with "Unknown field collection on type ReplaySession". 0.3.0 added the transport/latest vs transport/v0_56 split that negotiates per instance. Latent bug: caido_replay_request never worked mcp/caido.py passed ReplaySendOptions(host=, port=, tls=) and read result.task_status. The real dataclass is (raw, connection, settings) with connection as ConnectionInfoInput(host, port, is_tls), and the result field is `status`. This raised TypeError on every call, on 0.2.0 and 0.3.0 alike. It survived because the existing tests stub the SDK with MagicMock, which accepts any kwargs. Same wrong call was copied into the caido-sdk skill's example, which agents run verbatim. Skills - caido-mode: Dreadnode skill format -- drop upstream's `tags:` (not read by the loader; carried as metadata.role), add compatibility + provenance metadata, and a re-sync note listing the local fork points. Replace the three-surface blurb with a four-surface routing table and an auth-isolation table showing why the credential stores don't collide. - caido-sdk: correct the replay example, add an "API shapes that bite" table and the version floor. - caido-proxy: the MCP Tools section documented `mcp__caido__*` names that exist on neither server (Automate et al are caido-go only). Replaced with the real 9-tool Python surface taken from source, plus a two-server routing table, the access-token-vs-Cloud-PAT distinction, and redaction notes. Tests (+377 total, all green) - test_caido_mode_skill.py: skill tree, Dreadnode frontmatter, the version contract across all declaration sites, provisioning, and cross-surface coexistence. Also validates the caido-sdk example's constructor kwargs and the caido-proxy tool names against mcp/caido.py by AST, so docs can't drift from code. - test_caido_mcp.py: static TestCaidoSdkApiContract that catches the class of bug the mocks hid. Each guard was mutation-tested by reintroducing the defect. Also drop tools/caido_proxy.py (a `CaidoTools = None` stub referenced only by its own test), clearing the sole real --strict validate warning; true up the skill count to 81; bump to 1.10.0.
Runtime verification against a live Caido 0.57.1 instance, which confirmed the
0.57 schema break empirically (ReplaySession is an INTERFACE exposing only
id/name/rank; ReplaySessionKind = HTTP|WS) and turned up two further bugs that
static analysis could not see.
1. Replay sessions must be seeded
On >= 0.57 `replay.send()` updates the draft of an EXISTING entry, then
starts a replay task. A bare `sessions.create()` yields a session with no
entries, so every send aborted with "Replay session has no entries". Create
the session with CreateReplaySessionFromRaw so it has its first entry.
2. `replay.send()` can hang forever
After starting the task the SDK waits on a task-finished *subscription*. If
the target responds before that subscription is established the event is
missed and the await never returns -- reproduced ~2 in 3 runs against a
localhost target; a 3s-delay target always succeeded, confirming the race.
An MCP tool that hangs is worse than one that errors, so the send is now
bounded by REPLAY_SEND_TIMEOUT and falls back to reading the session's
newest entry. The request really was sent, so recovery returns the true
result rather than a false failure.
Recovery re-fetches via `replay.entries.get(id)`: entries listed off a
session carry no response body, and `response` hangs off the entry, not off
entry.request.
3. Status enum leaked its repr
`status` arrives as TaskStatus, whose str() is "TaskStatus.DONE". Unwrapped
to the bare value so output is stable across SDK versions.
Verified end-to-end: the replay traversed Caido and hit the target, which
returned {"cap1187":"replay-ok"} -- observed in both the normal and the
recovery path. caido_health, caido_search_requests, caido_get_request,
caido_list_scopes, caido_list_findings, caido_replay_sessions and
caido_create_finding were all exercised against the same instance. The one
finding created by that check was deleted afterwards; the operator's token
file was copied, never written (sha verified unchanged).
Tests: 382 passing (+5). New AST-based assertions on the body of
caido_replay_request cover seeding, the timeout bound, the fallback path, the
entry re-fetch, and enum unwrapping; the SDK stub gained the two new imports.
Each was mutation-tested. The caido-sdk skill's example and its "API shapes
that bite" section were updated to match, and all documented constructions are
validated against caido-sdk-client 0.3.0.
PR #125 (HTTP desync tooling) merged to main first and shipped 1.10.0, so this branch's own 1.10.0 bump collided. Resolution: version 1.10.0 -> 1.11.0, since 1.10.0 is already published on main. Both branches independently claimed 1.10.0; git auto-merged the identical strings without flagging it, so the bump would have silently republished a shipped version. description keeps both contributions — "HTTP desync/request smuggling" from #125 and the four-Caido-surfaces wording from this branch. Playbook count set to 82, the actual skill directory count on the merged tree (both sides were stale: "81" here, "80+" on main). No code conflicts: the two PRs touch disjoint files apart from the manifest. Verified on the merged tree: 479 tests pass (429 from main + 50 Caido tests from this branch), capability validates at 1.11.0 with 48 tools / 82 skills, and the caido_proxy "defined no tools" warning present on main is resolved by this branch's changes.
Contributor
Author
|
Merged Version bumped 1.10.0 → 1.11.0. PR #125 merged first and shipped 1.10.0, so this branch's own 1.10.0 bump would have republished an already-shipped version. Worth flagging: both branches independently claimed 1.10.0 and git auto-merged the identical version strings without marking them as a conflict — only the adjacent description lines conflicted. The stale version would have gone unnoticed if the description hadn't also diverged. Description keeps both contributions — No code conflicts — the two PRs touch disjoint files apart from the manifest. Verified on the merged tree:
Status is now |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
caido/skills#22 turns out to be already vendored — our
skills/caido-mode/is byte-identical tocaido/skills@41697d8, and upstreammainis that merge. There was no content to sync.The value was in what the PR implied. Its central commit —
e60bb64"restore Caido 0.57 compatibility (SDK 0.2.0 → 0.4.0)" — had landed on only one of the four Caido surfaces this capability ships.Caido 0.57 changed the replay schema:
ReplaySessionbecame an interface,createReplaySessionrequireskind: ReplaySessionKind!, and sending moved to a task model (startReplayTask). This PR brings the rest of the stack onto that side of the break.Version contract
caido-cli(server)caido-sdk-client(Python)@caido/sdk-client(caido-mode)caido-mcp-server(Go)The server was ~2 years behind the client the skill runs. A bare
caido-sdk-clientspec resolves 0.2.0, which hardcodes the pre-0.57 documents and fails withUnknown field collection on type ReplaySession; 0.3.0 added thetransform/latestvstransport/v0_56split that negotiates per instance. Both 0.57.1 Linux arches were verified downloadable.Latent bugs:
caido_replay_requestnever workedRuntime-verified against a live Caido 0.57.1 instance. Introspection confirmed the schema break empirically —
ReplaySessionis anINTERFACEexposing onlyid/name/rank, andReplaySessionKind=HTTP|WS. Testing then surfaced two further bugs static analysis could not see.1. Wrong constructor + wrong result field
Chasing the version thread surfaced a live bug in
mcp/caido.py:The real dataclass is
(raw, connection, settings)withconnection=ConnectionInfoInput(host, port, is_tls=...), and the result field isstatus. Confirmed raisingTypeErroron both 0.2.0 and 0.3.0 — this tool has never functioned.It survived because the existing tests stub the SDK with
MagicMock, which accepts any kwargs. The same wrong call was copy-pasted into thecaido-sdkskill's example — code agents run verbatim.2. Replay sessions must be seeded
On >= 0.57
send()updates the draft of an existing entry. A baresessions.create()yields a session with no entries, so every send aborted withReplay session has no entries. Fixed by creating the session withCreateReplaySessionFromRaw.3.
replay.send()can hang foreverAfter starting the task the SDK waits on a task-finished subscription. If the target responds before that subscription is established, the event is missed and the await never returns — reproduced ~2 in 3 runs against a localhost target, while a 3s-delay target succeeded every time, confirming the race.
An MCP tool that hangs is worse than one that errors, so the send is now bounded and falls back to reading the session's newest entry. The request really was sent, so recovery returns the true result rather than a false failure.
End-to-end proof: the replay traversed Caido and hit the target, which returned
{"cap1187":"replay-ok"}— observed via both the normal and the recovery path.Skills
caido-mode— aligned to the Dreadnode skill format. Dropped upstream'stags:(the loader ignores it; carried asmetadata.role), addedcompatibility+ provenancemetadata, and a re-sync note listing the local fork points so the next upstream pull doesn't clobber them. Replaced the three-surface blurb with a four-surface routing table and an auth-isolation table showing why the credential stores don't collide.caido-sdk— corrected the replay example, added an "API shapes that bite" table and the version floor.caido-proxy— the MCP Tools section documentedmcp__caido__*names that exist on neither server (Automate et al. arecaido-goonly). Replaced with the real 9-tool Python surface extracted from source, plus a two-server routing table, the access-token-vs-Cloud-PAT distinction, and redaction notes.Tests
382 passing (up from 362).
test_caido_mode_skill.py(new) — skill tree, Dreadnode frontmatter, the version contract across all declaration sites, provisioning contract, cross-surface coexistence. Also validates thecaido-sdkexample's constructor kwargs and thecaido-proxytool names againstmcp/caido.pyby AST, so the docs cannot drift from the code again.test_caido_mcp.py— added a staticTestCaidoSdkApiContractthat catches the exact class of bug the mocks hid, including AST-based assertions on the body ofcaido_replay_requestcovering seeding, the timeout bound, the fallback path, the entry re-fetch, and enum unwrapping.Every guard was mutation-tested: each defect was reintroduced and the corresponding test confirmed to fail.
Also
tools/caido_proxy.py(aCaidoTools = Nonestub referenced only by its own test) — clears the sole real--strictvalidate warning.Test plan
pytest tests/→ 382 passeddreadnode capability validate --strict→ 0 failed (only environmentalcaido-cli/burpbinary checks error locally)ruff check --select E9,F63,F7,F82→ cleannpm test→ 68/68caido-sdk-client0.3.0caido_replay_requestround-trips end-to-end (normal + recovery path). Also exercised:caido_health,caido_search_requests,caido_get_request,caido_list_scopes,caido_list_findings,caido_replay_sessions,caido_create_finding.Testing hygiene
The instance used was the operator's own, so the run was kept non-destructive: the token file was copied and referenced via
CAIDO_TOKEN_PATH(never written — sha verified unchanged), replay targets were throwaway localhost servers so no external traffic was generated, and the single finding created to exercise the write path was deleted afterwards.