Skip to content

test(evals): exercise the workspace V4 CLI - #2838

Draft
shrey150 wants to merge 4 commits into
agent/browse-v4-5-remove-xpathfrom
agent/browse-v4-6-release
Draft

test(evals): exercise the workspace V4 CLI#2838
shrey150 wants to merge 4 commits into
agent/browse-v4-5-remove-xpathfrom
agent/browse-v4-6-release

Conversation

@shrey150

@shrey150 shrey150 commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Summary

Make the workspace evals and release path consume the V4 Browse CLI proven in #2835.

  • Point Stagehand evals at the workspace Browse package instead of published Browse 0.9.5.
  • Update native and external-agent adapters to the current Browse command grammar.
  • Keep eval session names within Unix socket limits, restrict the wrapper to driver commands, and clean daemons up on abort/signals.
  • Add command-mapping and real wrapper-process coverage.
  • Add the Browse minor changeset for the V4 migration and --return-xpath removal.

The unrelated eval CLI --verbose changes from the old draft are intentionally not included.

Stack

  1. chore(cli): import Browse V3 baseline #2833 — exact Browse V3 baseline import
  2. refactor(cli): migrate Browse runtime to Stagehand V4 #2834 — Stagehand V4 runtime cutover and CLI-owned cursor
  3. fix(cli): restore V3 network capture through a CDP sidecar #2849 — CLI-private CDP sidecar; restores V3 network capture
  4. refactor(cli): remove coordinate XPath return flag #2835 — remove --return-xpath; complete supported V3 parity
  5. test(evals): exercise the workspace V4 CLI #2838 — eval, packaging, and release wiring
  6. fix(cli): persist context names in Browserbase #2839 — managed Context names (fast-follow)
  7. refactor(cli): consume shared Functions core #2701 — shared Functions core consumer (fast-follow)

Review boundary

This PR is a consumer/packaging layer, not another runtime-parity layer. The CLI behavior belongs in #2834/#2835; this diff proves the released package and eval adapters invoke it correctly.

E2E Test Matrix

Command / flow Observed output Confidence / sufficiency
Original feature-head packaging proof: pack Stagehand 4.0.2 and Browse 0.9.6; install both in an isolated consumer npm ls reported Browse with Stagehand 4.0.2 deduped; both realpaths were under the isolated node_modules, not workspace links; 189 packages audited with 0 vulnerabilities Proves exact-head publish artifacts compose outside the monorepo
Run the installed node_modules/.bin/browse against real Chromium: open, full snapshot, fill, focus/press, bare type, ref click, and DOM reads Title PR 2838 restack E2E; type returned { "typed": true }; ref @0-15 produced Submitted: TypedByBareCommand Proves the packed CLI's main navigation, snapshot/ref, locator, and keyboard flow works
Exact-head installed CLI cursor and coordinate mouse Packaged Browse installed with packaged Stagehand 4.0.2; evaluated cursor DOM returned { left: "211px", top: "233px", svg: true } in real Chromium Proves the CLI-owned cursor injection and daemon lifecycle survive packing without a public Stagehand cursor API
Real BrowseCliTool adapter process smoke Input AdapterBareType; output Submitted: AdapterBareType; representation snapshot_refs; correct local/tool/launch metadata; cleanup removed daemon files Proves the eval adapter launches and drives the actual CLI process rather than a mock
Focused adapter suites on current restacked head e22fda93c 3 files / 41 tests passed Covers command mapping, wrapper routing, cleanup branches, and TUI integration
pnpm --filter @browserbasehq/stagehand-evals build Passed on current restacked head e22fda93c Proves current-head eval build integration
pnpm tsx scripts/release/check-changesets.ts Passed on current restacked head Proves release metadata is valid

The isolated real-browser packaging smoke is local Chromium. Browserbase remote mode and registry publication are outside this packaging check and were not rerun here. The fixture, installed-CLI daemon, adapter daemon, capture, and isolated consumer were cleaned up; the worktree remained clean.

@changeset-bot

changeset-bot Bot commented Aug 27, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: e22fda9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
browse Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant