Interactive rollout#1928
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
ApprovabilityVerdict: Needs human review 2 blocking correctness issues found. This PR introduces a substantial new feature (interactive rollout TUI with ~2000+ lines of new code) that adds new user-facing behavior and components. Additionally, there are 3 unresolved medium-severity review comments identifying potential bugs that warrant human review. You can customize Macroscope's approvability policy. Learn more. |
- schema_type: resolve anyOf/oneOf so Optional[...] tool args parse to their real JSON type instead of a raw string - gate file://, and bare-path images behind an opt-in flag (renamed --allow-remote-images -> --allow-external-images); only inline data: images render by default - infer_current_env_id: keep walking ancestors past nameless pyprojects - run_interactive_rollout: reject non-object --env-args with a clean CLI error instead of a TypeError traceback Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cation - HumanClient.start: race ready.wait() against the app task so a TUI startup failure raises instead of hanging forever - schema_type: only collapse anyOf/oneOf to a single non-null branch (Optional[X]); leave genuine unions (int | str) as free-form - truncate the URL in the image-load error text Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Treat only a literally-empty argument field as "not provided", so an intentionally blank or whitespace-only required string (e.g. empty file content) is sent verbatim instead of being rejected as missing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve conflicts against the nano-as-v1 refactor: - pyproject.toml: keep textual/textual-image deps + vf-play entry point; drop dead tomli and duplicate typing_extensions (min Python is now 3.11) - interception_utils.py: union imports (Error/State for the quit-error path + OPENAI_RESPONSES_OUTPUT_FIELD from main) - uv.lock: regenerated - play.py: drop the removed --taskset.*/--harness.* override mechanism; env_id is now a plain positional, env args come from --env-args only - app.py: Select.BLANK -> Select.NULL for textual 8.x Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 297c11c. Configure here.
| suffix=suffix, | ||
| ) | ||
| size = Image.open(io.BytesIO(response.content)).size | ||
| return ImagePayload(path=path, size=size, source=url) |
There was a problem hiding this comment.
External images block asyncio loop
Medium Severity
With --allow-external-images, loading http(s) image URLs uses synchronous requests.get inside message rendering on the Textual/asyncio event loop, which can freeze the TUI until the download finishes or times out.
Reviewed by Cursor Bugbot for commit 297c11c. Configure here.
| parts.append( | ||
| Text( | ||
| self._truncate_text( | ||
| json.dumps(part_data, indent=2, ensure_ascii=False) |
There was a problem hiding this comment.
🟡 Medium interactive/app.py:750
_content_renderables calls json.dumps(part_data, ...) on unknown content-part mappings, which raises TypeError when a part contains non-JSON-native values like bytes, Path, dataclasses, or Pydantic models. This crashes rendering of the entire messages pane for the turn instead of showing a fallback placeholder. Consider using the repo's existing _json_gate_default() as the default= argument (or otherwise pre-converting non-serializable values) so unknown parts degrade gracefully.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/cli/interactive/app.py around line 750:
`_content_renderables` calls `json.dumps(part_data, ...)` on unknown content-part mappings, which raises `TypeError` when a part contains non-JSON-native values like `bytes`, `Path`, dataclasses, or Pydantic models. This crashes rendering of the entire messages pane for the turn instead of showing a fallback placeholder. Consider using the repo's existing `_json_gate_default()` as the `default=` argument (or otherwise pre-converting non-serializable values) so unknown parts degrade gracefully.
| items.append(Group(description, table)) | ||
| return Group(*items) | ||
|
|
||
| def _argument_rows(self, tool: Tool) -> list[tuple[str, Mapping[str, Any], bool]]: |
There was a problem hiding this comment.
🟡 Medium interactive/app.py:698
_argument_rows only reads tool.parameters["properties"], so tools with valid non-object schemas (e.g. array/string root) or free-form additionalProperties objects produce zero rows. The form shows No declared arguments. and _build_current_tool_call can only submit {}, making it impossible to pass any arguments to those tools — even though MCP/OpenEnv pass such inputSchema through correctly. Consider supporting schemas whose root type is not object with named properties, or at minimum documenting that the interactive UI only supports object-shaped tool schemas.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @verifiers/cli/interactive/app.py around line 698:
`_argument_rows` only reads `tool.parameters["properties"]`, so tools with valid non-`object` schemas (e.g. `array`/`string` root) or free-form `additionalProperties` objects produce zero rows. The form shows `No declared arguments.` and `_build_current_tool_call` can only submit `{}`, making it impossible to pass any arguments to those tools — even though MCP/OpenEnv pass such `inputSchema` through correctly. Consider supporting schemas whose root type is not `object` with named properties, or at minimum documenting that the interactive UI only supports `object`-shaped tool schemas.
1 participant
Description
New interactive rollout terminal app via
uv run vf-play <environment>wordl env
tic-tac-toe env
loading state view
Type of Change
Testing
uv run pytestlocally.Checklist
Additional Notes
Note
Medium Risk
New optional CLI/TUI path with image-loading opt-in and rollout error plumbing changes; large test surface but limited impact on default eval/training flows.
Overview
Adds
vf-playso you can step through a single environment rollout as the model instead of calling an API. The command loads one dataset row, runs the normal rollout loop with aHumanClient, and drives a persistent Textual UI for each generation step.The TUI shows model-visible messages and tool schemas, and lets you submit one assistant turn with optional reasoning, message content, and parallel tool calls built from JSON-schema argument fields (including multi-line strings). Quitting raises
InteractiveSessionExit, which is recorded on rollout state like other framework errors. Optional--save,--no-score, and workspaceenv_idinference match eval-style workflows.PrimeCLIPlugingainsplay_modulewith the same--env-dir-pathinjection as eval/gepa. Dependencies addtextualand conditionaltextual-imagefor inline terminal images; external image URLs stay off unless--allow-external-images.InterceptionServernow wraps non-Errorexceptions and usesState._set_errorwhen storing rollout failures so interactive quit and other typed errors propagate consistently.Reviewed by Cursor Bugbot for commit 297c11c. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add
vf-playinteractive terminal UI for human-in-the-loop rollout playthroughvf-playCLI command backed by a Textual TUI (app.py) that lets a human step through a model rollout turn-by-turn, composing assistant messages, reasoning, and tool calls with schema-aware inputs.HumanClient(client.py) implements the standardClientinterface, routing model requests to the TUI and returning structuredResponseobjects with the human's input.env._run_rollout_state, renders a summary table, and optionally saves the result to JSON.--allow-external-images.InterceptionServerso exceptions during intercepted requests are stored onStateas properErrorsubclasses rather than being stringified.Macroscope summarized 297c11c.