feat(v1): multi-agent topologies — imperative go(), agent graphs, deferred cross-agent rewards

[eligotts]

What

Multi-agent environments for v1, in three layers on top of main's task-first core (this branch merges the landed #1948 and builds strictly above it — the diff is near-pure addition):

1. Agent — the execution primitive: harness × model context × runtime policy, one arrow run(task) -> Trace, plus provision() (share a runtime box between agents) and interact() (hold a live episode open and converse with it).
2. Topology — the config-driven packaging: named agents as typed config fields, orchestration as plain imperative code in go(), cross-agent rewards declared and deferred, each instance persisted as an AgentGraph.
3. Seven instantiations — two built-ins (llm-judge, agentic-judge) and five example envs, each emblematic of one contract, all verified live.

Core in agent.py / topology.py; guide at docs/v1/topologies.md.

Built on the task-first contract

A topology's whole premise is main's #1948: a task minted anywhere is first-class (its TaskData row + its behavior class). go() mints tasks mid-run three equal ways — a taskset's load(), inline CritiqueTask(vf.TaskData(...)), or the Task.from_trace(trace) convention #1948 documents: a task derived from a finished rollout, reading everything off the self-describing trace (trace.task.data). The judge topologies and proposer-solver implement from_trace verbatim.

go(): orchestration as plain imperative code

An agent is pure routing (harness + model/client/sampling + trainable); tasks arrive per episode. The forward arrow is ordinary host-side construction; loops are rounds, asyncio.gather is fan-out, awaiting several traces is fan-in:

async def go(self, task, run):
    proposer = await run.agent("proposer").run(task)
    derived = SolverTask.from_trace(proposer)          # trace -> task, off info["submission"]
    await asyncio.gather(*(
        run.agent("solver").run(derived, parents=[proposer])
        for _ in range(self.config.num_solvers)))

Episode failures never raise — they return as data on the trace, and go decides what a failed child means.

Sessions: agents interacting within each other's episodes

run() composes completed episodes; back-and-forth interaction (chess, debate — each agent effectively the other's user) needs episodes held open. run.agent(name).interact(task) yields a Session with exactly three members: turn(message) -> reply, end(), .trace. The episode suspends between turns; N sessions + imperative routing compose into alternation, simultaneous moves (gather over turns), moderated rooms, and hidden-information views — no scheduler, no message bus.

Mechanically a session is a user simulator without the server: the interception layer already suspends an episode between turns awaiting the user seat's Respond; a session supplies that callable over an in-process queue handshake (Rollout(user=)). The turn loop, budgets, @stops, and trace building are reused unchanged. The safety contract is loud, not patient: turn() on a dead episode raises SessionEnded (never hangs), double-driving and cancelled-mid-flight turns are refused, prompted tasks / own-Task.user tasks / non-user-capable harnesses are refused at the call, and there is no retry= — half-played interactions can't be re-run. Each seat's trace is ONE multi-turn trajectory with its counterparts as user turns — the self-play training-sample shape, not one episode per move.

Deferred rewards: task-tied or topology-declared

Task rewards run at episode end, inside the rollout, runtime still live (main's contract — including config.judges). Topology rewards are declared @vf.reward(agent=...)/@vf.metric(agent=...) methods, run at instance end once per matching trace — which is what makes "proposer reward = distance of solver pass-rate from 50%" expressible. Scopes are validated at load; metrics run before rewards; both land in the same trace.rewards under distinct keys (verified live: {'correct': 1.0, 'judge': 1.0}).

Seeds, the agent graph, and full config addressability

• Seeds: --topology.taskset.id (same slot and grammar as route 1) XOR a load_tasks override — a self-seeding topology passed the flag is refused rather than silently ignoring it. The seed taskset's shared tool servers (Taskset.tools) are served once by TopologyRunner.serving, mirroring Environment.serving.
• Artifact: every trace carries agent / parents / trainable; a topology run persists one AgentGraph per traces.jsonl line (traces nested, completion order). The links live on the traces, so the DAG is recoverable from a flat dump.
• Config: agents are typed AgentConfig fields (field name = agent name) — everything is CLI/TOML-addressable with typed help (--topology.black.model, --topology.judge.harness.runtime.type docker). Pins live on AgentConfig subclasses (vf.DirectAgentConfig / vf.NullAgentConfig shared); partial overrides deep-merge into a pin, an explicit harness.id swaps it, and a topology can lock a slot (llm-judge's judge) with a refusal that points at the alternative.

The instantiations (all verified live, deepseek/deepseek-v4-flash)

id	shape	emblematic of	live result
llm-judge (built-in)	solver → judge	from_trace + a locked agent slot; judge fixed to the in-process direct harness (episode ≈ one API call)	10/10 agreement w/ gsm8k's verifier; 8/8 w/ wiki-search's own judge; verdicts compose with task rewards
agentic-judge (built-in)	solver → judge-as-real-agent	task setup uploads the solver's entire serialized trace into the judge's runtime; harness + prompt configurable	judge investigated over 3 bash turns, SCORE: 10; also ran on harbor in a prime sandbox
proposer-solver-v1	proposer → n solvers	dynamic task generation: MCP submit_question tool → typed state → SolverTask.from_trace; asymmetric capability (proposer on null with tools, solvers on tool-less direct); difficulty peaked at 50% solve rate	4/4 solvers correct → difficulty 0.0 (too-easy punished); broken-code submissions eat well_formed 0
writer-editors-v1	writer → n editors → writer, × rounds	fan-out + fan-in (parents=[draft, *edits]); one vf.Judge call per instance puts the same reward on every trace	6 briefs, exact shape, identical improvement across each instance
chess-v1	two live sessions	back-and-forth: host-side chess.Board as referee, illegal-move feedback as ordinary turns, forfeit via session.end(), per-seat model routing	full 8-ply games (incl. Ruy Lopez theory), zero illegal moves, adjudication correct
debate-v1	n concurrent sessions	N-ary sessions of ONE agent config: rounds gathered N-wide, per-seat views, peer voting — the interaction scores itself	all ballots valid, vote-share rewards per seat
shared-runtime-v1	writer + reader, one box	provision() + borrowed runtime=: two agents' rollouts share one world; the run never starts/stops a borrowed box	both traces stamped borrowed=True on the same box, handoff verified in-runtime

uv run eval --topology.id llm-judge --topology.taskset.id gsm8k-v1 -n 4
uv run eval --topology.id chess-v1 -n 1 --max-turns 64 --timeout.rollout 900 --sampling.max-tokens 4096 \
  --topology.black.model openai/gpt-5.4-mini   # mixed-model self-play, one flag
uv run eval --topology.id proposer-solver-v1 -n 1 -r 1
uv run eval --topology.id debate-v1 -n 1 --max-turns 8

Framework deltas outside the new layer (small, deliberate)

• Rollout gains runtime= (borrowed boxes; never started/stopped by the run) and user= (the session seat, pre-empting Task.user).
• RolloutSession.refused() honors a pre-set stop condition ("a stopped trace takes no more turns" — what Session.end() rides on).
• Trace gains the agent-graph links: agent, parents, trainable.
• @reward/@metric gain the topology agent= scope.
• env.py: the inline stage-timeout block is extracted as resolve_stage_timeouts, and validate_pairing factors through a task-class-level validate_task_pairing — both shared by Environment and topology agents instead of duplicated.
• Chat dialect extend coerces reasoning-only null-content assistant turns to "" (upstreams 422 them on re-send — found by a live chess game; regression-tested).
• The null program's MCP connect is bounded (60s) — a wedged handshake was a silent 0-turn harness_timeout.
• The judge plugin tier is untouched: config.judges (in-episode verdicts), vf.Judge (verdicts in your reward code), and the judge topologies (verdicts as their own agent episodes) are three distinct tiers by where the verdict lives, and they compose.

Testing

95 unit tests, no keys needed: config narrowing + pin semantics + locked slots + seed XOR + scope validation; instance links, deferred rewards, go-failure capture, graph record round-trip; session handshake/poison/double-drive/cancel-desync semantics and every interact() refusal; scripted-session tests driving chess (fool's mate through the real board) and debate (vote tallying) without a model; borrowed-runtime lifetime. Key-gated e2e runs all seven topologies live. Everything above was additionally smoke-run live on the merged contract.

Out of scope (deliberately)

Trainer wire (env-server run_topology, grouping semantics), --server / --resume / --rich under topologies, agent-as-tool (consultation inside a single turn — composes on sessions later).

🤖 Generated with Claude Code

[macroscopeapp]

🟡 Medium direct/harness.py:67

When the interception server injects hidden user-simulator turns, DirectHarness.launch drops that conversation history before the tool-call bounce retry. The loop only appends completion.choices[0].message to its local messages list, so on the next iteration the model answers against an incomplete prompt — missing the user-sim turns the server already added — and produces incorrect recovery responses. Consider reconstructing messages from the full trace (or the server's response) rather than only the final assistant message, so the bounce retry preserves the prior conversation context.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/harnesses/direct/harness.py around line 67:

When the interception server injects hidden user-simulator turns, `DirectHarness.launch` drops that conversation history before the tool-call bounce retry. The loop only appends `completion.choices[0].message` to its local `messages` list, so on the next iteration the model answers against an incomplete prompt — missing the user-sim turns the server already added — and produces incorrect recovery responses. Consider reconstructing `messages` from the full trace (or the server's response) rather than only the final assistant message, so the bounce retry preserves the prior conversation context.

[macroscopeapp]

🟡 Medium v1/topology.py:566

When a @reward or @metric method in Topology.score raises, run_instance catches the TopologyError and records graph.error, but traces that were already mutated by earlier scoring methods keep their partial rewards and metrics. The persisted AgentGraph for a failed instance ends up with a silently incomplete, inconsistent set of topology scores that downstream code can mistake for complete judgement.

This happens because score mutates traces in place as it iterates, and run_instance has no rollback of those mutations when a later scoring method crashes. Consider either scoring into a side structure and committing only after all methods succeed, or clearing partial topology rewards/metrics on the traces when the TopologyError is caught.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/topology.py around line 566:

When a `@reward` or `@metric` method in `Topology.score` raises, `run_instance` catches the `TopologyError` and records `graph.error`, but traces that were already mutated by earlier scoring methods keep their partial rewards and metrics. The persisted `AgentGraph` for a failed instance ends up with a silently incomplete, inconsistent set of topology scores that downstream code can mistake for complete judgement.

This happens because `score` mutates traces in place as it iterates, and `run_instance` has no rollback of those mutations when a later scoring method crashes. Consider either scoring into a side structure and committing only after all methods succeed, or clearing partial topology rewards/metrics on the traces when the `TopologyError` is caught.

[macroscopeapp]

🟡 Medium v1/topology.py:292

A subclass that overrides an inherited @vf.reward/@vf.metric method with an undecorated replacement fails at load time with a spurious scope error for the base-class method — even though score() would never run it. The validation in agents iterates every class in type(self).__mro__ and every entry in vars(klass), so it processes the base class's decorated method without checking whether a subclass overrides that name. discover_decorated, used by score(), follows normal MRO override semantics and would skip the shadowed base method. Consider skipping names overridden by a subclass during validation.

+        seen = set()
         for klass in type(self).__mro__:
             for fn in vars(klass).values():
+                name = getattr(fn, "__name__", None) or getattr(getattr(fn, "__func__", None), "__name__", None)
+                if name in seen:
+                    continue
+                if name is not None:
+                    seen.add(name)
                 fn = getattr(fn, "__func__", fn)

Also found in 1 other location(s)

verifiers/v1/env.py:201

validate_pairing() treats any override of Task.load_tools or Task.load_user as meaning every instance of that class requires MCP or a user simulator. Because these methods are instance methods and may legitimately return []/None based on task data, Environment.__init__ can now reject an otherwise valid eval before any tasks are loaded. A task class with conditional tools/user support will raise here even when the actual tasks in this run do not use those features.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/topology.py around lines 292-294:

A subclass that overrides an inherited `@vf.reward`/`@vf.metric` method with an undecorated replacement fails at load time with a spurious scope error for the base-class method — even though `score()` would never run it. The validation in `agents` iterates every class in `type(self).__mro__` and every entry in `vars(klass)`, so it processes the base class's decorated method without checking whether a subclass overrides that name. `discover_decorated`, used by `score()`, follows normal MRO override semantics and would skip the shadowed base method. Consider skipping names overridden by a subclass during validation.

Also found in 1 other location(s):
- verifiers/v1/env.py:201 -- `validate_pairing()` treats any override of `Task.load_tools` or `Task.load_user` as meaning *every* instance of that class requires MCP or a user simulator. Because these methods are instance methods and may legitimately return `[]`/`None` based on task data, `Environment.__init__` can now reject an otherwise valid eval before any tasks are loaded. A task class with conditional tools/user support will raise here even when the actual tasks in this run do not use those features.

[macroscopeapp]

🟡 Medium v1/agent.py:208

When Agent.run is called with both a borrowed runtime and a retry config, each retry attempt reuses the same runtime instance without resetting its state. A failed attempt can leave files, processes, or other side effects behind, so subsequent retries execute against contaminated state rather than starting fresh. This means run_with_retry can produce a passing trace that reflects leftover state from a previous failed attempt, or fail again for the same reason, instead of giving each retry a clean environment.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/agent.py around line 208:

When `Agent.run` is called with both a borrowed `runtime` and a `retry` config, each retry attempt reuses the same runtime instance without resetting its state. A failed attempt can leave files, processes, or other side effects behind, so subsequent retries execute against contaminated state rather than starting fresh. This means `run_with_retry` can produce a passing trace that reflects leftover state from a previous failed attempt, or fail again for the same reason, instead of giving each retry a clean environment.

[macroscopeapp]

🟡 Medium v1/topology.py:329

A @staticmethod decorated @vf.reward(agent=...) or @vf.metric(agent=...) method passes the load-time validation in Topology.agents but is silently skipped during Topology.score: the traces it should have scored persist without the declared metric/reward, producing incorrect training/eval results with no error. score discovers methods via discover_decorated, which filters by inspect.ismethod, and a @staticmethod on an instance is a plain function, not a bound method — so it is filtered out. The validation in agents unwraps __func__ and catches static methods, but score does not apply the same unwrapping, so the two paths disagree. Consider making discover_decorated (or its call in score) unwrap __func__ the same way the validation does.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/topology.py around line 329:

A `@staticmethod` decorated `@vf.reward(agent=...)` or `@vf.metric(agent=...)` method passes the load-time validation in `Topology.agents` but is silently skipped during `Topology.score`: the traces it should have scored persist without the declared metric/reward, producing incorrect training/eval results with no error. `score` discovers methods via `discover_decorated`, which filters by `inspect.ismethod`, and a `@staticmethod` on an instance is a plain function, not a bound method — so it is filtered out. The validation in `agents` unwraps `__func__` and catches static methods, but `score` does not apply the same unwrapping, so the two paths disagree. Consider making `discover_decorated` (or its call in `score`) unwrap `__func__` the same way the validation does.

[macroscopeapp]

🟡 Medium v1/agent.py:192

Calling Agent.run(task) without entering the agent via async with or passing services= leaves services as None, so any task that declares a shared tool server raises "task declares a shared tool server but no shared-server registry is serving" inside Rollout._shared_urls(). This breaks the documented standalone run(task) path: services is only populated from self._services, which stays None unless the caller separately entered the agent or supplied services. Consider provisioning a RunServices scope inside run() when none is available, or documenting that the caller must enter the agent context before calling run() for shared-server tasks.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/agent.py around line 192:

Calling `Agent.run(task)` without entering the agent via `async with` or passing `services=` leaves `services` as `None`, so any task that declares a shared tool server raises `"task declares a shared tool server but no shared-server registry is serving"` inside `Rollout._shared_urls()`. This breaks the documented standalone `run(task)` path: `services` is only populated from `self._services`, which stays `None` unless the caller separately entered the agent or supplied services. Consider provisioning a `RunServices` scope inside `run()` when none is available, or documenting that the caller must enter the agent context before calling `run()` for shared-server tasks.

[macroscopeapp]

🟡 Medium v1/agent.py:211

When Agent.run() is called with a borrowed runtime=, runtime_for() returns runtime.config directly (line 147) and skips resolve_runtime_config(...), so task-specific overrides for image, workdir, and resources are silently dropped. A task that requires a specific container image can run inside an unrelated borrowed runtime with no error. Consider still resolving task-specific settings against the borrowed runtime's config, or documenting that borrowed runtimes intentionally ignore task requirements.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/agent.py around line 211:

When `Agent.run()` is called with a borrowed `runtime=`, `runtime_for()` returns `runtime.config` directly (line 147) and skips `resolve_runtime_config(...)`, so task-specific overrides for `image`, `workdir`, and `resources` are silently dropped. A task that requires a specific container image can run inside an unrelated borrowed runtime with no error. Consider still resolving task-specific settings against the borrowed runtime's config, or documenting that borrowed runtimes intentionally ignore task requirements.

[macroscopeapp]

🟡 Medium topologies/llm_judge.py:71

parse_score returns nan for a SCORE: nan line — float('nan') succeeds, and min/max leave NaN untouched, so the clamp at the end doesn't catch it. The caller (JudgeTask.committed) treats this as a committed verdict and judge() records a NaN reward, poisoning trace.rewards/trace.reward and downstream aggregations. Consider rejecting non-finite values (e.g. math.isnan) and returning None.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/topologies/llm_judge.py around line 71:

`parse_score` returns `nan` for a `SCORE: nan` line — `float('nan')` succeeds, and `min`/`max` leave NaN untouched, so the clamp at the end doesn't catch it. The caller (`JudgeTask.committed`) treats this as a committed verdict and `judge()` records a NaN reward, poisoning `trace.rewards`/`trace.reward` and downstream aggregations. Consider rejecting non-finite values (e.g. `math.isnan`) and returning `None`.

[macroscopeapp]

🟠 High direct/harness.py:51

AsyncOpenAI is created with max_retries left at the SDK default, so a transient transport failure on POST /chat/completions triggers an automatic retry that re-samples the model and appends a duplicate turn to messages. This forks the recorded rollout into an extra trajectory instead of preserving a single one. Set max_retries=0 when constructing the client so the harness controls retry behavior rather than the SDK.

Suggested change

	client = AsyncOpenAI(base_url=endpoint, api_key=secret)
	client = AsyncOpenAI(base_url=endpoint, api_key=secret, max_retries=0)

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/harnesses/direct/harness.py around line 51:

`AsyncOpenAI` is created with `max_retries` left at the SDK default, so a transient transport failure on `POST /chat/completions` triggers an automatic retry that re-samples the model and appends a duplicate turn to `messages`. This forks the recorded rollout into an extra trajectory instead of preserving a single one. Set `max_retries=0` when constructing the client so the harness controls retry behavior rather than the SDK.

[macroscopeapp]

🟡 Medium topologies/llm_judge.py:79

JudgeTask.for_attempt renders the judge's <task> section from task.data.prompt_text, but TaskData.prompt can be None for user-simulator tasks where the task framing is only established conversationally. For those tasks the judge prompt contains an empty <task> block, so the judge scores an attempt without ever seeing what problem was being solved — producing incorrect rewards for any user-driven taskset. Consider falling back to the full conversation text (or the trace's messages) when prompt_text is empty, so the judge always sees the task framing.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/topologies/llm_judge.py around line 79:

`JudgeTask.for_attempt` renders the judge's `<task>` section from `task.data.prompt_text`, but `TaskData.prompt` can be `None` for user-simulator tasks where the task framing is only established conversationally. For those tasks the judge prompt contains an empty `<task>` block, so the judge scores an attempt without ever seeing what problem was being solved — producing incorrect rewards for any user-driven taskset. Consider falling back to the full conversation text (or the trace's messages) when `prompt_text` is empty, so the judge always sees the task framing.

[macroscopeapp]

🟡 Medium

verifiers/verifiers/v1/cli/eval/main.py

Line 107 in e2852fa

if config.push and not rich:

When a topology eval runs with --push, push_traces(traces, config) derives the environment/dataset name from config.taskset.id or config.id, but EvalConfig.check_topology() rejects both of those fields when config.topology is set. As a result, pushed topology results upload with an empty environment/dataset name instead of config.topology.id, producing incorrect metadata or failing to resolve the environment. Consider deriving the topology upload name from config.topology.id (e.g., by short-circuiting push_traces before the taskset/id lookup), or document the intended push semantics for topology runs if this is by design.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/cli/eval/main.py around line 107:

When a topology eval runs with `--push`, `push_traces(traces, config)` derives the environment/dataset name from `config.taskset.id or config.id`, but `EvalConfig.check_topology()` rejects both of those fields when `config.topology` is set. As a result, pushed topology results upload with an empty environment/dataset name instead of `config.topology.id`, producing incorrect metadata or failing to resolve the environment. Consider deriving the topology upload name from `config.topology.id` (e.g., by short-circuiting `push_traces` before the taskset/id lookup), or document the intended push semantics for topology runs if this is by design.

[macroscopeapp]

🟡 Medium debate_v1/topology.py:117

A single debater's seat.turn(...) raising SessionEnded (timeout, stop condition, or episode death) aborts the entire debate — asyncio.gather propagates the exception, unwinding the AsyncExitStack and converting the whole topology run into a TopologyError instead of recording that one seat's failure as data. This happens at all three gather sites (lines 117, 130, 149) because gather is called without return_exceptions=True and there is no try/except around the turn calls. Consider passing return_exceptions=True and handling SessionEnded per-seat so a failed debater is recorded in trace.info rather than crashing the debate.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/debate_v1/debate_v1/topology.py around line 117:

A single debater's `seat.turn(...)` raising `SessionEnded` (timeout, stop condition, or episode death) aborts the entire debate — `asyncio.gather` propagates the exception, unwinding the `AsyncExitStack` and converting the whole topology run into a `TopologyError` instead of recording that one seat's failure as data. This happens at all three gather sites (lines 117, 130, 149) because `gather` is called without `return_exceptions=True` and there is no `try`/`except` around the turn calls. Consider passing `return_exceptions=True` and handling `SessionEnded` per-seat so a failed debater is recorded in `trace.info` rather than crashing the debate.

[macroscopeapp]

🟡 Medium topologies/agentic_judge.py:69

for_trace calls prompt.format(path=TRACE_PATH) on the user-configurable --topology.prompt string, so any custom rubric that contains other braces — e.g. a JSON snippet like { "score": 10 } or a placeholder like {answer} — raises KeyError/ValueError and crashes the agentic-judge run before the judge rollout starts. Only the {path} placeholder should be substituted. Consider replacing the literal with a targeted replacement like prompt.replace("{path}", TRACE_PATH).

Suggested change

	prompt=prompt.format(path=TRACE_PATH),
	prompt=prompt.replace("{path}", TRACE_PATH),

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/topologies/agentic_judge.py around line 69:

`for_trace` calls `prompt.format(path=TRACE_PATH)` on the user-configurable `--topology.prompt` string, so any custom rubric that contains other braces — e.g. a JSON snippet like `{ "score": 10 }` or a placeholder like `{answer}` — raises `KeyError`/`ValueError` and crashes the `agentic-judge` run before the judge rollout starts. Only the `{path}` placeholder should be substituted. Consider replacing the literal with a targeted replacement like `prompt.replace("{path}", TRACE_PATH)`.

[macroscopeapp]

🟠 High v1/topology.py:685

_agents_for passes the taskset-wide _shared_tools to every agent, including agents that consume derived tasks. Agent._validate_pairing treats nonempty shared_tools as tools exposed by every task, so a tool-less harness like direct (SUPPORTS_MCP = False) is rejected even when its derived task has no tools. An llm-judge topology over a tool-bearing taskset runs the solver but fails when the direct judge starts. The shared tools should be scoped to the agent(s) that actually consume the seed task, not passed to every agent unconditionally.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/topology.py around line 685:

`_agents_for` passes the taskset-wide `_shared_tools` to every agent, including agents that consume derived tasks. `Agent._validate_pairing` treats nonempty `shared_tools` as tools exposed by every task, so a tool-less harness like `direct` (`SUPPORTS_MCP = False`) is rejected even when its derived task has no tools. An `llm-judge` topology over a tool-bearing taskset runs the solver but fails when the `direct` judge starts. The shared tools should be scoped to the agent(s) that actually consume the seed task, not passed to every agent unconditionally.

[macroscopeapp]

🟠 High eval/resume.py:79

AgentGraph.load(row) silently accepts legacy flat trace rows (which have id, task, and error fields but no traces key), because the overlapping fields satisfy the model and traces defaults to []. When resuming an interrupted eval directory written by the previous implementation, these flat rows are treated as complete invocations with empty trace lists. As a result, prior successful traces are dropped from the returned graphs, dashboard state, and the final push_traces upload; when all rows are flat the command falsely reports every invocation complete while returning empty graphs. Consider detecting flat rows (e.g., rows lacking a traces key) and wrapping them into a single-trace graph, or rejecting/migrating the old format before counting a row as complete.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @verifiers/v1/cli/eval/resume.py around line 79:

`AgentGraph.load(row)` silently accepts legacy flat trace rows (which have `id`, `task`, and `error` fields but no `traces` key), because the overlapping fields satisfy the model and `traces` defaults to `[]`. When resuming an interrupted eval directory written by the previous implementation, these flat rows are treated as complete invocations with empty trace lists. As a result, prior successful traces are dropped from the returned `graphs`, dashboard state, and the final `push_traces` upload; when all rows are flat the command falsely reports every invocation complete while returning empty graphs. Consider detecting flat rows (e.g., rows lacking a `traces` key) and wrapping them into a single-trace graph, or rejecting/migrating the old format before counting a row as complete.

[macroscopeapp]

🟡 Medium proposer_solver_v1/topology.py:45

extract_boxed returns None when the last \boxed{ in the reply is unbalanced, even if an earlier \boxed{...} was valid. For example, a reply containing \boxed{123} followed later by an incomplete \boxed{ incorrectly returns None, so the solver loses its parseable reward despite a valid boxed answer. Consider scanning for the last balanced \boxed{...} occurrence rather than starting only at the rightmost literal \boxed{.

🚀 Reply "fix it for me" or copy this AI Prompt for your agent:

In file @environments/proposer_solver_v1/proposer_solver_v1/topology.py around line 45:

`extract_boxed` returns `None` when the last `\boxed{` in the reply is unbalanced, even if an earlier `\boxed{...}` was valid. For example, a reply containing `\boxed{123}` followed later by an incomplete `\boxed{` incorrectly returns `None`, so the solver loses its `parseable` reward despite a valid boxed answer. Consider scanning for the last *balanced* `\boxed{...}` occurrence rather than starting only at the rightmost literal `\boxed{`.