Skip to content

docs: clarify failed rollout handling and reward semantics - #706

Open
Baixin Guo (GBX-Max1220) wants to merge 2 commits into
microsoft:mainfrom
GBX-Max1220:docs/failed-rollout-handling
Open

Baixin Guo (GBX-Max1220) wants to merge 2 commits into
microsoft:mainfrom
GBX-Max1220:docs/failed-rollout-handling

Conversation

@GBX-Max1220

Copy link
Copy Markdown

Fixes #448.

This is a documentation-only change to docs/05-basics.md. It adds a short "Handling failed rollouts" subsection to the existing Basics guide. No production code is modified.

Issue #448 discusses APIs that no longer exist at current HEAD (RolloutConfig.max_attempts, agl.emit_exception, agl.TracerTraceToTriplet). This contribution documents the behavior actually implemented in v1.0 instead of reviving the older workaround.

What the section clarifies

  • Rollout-level retry. There is no automatic retry of a FAILED rollout; FAILED is terminal. The Gateway only makes up to six attempts for each proxied model call on transport errors, timeouts, 408, 409, 429, and 5xx (six includes the initial request). Re-enqueuing needs a fresh rollout_id.

  • None versus FAILED. Returning None from run() is not a failure signal: the worker ignores the return value and exits with code 0, so the rollout becomes SUCCEEDED regardless of whether a reward event was recorded. Only an uncaught exception exits non-zero and marks the rollout FAILED.

  • FAILED rollouts still enter adapter processing. The trainer builds a CompletedRollout from any terminal rollout, including FAILED, and passes it to the adapter. Model calls recorded before the failure remain in the partial trace; they are not dropped automatically.

  • Missing reward default. When no reward event was recorded, reward_fillna_value (default 0.0) supplies the reward for any rollout, successful or failed. An infrastructure failure after a few successful model calls can therefore produce zero-reward training rows indistinguishable from a genuine zero-reward outcome.

  • Infrastructure failure versus legitimate task failure. A decision table separates framework behavior from user policy across five situations: transient and persistent infrastructure failures, exceptions in sub-agents, valid task failures, and excessive failures in a batch. Task outcomes — including wrong answers — should be reported with an explicit numeric reward event; infrastructure errors should not be converted into 0.0 rewards.

Scope notes

  • No built-in batch-skip or automatic filtering feature is claimed. RolloutHooks.on_failed is documented as a lifecycle callback, not a filter; filtering must happen on completed_rollouts before the adapter consumes them.
  • No custom-adapter example is included: the built-in trainer constructs RolloutAdapter directly and exposes no adapter injection point.
  • Includes a stdlib-only minimal example separating task outcomes from infrastructure errors.

Validation: mkdocs build --strict, embedded snippet py_compile, git diff --check, and git apply --check all pass. Single-file diff.

Copilot AI balanced review requested due to automatic review settings November 18, 2026 19:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The documentation omits that a non-numeric final reward event causes training to use the configured fallback rather than an earlier numeric reward.

1 open issue
What changed in this PR

This documentation-only PR addresses #448 by explaining failed-rollout handling in Agent Lightning v1.0 and distinguishing framework behavior from user-defined training policy.

Changes:

  • Documents retry boundaries, terminal states, and how failed rollouts reach training.
  • Explains reward defaults and provides an example separating task outcomes from infrastructure errors.
  • Clarifies filtering options, lifecycle hooks, and unsupported older APIs.
File Description
docs/​05-basics.md Adds failed-rollout guidance, a decision table, a minimal example, and adapter limitations.

Review effort: Balanced

Comment thread docs/05-basics.md Resolved

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to deal with failed rollouts?

2 participants