Skip to content

fix(verl): prevent stale/failed rollouts from breaking batch completion - #649

Open
Anvarbek Ziyodov (ziyodovanvar) wants to merge 2 commits into
microsoft:stable/v0.2.xfrom
ziyodovanvar:fix/355-stale-completed-rollouts
Open

Anvarbek Ziyodov (ziyodovanvar) wants to merge 2 commits into
microsoft:stable/v0.2.xfrom
ziyodovanvar:fix/355-stale-completed-rollouts

Conversation

@ziyodovanvar

Copy link
Copy Markdown

Summary

  • Fixes Completed rollouts > total_tasks_queued #355 on stable/v0.2.x (AgentModeDaemon was removed from main).
  • Drain leftover server completions when setting up a new batch, and reject unregistered rollout IDs (maintainer workaround from the issue).
  • Allow a successful store retry to overwrite an empty failed placeholder for the same rollout_id, with a short grace poll window so in-flight retries can land.
  • Replace the hard completed == queued asserts with reconciliation that drops stale IDs and warns instead of crashing training.
  • Add unit tests for failure detection, stale-ID dropping, and failed→success overwrite.

This is a fresh take on the closed #480 approach: failed rollouts can still satisfy the wait loop (so permanent backend failures do not hang forever), but they no longer permanently block a later successful retry, and leftover results from previous steps cannot inflate the completed count.

Test plan

  • pytest -q tests/verl/test_daemon_completed_rollouts.py (4 passed)
  • Optional: run a VERL train step that previously hit Completed N/M with N > M after vLLM 400s and confirm the assert no longer fires

Related issues

Made with Cursor

Copilot AI balanced review requested due to automatic review settings October 17, 2026 13:54

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The grace-poll branch exits immediately after initialization, preventing successful retries from being collected once all IDs have results.

1 open issue
What changed in this PR

Addresses #355 on stable/v0.2.x by hardening VERL batch completion against stale results and failed rollouts.

Changes:

  • Drains stale completions, rejects unknown IDs, and reconciles batch counts.
  • Allows successful retries to replace failed placeholders and introduces bounded grace polling.
  • Adds bookkeeping regression tests and lightweight VERL import stubs.
File Description
tests/​verl/​test_daemon_completed_rollouts.py Tests failure detection, stale-ID removal, and retry replacement.
tests/​verl/​conftest.py Provides a VERL stub for unit tests.
tests/​verl/​__init__.py Marks the test directory as a package.
agentlightning/​verl/​daemon.py Updates completion filtering, reconciliation, and retry handling.

Review effort: Balanced

Comment thread agentlightning/verl/daemon.py Outdated
if failed_grace_polls_remaining is None:
# A few extra polls (~15s) for in-flight retries to land.
failed_grace_polls_remaining = 3
elif failed_grace_polls_remaining > 0:

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fixed.

The grace counter init used elif, so the first all-present-but-failed poll set the counter and then fell through to break without sleeping. It is now an independent if, and test_async_run_until_finished_grace_polls_when_only_failure_present covers fail-on-first / succeed-on-next.

@ziyodovanvar

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

Reject leftover server results, allow successful retries to overwrite
empty failed placeholders, and soften the completed==queued assert so
training does not crash when counts drift (microsoft#355).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants