Skip to content

fix(workflows): wake race branches at their deadlines while other branches run - #27

Merged
eersnington merged 3 commits into
fix/workflows-reject-removed-try-scopesfrom
fix/workflows-race-sleep-deadline
Nov 8, 2026
Merged

eersnington merged 3 commits into
fix/workflows-reject-removed-try-scopesfrom
fix/workflows-race-sleep-deadline

Conversation

@eersnington

@eersnington eersnington commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

Refs rivet-dev/rivet#5958.

A ctx.race with a ctx.sleep branch never timed out in-process work: the race waited for every branch to settle, so a slow branch won after the deadline had passed.

  • executeRace keeps a yielded branch and runs it again at its deadline while other branches still run.
  • The race writes its earliest deadline with driver.setAlarm, so an evicted run wakes again at that deadline.
  • When a branch wins, the race stops waiting for cancelled steps. Their late results are discarded and are not recorded as failures or retried.
  • Eviction does not interrupt a running step, so the step's result is still recorded.
  • Second commit: the race saves the winner before it deletes the losing branches, through one queued save. Before, a crash between the delete and the save lost the winner, and replay ran confirmed steps again.
main This PR
2000 ms step raced against ctx.sleep(300) step wins at ~2000 ms timeout wins at ~300 ms
Crash between deleting losers and saving the winner confirmed loser step runs again winner kept, no re-run

Same fix in engine-ee: rivet-dev/engine-ee#1777.


This is part 4 of 5 in a stack:

@eersnington
eersnington merged commit 082fcf9 into fix/workflows-reject-removed-try-scopes Nov 8, 2026
2 checks passed
@eersnington
eersnington deleted the fix/workflows-race-sleep-deadline branch November 8, 2026 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant