Repository navigation
key scheduled work by its start time and sweep commit order for stragglers - #224
Merged
Merged
Conversation
…glers An entry scheduled for later is now keyed at its start time (rounded up to the next whole millisecond), so it sorts above the loop's read bound until due — no re-keying, and a bulk enqueue of scheduled work never sits in front of ready work. Since such a key is chosen before the enqueue commits, the entry can land behind the loop's cursor when the commit takes longer than the delay; every wall-clock-keyed entry therefore records its commit stamp in scheduledAt, and the loop sweeps that index in commit order — which nothing can land behind — inspecting each entry once and directly starting any entry the cursor already passed. The incoming cursor's bounds are now explicit values instead of a stateful guard: the work query computes a ceiling (the highest commit timestamp its snapshot observed, seeded by the previous run's own stamp recorded in internalState) and the worker mutation raises every segment it writes to at least the cursor and advances to at most the lowest segment it wrote and the ceiling. The ceiling is what keeps a cursor that lands on a wall-clock key from passing a racing commit; nothing anywhere relates wall clocks to the commit-timestamp clock. The runAt and hasRunAt fields this branch previously introduced are no longer written; the loop clears them from entries earlier revisions wrote as it re-keys them. The five-minute safe-future threshold is gone: the sweep covers any delay uniformly.
Every site that looked up a pendingStart by workId has the work document in hand, so the work doc now carries pendingStartId and the index is gone, leaving pendingStart with two indexes (segment, scheduledAt). The pointer can be stale — the entry starts and is deleted — so readers check the entry still exists. Entries the pointer can't reach (work deleted out from under them, or written by a version without the pointer) are dropped reactively: the segment scan deletes an entry whose work is gone, and now also one whose work was canceled, finishing the cancelation the pointer couldn't. Canceling pre-pointer work therefore marks it immediately but clears its queue entry only when the entry comes due.
Only unreleased revisions ever wrote them, no deployment still holds them (schema validation on push confirms), and the current design doesn't need them: an entry's segment is its start time, scheduledAt marks it as wall-clock-keyed, and due-ness is visibility. The only remaining upgrade shape is 0.4.9's 100ms bucket, recovered from segment by magnitude.
Work docs written before the pointer existed could be queued or mid-attempt, and status can't tell which without the old index. Queued is the longer-lived of the two (it can last until a far-future start; mid-attempt lasts one attempt, after which the work finishes or retries and sets the pointer), so report pending rather than running.
Replaces the two-lane proposal with what shipped: start-time keys with a commit-ordered scheduledAt sweep, the floor/ceiling cursor rules, the pendingStartId pointer, and the upgrade behavior.
Member
Author
|
Warning This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
This stack of pull requests is managed by Graphite. Learn more about stacking. |
commit: |
Member
Author
|
Diff from main, in case that's easier to review: main...ian/always-set-segment |
reeceyang
approved these changes
Aug 17, 2026
Contributor
|
this looks good directionally! |
reeceyang
reviewed
Aug 17, 2026
| // got retired. | ||
| const scanned = await ctx.db | ||
| .query("pendingStart") | ||
| .withIndex("scheduledAt", (q) => q.gt("scheduledAt", sweepCursor)) |
Contributor
There was a problem hiding this comment.
is it ok to use gt here? work scheduled in the same transaction would have the same scheduledAt, right?
The sweep cursor can't split a commit stamp, so a batch enqueue bigger than one sweep batch could otherwise pin it forever. Paging to the end of the boundary stamp within the iteration (continuing on the index's implicit _creationTime) makes the batch size a pure read budget, with no relation to transaction write limits.
Answers from review: retries carry scheduledAt not for the sweep (they can't commit out of order) but so their wall-clock keys aren't counted as observed commit stamps in the cursor ceiling; lastCommitTs only bounds cursor advancement, and without it a pool running only scheduled work would re-read started entries' tombstones until each completion's commit stamp arrived; keys stay whole-millisecond because that's the clock's resolution and a fractional key would start work before any clock reading equals its start time. Also scopes the 100ms bucket helpers to their two remaining uses (recovery pacing and decoding what older versions wrote), moves getNextSegment into the one test that uses it, and stops passing the ignored segment range to calculateBacklogAndReport.
This was referenced Sep 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
key scheduled work by its start time and sweep commit order for stragglers
An entry scheduled for later is now keyed at its start time (rounded up to
the next whole millisecond), so it sorts above the loop's read bound until
due — no re-keying, and a bulk enqueue of scheduled work never sits in front
of ready work. Since such a key is chosen before the enqueue commits, the
entry can land behind the loop's cursor when the commit takes longer than
the delay; every wall-clock-keyed entry therefore records its commit stamp
in scheduledAt, and the loop sweeps that index in commit order — which
nothing can land behind — inspecting each entry once and directly starting
any entry the cursor already passed.
The incoming cursor's bounds are now explicit values instead of a stateful
guard: the work query computes a ceiling (the highest commit timestamp its
snapshot observed, seeded by the previous run's own stamp recorded in
internalState) and the worker mutation raises every segment it writes to at
least the cursor and advances to at most the lowest segment it wrote and the
ceiling. The ceiling is what keeps a cursor that lands on a wall-clock key
from passing a racing commit; nothing anywhere relates wall clocks to the
commit-timestamp clock.
The runAt and hasRunAt fields this branch previously introduced are no
longer written; the loop clears them from entries earlier revisions wrote as
it re-keys them. The five-minute safe-future threshold is gone: the sweep
covers any delay uniformly.
replace the pendingStart workId index with a pointer on work
Every site that looked up a pendingStart by workId has the work document in
hand, so the work doc now carries pendingStartId and the index is gone,
leaving pendingStart with two indexes (segment, scheduledAt). The pointer
can be stale — the entry starts and is deleted — so readers check the entry
still exists.
Entries the pointer can't reach (work deleted out from under them, or
written by a version without the pointer) are dropped reactively: the
segment scan deletes an entry whose work is gone, and now also one whose
work was canceled, finishing the cancelation the pointer couldn't. Canceling
pre-pointer work therefore marks it immediately but clears its queue entry
only when the entry comes due.