Order pending work by commit and snapshot timestamps by ianmacartney · Pull Request #220 · convex/workpool

ianmacartney · 2026-08-04T00:43:01Z

Order ready work, completions, and cancellations by commit timestamp. A sweep finds scheduled enqueues that commit behind the incoming cursor, which stays bounded by the transaction snapshot.

Upgrade legacy keys and pointers before scanning, preserving their original time ordering. Requires Convex 1.43.

Validation: 162 tests pass; build, typecheck, lint, and changed-file formatting pass.

ianmacartney · 2026-08-04T00:43:14Z

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

Order pending work by commit and snapshot timestamps #220 : 2 dependent PRs (#234 , #240 ) 👈 (View in Graphite)
Record pendingStartId on work and upgrade batch-worker to 0.3 #252
main

This stack of pull requests is managed by Graphite. Learn more about stacking.

pkg-pr-new · 2026-08-04T00:43:34Z

Open in StackBlitz

npm i https://pkg.pr.new/convex/workpool/@convex-dev/workpool@220

commit: 0287944

ianmacartney · 2026-08-04T00:46:24Z

One thing to note here is that reverting to an older version would require a migration to convert segments into their old format, - which could be hard to get right, but could make some assumptions about the maximum segment a user would have previously used, or migrate based on creation time of the documents or something..?

reeceyang

When users migrate to this version, would work that was originally scheduled to run in the future get executed immediately?

ianmacartney · 2026-09-09T18:14:01Z

@cubic review

cubic-dev-ai · 2026-09-09T18:14:06Z

@cubic review

@ianmacartney I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 26 files

Not reviewed (too large): example/benchmarks/2026-09-08.json (~5,865 lines) - if these are generated or fixture files, add them to ignored paths to exclude them from future reviews.

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 17 files (changes from recent commits).

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

ianmacartney · 2026-09-09T23:57:12Z

I split out the pendingStart batching into a stacked PR

cubic-dev-ai

All reported issues were addressed across 20 files

_{Reply with feedback, questions, or to request a fix.

Re-trigger cubic}

reeceyang · 2026-09-15T21:47:44Z

-    segment: max(toSegment(runAt), getCurrentSegment()),
+    segment,
+    // Only near-term starts can commit behind the cursor.
+    ...(scheduled && (segment as bigint) <= toTimestamp(now + 5 * MINUTE)


is there a way to avoid the cast here?

we could do different inserts based on scheduled, since segment is only ever the placeholder if scheduled is false?

reeceyang · 2026-09-15T21:56:48Z

 const batchFields = {
-  // The segment at query time — what "now" was when the batch was built.
-  segment: v.int64(),
+  upgrade: v.optional(v.object({ starts: v.array(vStart), done: v.boolean() })),


iiuc, these are the legacy entries that we now want to convert to use nanoseconds?

reeceyang · 2026-09-15T22:04:59Z

+    // (1) Out-of-order entries left at the boundary stamp. Nothing at or above
+    // the cursor is read, so a bulk enqueue sharing this stamp costs nothing.


which cursor is meant in

Nothing at or above the cursor is read, so a bulk enqueue sharing this stamp costs nothing.

It looks like we do read at least one doc at or above the sweep cursor?

We start at the sweepCursor for the scanTs field, but we only read up to incomingCursor for the segment, since anything greater than that will get picked up by the normal incomingCursor query.
So if sweep is at 3, incomingCursor is at 10, the queries are:

1. scanTs == sweepCursor && segment < incomingCursor
2. scanTs ≥ sweepCursor
3. segment ≥ incomingCursor

(1) is an optimization for the case where a huge transaction lands in a single scanTs. Instead of scanning over the whole transaction, once we land in the middle of it, the next iteration will only consider the section of it that is actually actionable, before moving on to the next scanTs.

reeceyang · 2026-09-15T22:07:51Z

+  // Raised to at least the snapshot, which the cursor may reach this run, so
+  // the retry can't land behind it; a backoff shorter than the clocks' skew
+  // just starts next iteration.
+  const segment = maxBigint(


is this still needed now that snapshotTs <= Date.now <= commitTs?

It's not needed for contexts where that's gone out. I'll wait for convex-test to support it before removing it though. I'll leave a TODO

reeceyang · 2026-09-15T22:13:49Z

  // ────────────────────────────────────────────────────────────────────

  describe("getBatch", () => {
+    it("waits for wall time when the snapshot clock is ahead", async () => {


this scenario is not possible in practice, right?

it was possible before, but I believe the Date.now() change should be out soon/already. And it can happen in convex-test pretty often, since Date.now() is fixed for tests, but each transaction bumps the db clock

reeceyang · 2026-09-15T22:15:46Z

+      expect(o.running.map((r) => r.workId)).toEqual([scheduledId]);
+    });
+
+    it("starts scheduled work once it's due, without ever rewriting it", async () => {


we don't ever rewrite scheduled work now, right? (unless it's being upgraded from a legacy entry?)

ianmacartney requested a review from reeceyang August 4, 2026 00:43

ianmacartney changed the title ~~update convex~~ use commitTS for workpool Aug 4, 2026

reeceyang reviewed Aug 4, 2026

View reviewed changes

Comment thread src/component/loop.test.ts Outdated

Comment thread src/component/loop.test.ts Outdated

Comment thread src/component/loop.ts Outdated

ianmacartney mentioned this pull request Aug 17, 2026

key scheduled work by its start time and sweep commit order for stragglers #224

Merged

ianmacartney force-pushed the ian/use-commit-ts branch from 4e3cbd1 to 99c41ba Compare August 21, 2026 01:11

ianmacartney mentioned this pull request Sep 9, 2026

bound the incoming cursor by the snapshot timestamp #233

Merged

ianmacartney force-pushed the ian/use-commit-ts branch from d55a95d to b619fc5 Compare September 9, 2026 18:12

ianmacartney marked this pull request as ready for review September 9, 2026 18:13

cubic-dev-ai Bot reviewed Sep 9, 2026

View reviewed changes

Comment thread src/component/README.md Outdated

ianmacartney force-pushed the ian/use-commit-ts branch from 90bf8f7 to 0b0b60e Compare September 9, 2026 23:56

ianmacartney changed the title ~~use commitTS for workpool~~ Order pending work by commit and snapshot timestamps Sep 9, 2026

This was referenced Sep 9, 2026

Pack pending starts into shared documents #234

Open

Add queue benchmarks and record results #235

Open

ianmacartney requested a review from reeceyang September 10, 2026 00:38

cubic-dev-ai Bot reviewed Sep 10, 2026

View reviewed changes

Comment thread src/component/stateMachine.test.ts Outdated

ianmacartney force-pushed the ian/use-commit-ts branch 2 times, most recently from 2ccceca to ba7eb18 Compare September 12, 2026 00:02

reeceyang approved these changes Sep 15, 2026

View reviewed changes

ianmacartney force-pushed the ian/use-commit-ts branch from ba7eb18 to 98666b0 Compare September 15, 2026 23:31

ianmacartney mentioned this pull request Sep 18, 2026

attempt recovery of wedged mutations #240

Draft

ianmacartney force-pushed the ian/use-commit-ts branch 2 times, most recently from 65c31bd to 65184eb Compare October 10, 2026 01:35

ianmacartney force-pushed the ian/use-commit-ts branch from 65184eb to 155405d Compare October 22, 2026 01:39

ianmacartney added 17 commits October 27, 2026 18:44

feat(queue): order work by commit and snapshot timestamps

290a597

docs(queue): name the persisted sweep cursor

5a6669d

nit

184a9b6

refactor(queue): confine legacy handling to the upgrade pass

c68d36d

perf(queue): avoid redundant cancellation reads and writes

a7a60d6

fix(queue): gate starts on wall-clock time

c758c8a

perf(stats): restore bounded backlog reporting in the loop

2a0ba69

refactor(queue): select start capacity once per batch

864a0fe

fix(queue): preserve legacy start ordering during upgrade

677a7c4

test(queue): derive recovery timing from the production period

309f7d0

feedback

f2c9ff7

changelog update

a210968

drop paginator again

56ce9e3

clean up test

5890e72

update peer deps

2bd9f20

leverage snapshotTs ≤ Date.now in backend & tests

ca4c97a

feedback

0287944

ianmacartney changed the base branch from main to graphite-base/220 October 28, 2026 02:00

ianmacartney force-pushed the ian/use-commit-ts branch from 155405d to 0287944 Compare October 28, 2026 02:01

ianmacartney changed the base branch from graphite-base/220 to ian/pending-start-pointer October 28, 2026 02:01

ianmacartney mentioned this pull request Oct 28, 2026

Record pendingStartId on work and upgrade batch-worker to 0.3 #252

Open

		// (1) Out-of-order entries left at the boundary stamp. Nothing at or above
		// the cursor is read, so a bulk enqueue sharing this stamp costs nothing.

Conversation

ianmacartney commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianmacartney commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pkg-pr-new Bot commented Aug 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianmacartney commented Aug 4, 2026

Uh oh!

reeceyang left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ianmacartney commented Sep 9, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 9, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ianmacartney commented Sep 9, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

ianmacartney commented Aug 4, 2026 •

edited

Loading

ianmacartney commented Aug 4, 2026 •

edited

Loading

pkg-pr-new Bot commented Aug 4, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading