Skip to content

Batch the delayed-to-ready move to avoid Lua unpack limit - #14

Open
himynameisjonas wants to merge 1 commit into
socketry:mainfrom
himynameisjonas:fix/batch-delayed-move
Open

Batch the delayed-to-ready move to avoid Lua unpack limit#14
himynameisjonas wants to merge 1 commit into
socketry:mainfrom
himynameisjonas:fix/batch-delayed-move

Conversation

@himynameisjonas

@himynameisjonas himynameisjonas commented Sep 8, 2026

Copy link
Copy Markdown

DelayedJobs#move reads every due job and unpacks all of them into one LPUSH. Redis caps Lua's unpack at 8000 arguments, so the script blows up as soon as 8000 jobs are due in the same tick. Worse, Redis doesn't roll a script back when it raises halfway through: the ZREMRANGEBYSCORE has already committed by then, so those jobs are gone from the sorted set and never reach the ready list. They're not late, they're lost, and nothing logs it.

You need a big pile of due jobs to hit this, which means a stretch where nothing is sweeping: a long deploy, an incident, a queue scaled to zero. We schedule with a 30 minute coalescing delay so everything sits in the delayed set, and at our current rate a worker gap of a few minutes would do it.

Reproduced against Redis 8.6, where 7999 is fine and 8000 loses the lot:

n=7999 -> moved 7999, ready=7999, delayed left=0
n=8000 -> RAISED: too many results to unpack
          delayed=0 ready=0

Tested with the existing suite plus a new case that puts 8500 due jobs through drain. move keeps its signature apart from an optional limit:.

I've guessed at 1000 for the default, and drain might be better folded into move than sitting next to it. Happy to reshape either.

@himynameisjonas
himynameisjonas marked this pull request as ready for review September 8, 2026 13:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant