Batch the delayed-to-ready move to avoid Lua unpack limit - #14
Open
himynameisjonas wants to merge 1 commit into
Open
Batch the delayed-to-ready move to avoid Lua unpack limit#14himynameisjonas wants to merge 1 commit into
himynameisjonas wants to merge 1 commit into
Conversation
himynameisjonas
force-pushed
the
fix/batch-delayed-move
branch
from
September 8, 2026 13:21
02c261f to
39d27dc
Compare
himynameisjonas
marked this pull request as ready for review
September 8, 2026 13:27
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DelayedJobs#movereads every due job and unpacks all of them into oneLPUSH. Redis caps Lua'sunpackat 8000 arguments, so the script blows up as soon as 8000 jobs are due in the same tick. Worse, Redis doesn't roll a script back when it raises halfway through: theZREMRANGEBYSCOREhas already committed by then, so those jobs are gone from the sorted set and never reach the ready list. They're not late, they're lost, and nothing logs it.You need a big pile of due jobs to hit this, which means a stretch where nothing is sweeping: a long deploy, an incident, a queue scaled to zero. We schedule with a 30 minute coalescing delay so everything sits in the delayed set, and at our current rate a worker gap of a few minutes would do it.
Reproduced against Redis 8.6, where 7999 is fine and 8000 loses the lot:
Tested with the existing suite plus a new case that puts 8500 due jobs through
drain.movekeeps its signature apart from an optionallimit:.I've guessed at 1000 for the default, and
drainmight be better folded intomovethan sitting next to it. Happy to reshape either.