Add pgqueuer - #3360
Open
janbjorge wants to merge 1 commit into
Open
Add pgqueuer#3360janbjorge wants to merge 1 commit into
janbjorge wants to merge 1 commit into
Conversation
Task Queues is at the five-entry cap. pgqueuer takes taskiq's challenger seat: an asyncio queue that uses PostgreSQL instead of a separate broker.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Project
pgqueuer
Checklist
Add project-name- [pypi-name](https://github.com/owner/repo) - Description ending with period.Which Tier
Pick one:
Explain:
Challenger, on the asyncio seat. pgqueuer is a PostgreSQL-backed job queue: an enqueue commits in the same transaction as the data it belongs to, workers claim with
FOR UPDATE SKIP LOCKED, andLISTEN/NOTIFYwakes them. Cron-style schedules run in the same process. PyPI namepgqueuer, Production/Stable, public since April 2024, docs at https://janbjorge.github.io/pgqueuer/. I maintain it.Adoption, from pypistats overall downloads without mirrors:
Rolling last 30 days (
/recent): pgqueuer 482,135, taskiq 2,012,170, huey 14,082,843. pgqueuer has about tripled since April and is still about a quarter of taskiq. It is placed last in the challenger tier, after huey, which is the download order.Displacement
Task Queues is at its cap of five. This PR removes taskiq and adds pgqueuer.
The seat taskiq holds is native asyncio. taskiq's production path is a broker (its docs recommend aio-pika, NATS, or Redis). pgqueuer does that asyncio job with the PostgreSQL database the application already runs, so the job and the rows it refers to commit together. Huey stays: it is the lightweight small-app queue, including SQLite, and its download count is an order of magnitude higher.
The download signal still favors taskiq. The case for displacement is the job, not the count.
website/data/category_intros/task-queues.mdstill names taskiq as the asyncio pick. I left that page alone. I can update it here if you want the intro to match the list.