Repository navigation
Queue fix for workflow engines - #6425
Open
guzzijones wants to merge 3 commits into
Open
guzzijones wants to merge 3 commits into
guzzijones wants to merge 3 commits into
Conversation
Contributor
Author
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stop services from pulling more RabbitMQ messages than they can process
Problem
st2 workflow engine gets bogged down with cpu load zstandard compressing and decompressing large workflow contexts. There is no real way to scale workflow engines currently.
Services that consume from RabbitMQ acked every message as soon as it arrived and queued the work in an
in-memory
BufferedDispatcherbuffer until a green thread was free. Under load (or when a service wasslow), that meant:
Change
Consumers now only take a message from RabbitMQ when they have a free slot to process it.
QueueConsumersetsprefetch_count=1, and its callback waits indispatcher.acquire()for a freeslot before acking. Until the ack goes out, RabbitMQ doesn't send this consumer another message,
so the backlog stays in the broker.
BufferedDispatcherno longer buffers. It's now a slot limiter:acquire()/release()/dispatch(), built on a semaphore.acquire()checks for shutdown every second and still logs whenthe pool has been busy for more than 60 seconds.
message to another instance. Work that is already running is not interrupted.
pre_ack_processfails (staged handlers) or dispatch fails, the slot is released. Payloads thatfail validation are acked and dropped, as before.
MessageHandler.get_concurrency().New config options
[workflow_engine].concurrency1[notifier].concurrency50[rulesengine].concurrency50[scheduler].concurrency50Each option is the maximum number of messages one process of that service works on at the same time.
Further messages stay in RabbitMQ until a slot frees up.
st2actionrunner
The actionrunner keeps its two pools (
actions_pool_size=60,workflows_pool_size=40). Defaults andhow many actions run at once are unchanged; only where the backlog waits is different (RabbitMQ instead
of memory).
Both pools are fed by one consumer, so while the pool the next message needs is full, nothing more is
pulled for either pool. This doesn't matter in practice: orquesta workflows run in st2workflowengine,
and the actionrunner only holds a workflow slot long enough to hand the request off, so the workflows
pool doesn't fill. A full runner now holds at most one waiting message, and the rest go to runners with
free capacity.
Capacity is still added by running more actionrunner processes or replicas.
Tests
st2common/tests/unit/test_queue_consumer.pyget_concurrency()sets the dispatcher sizepre_ack_processfailure the message is acked and the slot is releasedActionsQueueConsumerunit and integration tests are unchanged.Upgrade notes
No migration is needed. When a service process is busy, operators will see more messages ready in
RabbitMQ and lower service memory use. That's expected. The new
concurrencyoptions are the knobs ifa service should process more at once per process.