Skip to content

Queue fix for workflow engines - #6425

Open
guzzijones wants to merge 3 commits into
StackStorm:masterfrom
guzzijones:queue_fix
Open

guzzijones wants to merge 3 commits into
StackStorm:masterfrom
guzzijones:queue_fix

Conversation

@guzzijones

Copy link
Copy Markdown
Contributor

Stop services from pulling more RabbitMQ messages than they can process

Problem

st2 workflow engine gets bogged down with cpu load zstandard compressing and decompressing large workflow contexts. There is no real way to scale workflow engines currently.
Services that consume from RabbitMQ acked every message as soon as it arrived and queued the work in an
in-memory BufferedDispatcher buffer until a green thread was free. Under load (or when a service was
slow), that meant:

  • The backlog moved from RabbitMQ into service memory, with no upper limit. An st2 workflow engine would swallow 50 jobs. then get bogged down on one large with items workflow compressing and decompressing context.
  • A busy instance kept taking messages that idle instances could have handled.
  • Everything in that buffer had already been acked, so it was lost if the process died or restarted.

Change

Consumers now only take a message from RabbitMQ when they have a free slot to process it.

  • QueueConsumer sets prefetch_count=1, and its callback waits in dispatcher.acquire() for a free
    slot before acking. Until the ack goes out, RabbitMQ doesn't send this consumer another message,
    so the backlog stays in the broker.
  • BufferedDispatcher no longer buffers. It's now a slot limiter: acquire() / release() /
    dispatch(), built on a semaphore. acquire() checks for shutdown every second and still logs when
    the pool has been busy for more than 60 seconds.
  • On shutdown, a callback waiting for a slot returns without acking, so RabbitMQ redelivers that
    message to another instance. Work that is already running is not interrupted.
  • When pre_ack_process fails (staged handlers) or dispatch fails, the slot is released. Payloads that
    fail validation are acked and dropped, as before.
  • Handlers set their own limit with MessageHandler.get_concurrency().
  • New config options

Option Default Service
[workflow_engine].concurrency 1 st2workflowengine
[notifier].concurrency 50 st2notifier
[rulesengine].concurrency 50 st2rulesengine
[scheduler].concurrency 50 st2scheduler (entrypoint consumer)

Each option is the maximum number of messages one process of that service works on at the same time.
Further messages stay in RabbitMQ until a slot frees up.

st2actionrunner

The actionrunner keeps its two pools (actions_pool_size=60, workflows_pool_size=40). Defaults and
how many actions run at once are unchanged; only where the backlog waits is different (RabbitMQ instead
of memory).

Both pools are fed by one consumer, so while the pool the next message needs is full, nothing more is
pulled for either pool. This doesn't matter in practice: orquesta workflows run in st2workflowengine,
and the actionrunner only holds a workflow slot long enough to hand the request off, so the workflows
pool doesn't fill. A full runner now holds at most one waiting message, and the rest go to runners with
free capacity.

Capacity is still added by running more actionrunner processes or replicas.

Tests

  • st2common/tests/unit/test_queue_consumer.py
    • the message is acked before processing
    • the next message isn't acked or processed while every slot is in use
    • a message waiting for a slot isn't acked on shutdown
    • get_concurrency() sets the dispatcher size
    • on a staged pre_ack_process failure the message is acked and the slot is released
  • The existing ActionsQueueConsumer unit and integration tests are unchanged.

Upgrade notes

No migration is needed. When a service process is busy, operators will see more messages ready in
RabbitMQ and lower service memory use. That's expected. The new concurrency options are the knobs if
a service should process more at once per process.

@pull-request-size pull-request-size Bot added the size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review. label Nov 20, 2026
@guzzijones guzzijones changed the title Queue fix Queue fix for workflow engines Nov 20, 2026
@guzzijones

Copy link
Copy Markdown
Contributor Author
1. Message map (which service consumes which queue)
Every consumer marked [*] now uses the new pull model and has a limit on how many messages it works on at once.
 PUBLISHERS                     EXCHANGE                      QUEUE                                  CONSUMER
 ──────────                     ────────                      ─────                                  ────────

 sensors ──────────────────▶ st2.trigger_instances_dispatch ──▶ st2.trigger_instances_dispatch     ──▶ [*] st2rulesengine
                                                                 .rules_engine        (rk: #)           [rulesengine].concurrency = 50
                                                                                                        (staged: TriggerInstance saved
                                                                                                         to DB before ack)
                                                                                                              │
 st2api / rules engine /                                                                                      │ requests action
 workflow engine ───────────▶ st2.liveaction.status ─┬─────────▶ st2.actionrunner.req   (requested) ──▶ [*] st2scheduler (entrypoint)
   ▲                                                  │                                                  [scheduler].concurrency = 50
   │                                                  │                                                         │
   │                                                  │                                                         ▼
   │                                                  │                                               Mongo execution queue
   │                                                  │                                                         │
   │                                                  │                                              scheduler handler loop
   │                                                  │                                              (polls Mongo, not RabbitMQ)
   │                                                  │                                                         │ publishes "scheduled"
   │                                                  │◀────────────────────────────────────────────────────────┘
   │                                                  │
   │                                                  ├─────────▶ st2.actionrunner.work   (scheduled) ──┐
   │                                                  ├─────────▶ st2.actionrunner.cancel (canceling) ──┤
   │                                                  ├─────────▶ st2.actionrunner.pause  (pausing)   ──┼──▶ [*] st2actionrunner
   │                                                  └─────────▶ st2.actionrunner.resume (resuming)  ──┘      actions pool   = 60
   │                                                                                                           workflows pool = 40
   │                                                                                                           (one consumer, 4 queues)
   │                                                                                                                │
   │                                                                     orquesta action: hand off ◀────────────────┘
   │                                                                                │
   │                          st2.workflow.status ─┬─────────▶ st2.workflow.work   (requested) ──┐
   │                                               └─────────▶ st2.workflow.resume (resuming)  ──┼──▶ [*] st2workflowengine
   │                          st2.execution ───────┬─────────▶ st2.workflow.action.update      ──┘      [workflow_engine].concurrency = 1  (!)
   │                                               │                       (update)                            │
   └───────────────────────────────────────────────┼─── workflow engine requests task actions ◀────────────────┘
                                                   │
                                                   └─────────▶ st2.notifiers.execution.work      ──▶ [*] st2notifier
                                                                           (update)                    [notifier].concurrency = 50

 Not changed: st2stream (temporary exclusive queues), sensor container, st2api.
2. Inside one consumer: before and after
 BEFORE: ack on arrival, buffer in memory
 ────────────────────────────────────────

   RabbitMQ queue                    service process
  ┌───────────────┐   deliver   ┌─────────────────────────────────────────────────┐
  │ ▓▓▓▓▓▓▓▓▓▓▓▓▓ │ ──────────▶ │ callback: ack() immediately                     │
  │               │ ◀── ack ─── │      │                                          │
  │ (nearly empty │             │      ▼                                          │
  │  -- everything│             │ ┌───────────────────────────────┐               │
  │  was pulled)  │             │ │ in-memory buffer (no limit)   │  ◀── grows    │
  └───────────────┘             │ │ ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓ │      under    │
                                │ └──────────────┬────────────────┘      load     │
                                │                ▼                                │
                                │ ┌───────────────────────────────┐               │
                                │ │ green pool  [■■■■■■■■■■] full │               │
                                │ └───────────────────────────────┘               │
                                └─────────────────────────────────────────────────┘
   Problems: unbounded memory, a busy instance takes work idle ones could do,
             and acked messages still in the buffer are lost on crash or restart.


 AFTER: prefetch_count=1, wait for a slot, then ack
 ──────────────────────────────────────────────────

   RabbitMQ queue                    service process
  ┌───────────────┐   deliver   ┌─────────────────────────────────────────────────┐
  │ ▓▓▓▓▓▓▓▓▓▓▓▓▓ │ ──(1 max)─▶ │ callback:                                       │
  │ ▓▓▓▓▓▓▓▓▓▓▓▓▓ │             │   1. validate      (bad payload → ack + drop)   │
  │ backlog stays │             │   2. acquire()     waits here while pool full   │
  │ here, other   │             │   3. pre_ack       (staged handlers only)       │
  │ instances can │ ◀── ack ─── │   4. ack()         → broker sends next message  │
  │ take it       │             │   5. dispatch()    → runs in a green thread     │
  └───────────────┘             │                         │                       │
                                │ ┌───────────────────────▼──────────┐            │
                                │ │ slots (semaphore) [■■■■■□□□] N   │            │
                                │ │ slot released when handler ends  │            │
                                │ └──────────────────────────────────┘            │
                                └─────────────────────────────────────────────────┘
   On shutdown: acquire() returns False → no ack → RabbitMQ redelivers elsewhere.
3. The actionrunner special case
  st2.actionrunner.work   ─┐
  st2.actionrunner.cancel ─┤     ONE consumer, ONE channel, prefetch_count=1
  st2.actionrunner.pause  ─┼──▶  ┌─────────────────────────────────────────────┐
  st2.actionrunner.resume ─┘     │ _get_dispatcher(body)                       │
                                 │   action_is_workflow ?                      │
                                 └──────────┬───────────────────────┬──────────┘
                                       no   │                       │  yes
                                            ▼                       ▼
                          ┌──────────────────────────┐  ┌──────────────────────────┐
                          │ actions pool   (60)      │  │ workflows pool  (40)     │
                          │ [■■■■■■■■■■■■■■■■□□□□□]  │  │ [■□□□□□□□□□□□□□□□□□□□□]  │
                          │ python / shell: child    │  │ orquesta: hand off to    │
                          │   processes (other CPUs) │  │   st2workflowengine and  │
                          │ http / ssh / winrm:      │  │   return right away      │
                          │   in-process (1 CPU)     │  │ → this pool barely fills │
                          └──────────────────────────┘  └──────────────────────────┘

   If the next message's pool is full, the consumer waits and nothing is pulled for EITHER pool.
   That's fine in practice: the workflows pool stays nearly empty, and a full runner holds
   only ONE waiting message, so the rest of the queue goes to other runners:

        RabbitMQ st2.actionrunner.work
        ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
          │        │         │
          ▼        ▼         ▼
       runner A  runner B  runner C
       [full]    [free]    [free]
       holds 1   takes     takes
       waiting   work      work
The (!) on the workflow engine marks the default drop from 50 to 1 that I mentioned earlier. You'll want to decide whether that's intended before you post the PR.

@guzzijones guzzijones added this to the 3.11.0 milestone Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XL PR that changes 500-999 lines. Consider splitting work into several ones that easier to review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant