feat(webhook): support multi-region SQS dispatch in runner webhook#5099
Open
bogdankrasko wants to merge 1 commit intogithub-aws-runners:mainfrom
Open
Conversation
Derive the SQS client region from the queue URL instead of always using AWS_REGION. This prevents cross-region SignatureDoesNotMatch failures when the webhook sends messages to queues outside the Lambda region. Cache traced SQS clients by region to avoid recreating clients for repeated sends, while still creating separate clients for different queue regions. Fall back to AWS_REGION when the queue URL cannot be parsed, and fall back to the SDK default region resolution when no region is available. Expand unit coverage for queue URL region parsing, AWS_REGION fallback, missing-region behavior, same-region client reuse, and per-region client separation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR updates the webhook SQS dispatch logic to select the SQS client region from the target queue URL instead of always using
AWS_REGION.This fixes cross-region dispatch scenarios where the webhook needs to send workflow job messages to an SQS queue in a different AWS region than the Lambda runtime. The change also caches SQS clients per region so repeated sends reuse the correct client while still creating separate clients when queues are in different regions.
To support this model, the webhook
runner_matcher_configmust include each runner group'smatcherConfigtogether with the region-specific SQS queue URL (id) and queue ARN (arn) for every runner group in every region. This allows the webhook to match the job and dispatch it to the correct regional queue.This also requires runner pool labels to be unique across all configured regions so webhook matching remains deterministic and a given job maps to exactly one regional queue.
Test Plan
lambdas/functions/webhook/src/sqs/index.test.tsto cover:AWS_REGIONwhen the queue URL cannot be parsedyarn vitest run functions/webhook/src/sqs/index.test.tsus-east-2, so a single API Gateway/webhook entrypoint handles workflow dispatch for runner queues across multiple AWS regionsmodules/runners. This setup provisions per-runner-group Lambda functions (scale-up, scale-down, pool, and job-retry), along with the SQS queues defined in modules/multi-runner/queues.tf. This was used to validate both same-region and cross-region dispatch behavior.To enable it, an additional wrapper was introduced within
modules/runnersto incorporate the required SQS queues.us-east-2: Ubuntu 24.04 runner pools for bothamd64andarm64. This region was used to validate that jobs with architecture-specific labels were matched correctly and dispatched successfully to queues in the local region.us-west-2: Ubuntu 24.04amd64-2runner pool. This region was used as the secondary-region target to validate that the webhook could dispatch a workflow job to an SQS queue outside the Lambda runtime region.Related Issues