Conversation
Serverless Land pattern for Kafka Queue mode (KIP-932) with self-managed Apache Kafka 4.2+ and AWS Lambda. Pattern includes: - 4 CloudFormation stacks (network, broker, app, observability) - Python 3.12 worker Lambda with partial batch response - Python 3.12 producer Lambda for testing - create-esm.sh script (ConsumptionMode not yet in SAM/CFN schema) - 3 deployment paths: full / bring-your-own-Kafka / bring-your-own-Kafka+VPC - CloudWatch dashboard and alarms - Local test event Author: Vaibhav Jain (AWS Senior Delivery Consultant)
Keeps records inflight long enough to observe multiple pollers processing the same partition simultaneously during scaling tests.
Unterminated string literal in --payload print statement caused the script to exit with SyntaxError after ESM creation succeeded. Tested end-to-end against us-west-2.
Replaces all kqd- prefixes with kafka-queue- in stack names, CFN exports, resource names, scripts, and README so the pattern is self-contained and doesn't conflict with the kqd demo account.
- Dimension: FunctionName -> EventSourceMappingUUID - MaxOffsetLag -> MaxShareGroupLag - SumOffsetLag -> SumShareGroupLag - Remove unused SNS topic and AcknowledgedEventCount - Add ESMUuid parameter - Verified all metrics populate in us-west-2 kqd account
- Add scripts/setup-vpc-endpoints.sh — idempotent creation of Lambda, STS, and SQS interface endpoints. Supports Path A (resolves from kafka-queue-network stack) and Path B/C (accepts --vpc-id, --subnet-ids, --security-group-id flags for BYO VPC). - README Path A: add Step 4 (VPC endpoints) and Step 5 (ESM), move observability to Step 6 with required ESMUuid parameter. - README Path B: add VPC endpoint step with BYO VPC flags. - README Path C: clarify setup-vpc-endpoints.sh usage. - 1-network.yaml: update comment to document all 3 required endpoints including SQS, with explanation of silent failure if SQS is missing.
- Remove DynamoDB from architecture diagram and costs (table removed) - Fix 'KQD Stack N' descriptions in stacks 1, 2, 3 - Add topic override example to Testing section - Add CloudWatch dashboard section with metric descriptions - Add VPC endpoint cleanup to Cleanup section
UserData was fragile — silent exits on newer AL2023 kernels when combining exec > >(tee ...) with set -euo pipefail. The wget also takes 15-20 min which required a 40-min CFN timeout. New approach: - 2-broker.yaml: launches EC2 instance only (no UserData) - scripts/setup-broker.sh: installs Kafka via 6 sequential SSM send-command steps with progress reporting and error handling - README: updated all 3 paths to reference setup-broker.sh No SSH or bastion required — all steps run via SSM.
Double quotes inside the SSM --parameters JSON string caused a ParamValidation error. Removed surrounding quotes from the advertised.listeners value — no quotes needed since there are no spaces. Validated end-to-end: pattern deploys successfully and processes records with Queue mode on a fresh account deployment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a new pattern that deploys an AWS Lambda function consuming from a self-managed Apache Kafka 4.2+ cluster using Queue consumption mode (KIP-932 Share Groups). In Queue mode, multiple Lambda pollers can process records from the same partition concurrently — parallelism is decoupled from partition count, unlike Stream mode where each partition maps to a single consumer.
Architecture
Producer Lambda -> self-managed Apache Kafka 4.2+ cluster (KRaft, share groups enabled) -> Lambda ESM in Queue mode (
ConsumptionMode: Queue, provisioned pollers 2-10) -> Worker Lambda (Python 3.12, partial batch response) -> Amazon SQS DLQ for records that exhaustMaximumRetryAttempts. Observability via a CloudWatch dashboard and alarms for share-group lag, DLQ delivery, and poller errors.Deployment is split into four independent stacks (network, broker, app, observability) so users can bring their own Kafka cluster and/or VPC.
Language / Framework
Python 3.12, AWS SAM + AWS CloudFormation.
Note for reviewers —
ConsumptionMode: QueueschemaConsumptionMode: Queueis a pre-release Lambda API field not yet in the AWS CLI or CloudFormation schema. The pattern therefore creates the ESM viascripts/create-esm.sh, which calls the Lambda REST API directly usingcurl --aws-sigv4. AWS CLI support for this field is landing on Sep 25, 2026. Once available, we intend to submit a follow-up PR replacing the curl call increate-esm.shwith the standardaws lambda create-event-source-mappingCLI command and updating the accompanying blog.Testing
Deployed and validated end-to-end (produce, consume, retry, DLQ, share-group behavior with concurrent pollers > partitions).