Conversation
Serverless Land pattern for Kafka Queue mode (KIP-932) with self-managed Apache Kafka 4.2+ and AWS Lambda. Pattern includes: - 4 CloudFormation stacks (network, broker, app, observability) - Python 3.12 worker Lambda with partial batch response - Python 3.12 producer Lambda for testing - create-esm.sh script (ConsumptionMode not yet in SAM/CFN schema) - 3 deployment paths: full / bring-your-own-Kafka / bring-your-own-Kafka+VPC - CloudWatch dashboard and alarms - Local test event Author: Vaibhav Jain (AWS Senior Delivery Consultant)
Keeps records inflight long enough to observe multiple pollers processing the same partition simultaneously during scaling tests.
Unterminated string literal in --payload print statement caused the script to exit with SyntaxError after ESM creation succeeded. Tested end-to-end against us-west-2.
Replaces all kqd- prefixes with kafka-queue- in stack names, CFN exports, resource names, scripts, and README so the pattern is self-contained and doesn't conflict with the kqd demo account.
- Dimension: FunctionName -> EventSourceMappingUUID - MaxOffsetLag -> MaxShareGroupLag - SumOffsetLag -> SumShareGroupLag - Remove unused SNS topic and AcknowledgedEventCount - Add ESMUuid parameter - Verified all metrics populate in us-west-2 kqd account
- Add scripts/setup-vpc-endpoints.sh — idempotent creation of Lambda, STS, and SQS interface endpoints. Supports Path A (resolves from kafka-queue-network stack) and Path B/C (accepts --vpc-id, --subnet-ids, --security-group-id flags for BYO VPC). - README Path A: add Step 4 (VPC endpoints) and Step 5 (ESM), move observability to Step 6 with required ESMUuid parameter. - README Path B: add VPC endpoint step with BYO VPC flags. - README Path C: clarify setup-vpc-endpoints.sh usage. - 1-network.yaml: update comment to document all 3 required endpoints including SQS, with explanation of silent failure if SQS is missing.
- Remove DynamoDB from architecture diagram and costs (table removed) - Fix 'KQD Stack N' descriptions in stacks 1, 2, 3 - Add topic override example to Testing section - Add CloudWatch dashboard section with metric descriptions - Add VPC endpoint cleanup to Cleanup section
UserData was fragile — silent exits on newer AL2023 kernels when combining exec > >(tee ...) with set -euo pipefail. The wget also takes 15-20 min which required a 40-min CFN timeout. New approach: - 2-broker.yaml: launches EC2 instance only (no UserData) - scripts/setup-broker.sh: installs Kafka via 6 sequential SSM send-command steps with progress reporting and error handling - README: updated all 3 paths to reference setup-broker.sh No SSH or bastion required — all steps run via SSM.
Double quotes inside the SSM --parameters JSON string caused a ParamValidation error. Removed surrounding quotes from the advertised.listeners value — no quotes needed since there are no spaces. Validated end-to-end: pattern deploys successfully and processes records with Queue mode on a fresh account deployment.
bfreiberg
requested changes
Sep 21, 2026
| - ec2:DescribeVpcs | ||
| - ec2:DescribeSubnets | ||
| - ec2:DescribeSecurityGroups | ||
| Resource: '*' |
Contributor
There was a problem hiding this comment.
WorkerFunction is granted ec2:CreateNetworkInterface, DescribeNetworkInterfaces, DeleteNetworkInterface, etc., but the function has no VpcConfig. The worker is invoked by the ESM pollers and runs in the Lambda-managed VPC — it never creates ENIs in the customer VPC (the pollers handle Kafka connectivity via SourceAccessConfigurations). These permissions appear unused and can be removed; the inline policy can be reduced to just the sqs:SendMessage statement on the DLQ.
| **Step 3: Build and deploy the application** | ||
|
|
||
| ```bash | ||
| sam build --template stacks/3-app.yaml |
Contributor
There was a problem hiding this comment.
confluent-kafka ships as a compiled wheel (wraps librdkafka), so a plain sam build on a host whose platform/Python doesn't match the Lambda python3.12 x86_64 runtime can produce an incompatible artifact that fails at import time. Recommend sam build --template stacks/3-app.yaml --use-container to guarantee a Lambda-compatible build (verified working). Worth applying to the Path B block too.
Comment on lines
+364
to
+374
| ## Pattern details | ||
|
|
||
| | Property | Value | | ||
| |----------|-------| | ||
| | Kafka version required | Apache Kafka 4.2+ | | ||
| | Lambda runtime | Python 3.12 | | ||
| | IaC framework | AWS SAM + AWS CloudFormation | | ||
| | Delivery semantics | At-least-once | | ||
| | Ordering guarantees | None (Queue mode) | | ||
| | Authentication | PLAINTEXT (see notes for SASL/SCRAM) | | ||
| | Region | Configurable | |
Contributor
There was a problem hiding this comment.
Remove but add copyright footer
| @@ -0,0 +1,70 @@ | |||
| { | |||
| "title": "Self-managed Apache Kafka Queue mode to AWS Lambda (KIP-932)", | |||
| "description": "Deploy a Lambda function that consumes from a self-managed Apache Kafka 4.2+ cluster using Queue consumption mode (KIP-932 Share Groups). Queue mode allows multiple Lambda pollers to process records from the same partition concurrently, breaking the partition-count ceiling of traditional consumer groups.", | |||
Contributor
There was a problem hiding this comment.
description must be 175 characters or fewer
| # | ||
| # These are created via scripts/setup-vpc-endpoints.sh rather than as | ||
| # CloudFormation resources because AWS::EC2::VpcEndpoint is restricted | ||
| # in some accounts (e.g. Isengard/bindled accounts used for testing). |
Contributor
There was a problem hiding this comment.
This leaks internal information, remove
Co-authored-by: Ben <9841563+bfreiberg@users.noreply.github.com>
WorkerFunction has no VpcConfig — it runs in Lambda-managed VPC and never creates ENIs in the customer VPC. The ESM pollers handle Kafka connectivity via SourceAccessConfigurations. Only sqs:SendMessage is needed for the DLQ on-failure destination.
confluent-kafka ships as a compiled wheel wrapping librdkafka. Building without --use-container on a non-Linux host produces an incompatible binary that fails at Lambda import time.
Bare except without logging silently masks decode errors. Adding logger.warning with exc_info=True makes decode failures visible in CloudWatch logs.
bfreiberg
approved these changes
Sep 22, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds a new pattern that deploys an AWS Lambda function consuming from a self-managed Apache Kafka 4.2+ cluster using Queue consumption mode (KIP-932 Share Groups). In Queue mode, multiple Lambda pollers can process records from the same partition concurrently — parallelism is decoupled from partition count, unlike Stream mode where each partition maps to a single consumer.
Architecture
Producer Lambda -> self-managed Apache Kafka 4.2+ cluster (KRaft, share groups enabled) -> Lambda ESM in Queue mode (
ConsumptionMode: Queue, provisioned pollers 2-10) -> Worker Lambda (Python 3.12, partial batch response) -> Amazon SQS DLQ for records that exhaustMaximumRetryAttempts. Observability via a CloudWatch dashboard and alarms for share-group lag, DLQ delivery, and poller errors.Deployment is split into four independent stacks (network, broker, app, observability) so users can bring their own Kafka cluster and/or VPC.
Language / Framework
Python 3.12, AWS SAM + AWS CloudFormation.
Note for reviewers —
ConsumptionMode: QueueschemaConsumptionMode: Queueis a pre-release Lambda API field not yet in the AWS CLI or CloudFormation schema. The pattern therefore creates the ESM viascripts/create-esm.sh, which calls the Lambda REST API directly usingcurl --aws-sigv4. AWS CLI support for this field is landing on Sep 25, 2026. Once available, we intend to submit a follow-up PR replacing the curl call increate-esm.shwith the standardaws lambda create-event-source-mappingCLI command and updating the accompanying blog.Testing
Deployed and validated end-to-end (produce, consume, retry, DLQ, share-group behavior with concurrent pollers > partitions).