Event-Driven AWS: EventBridge, SQS, Lambda and DLQs

This designs an event-driven pipeline on Lambda and EventBridge that survives poison messages, retry storms, traffic spikes, and cold-start latency, rather than just the happy path.

EventBridge: route by pattern, not by function

Don't put business logic in the rule and don't fan every event into one general-purpose Lambda. Define narrow event patterns so each consumer only ever sees what it's built to handle.

json

{
  "source": ["com.acme.orders"],
  "detail-type": ["OrderCompleted"],
  "detail": {
    "status": ["completed"],
    "customerId": [{ "exists": true }]
  }
}

That exists check alone keeps a payload with a missing customerId off the consumer: it either gets caught earlier in the pipeline or routed to a separate "malformed event" rule that alerts a human. Pattern matching is free; use it as a validation layer as well as a router.

The bus and the rule live in Terraform with everything else (I keep the pattern above in patterns/order-completed.json):

hcl

resource "aws_cloudwatch_event_bus" "orders" {
  name = "orders"
}
 
resource "aws_cloudwatch_event_rule" "order_completed" {
  name           = "order-completed"
  event_bus_name = aws_cloudwatch_event_bus.orders.name
  event_pattern  = file("${path.module}/patterns/order-completed.json")
}

Lambda concurrency: reserved and provisioned, deliberately

Two settings matter here: reserved concurrency caps how many concurrent executions a function can consume, stopping one noisy event source from starving every other function in your account of the shared concurrency pool; provisioned concurrency keeps a set number of execution environments warm, eliminating cold starts for that traffic.

Provisioned concurrency can't be attached to $LATEST, only to a published version or an alias. So the function publishes a version on every code change (publish = true), a live alias points at it, and the warm pool hangs off the alias:

hcl

data "aws_iam_policy_document" "lambda_assume" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}
 
resource "aws_iam_role" "order_processor" {
  name               = "order-processor"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}
 
# CloudWatch Logs + sqs:ReceiveMessage/DeleteMessage/GetQueueAttributes
resource "aws_iam_role_policy_attachment" "order_processor_sqs" {
  role       = aws_iam_role.order_processor.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaSQSQueueExecutionRole"
}
 
# The CI job builds the bundle into dist/order-processor before terraform plan
data "archive_file" "order_processor" {
  type        = "zip"
  source_dir  = "${path.module}/dist/order-processor"
  output_path = "${path.module}/build/order-processor.zip"
}
 
resource "aws_lambda_function" "order_processor" {
  function_name    = "order-processor"
  role             = aws_iam_role.order_processor.arn
  runtime          = "nodejs24.x"
  handler          = "index.handler"
  filename         = data.archive_file.order_processor.output_path
  source_code_hash = data.archive_file.order_processor.output_base64sha256
  memory_size      = 512
  timeout          = 30
  publish          = true
 
  reserved_concurrent_executions = 50
 
  environment {
    variables = {
      IDEMPOTENCY_TABLE = aws_dynamodb_table.processed_events.name
    }
  }
}
 
resource "aws_lambda_alias" "live" {
  name             = "live"
  function_name    = aws_lambda_function.order_processor.function_name
  function_version = aws_lambda_function.order_processor.version
}
 
resource "aws_lambda_provisioned_concurrency_config" "order_processor_warm" {
  function_name                     = aws_lambda_alias.live.function_name
  qualifier                         = aws_lambda_alias.live.name
  provisioned_concurrent_executions = 10
}

If your pipeline already uploads build artifacts to S3, swap filename/source_code_hash for s3_bucket/s3_key; the rest stays the same.

Provisioned concurrency is billed whether or not it's invoked. For a Node.js function, a cold start is typically a few hundred milliseconds, more with a large bundle or heavy init code. I'd reserve provisioned concurrency for customer-facing traffic where that latency hurts, and let background and batch consumers cold-start.

SQS as a buffer: backpressure

Wiring EventBridge straight to Lambda means its concurrency scaling is dictated by the event source's burst rate. Add an SQS queue between the bus and the function and you get three things direct-invoke doesn't: a buffer that absorbs spikes, batching, and a place to attach a DLQ.

EventBridge needs explicit permission to write to the queue. Without a queue policy, the rule matches, delivery fails, and after its retries EventBridge drops the events. Scope the policy to this one rule:

hcl

data "aws_iam_policy_document" "order_buffer" {
  statement {
    sid       = "AllowOrderCompletedRule"
    actions   = ["sqs:SendMessage"]
    resources = [aws_sqs_queue.order_buffer.arn]
 
    principals {
      type        = "Service"
      identifiers = ["events.amazonaws.com"]
    }
 
    condition {
      test     = "ArnEquals"
      variable = "aws:SourceArn"
      values   = [aws_cloudwatch_event_rule.order_completed.arn]
    }
  }
}
 
resource "aws_sqs_queue_policy" "order_buffer" {
  queue_url = aws_sqs_queue.order_buffer.id
  policy    = data.aws_iam_policy_document.order_buffer.json
}
 
resource "aws_cloudwatch_event_target" "to_queue" {
  rule           = aws_cloudwatch_event_rule.order_completed.name
  event_bus_name = aws_cloudwatch_event_bus.orders.name
  arn            = aws_sqs_queue.order_buffer.arn
}

Lambda then polls the queue via an event source mapping, with its own concurrency and batching controls. It targets the live alias so it uses the warm environments, and ReportBatchItemFailures lets the handler fail individual messages instead of the whole batch:

hcl

resource "aws_lambda_event_source_mapping" "queue_to_lambda" {
  event_source_arn                   = aws_sqs_queue.order_buffer.arn
  function_name                      = aws_lambda_alias.live.arn
  batch_size                         = 10
  maximum_batching_window_in_seconds = 5
  function_response_types            = ["ReportBatchItemFailures"]
 
  scaling_config {
    maximum_concurrency = 20
  }
}

maximum_concurrency on the event source mapping is a second, more granular throttle than reserved concurrency on the function itself. Use it when one function is fed by multiple queues and you want per-queue limits.

The DLQ: where poison events go to die quietly

Instead of retrying an unrecoverable event indefinitely, route it to a dead-letter queue after a bounded number of attempts. Then a person, or an alarm, can look at what lands there.

hcl

resource "aws_sqs_queue" "order_dlq" {
  name                      = "order-processor-dlq"
  message_retention_seconds = 1209600 # 14 days, the maximum
}
 
resource "aws_sqs_queue" "order_buffer" {
  name = "order-buffer"
 
  # At least 6x the function timeout (30s) plus the batching window (5s)
  visibility_timeout_seconds = 190
 
  redrive_policy = jsonencode({
    deadLetterTargetArn = aws_sqs_queue.order_dlq.arn
    maxReceiveCount     = 5
  })
}

With an SQS event source mapping, Lambda invokes the function synchronously, so the failure path is this redrive policy. A message the handler keeps failing comes back after the visibility timeout, and after maxReceiveCount receives SQS moves it to the DLQ. Lambda's on_failure destinations only apply to asynchronous invocations, so they don't help here. The visibility timeout matters too. If it's shorter than the time Lambda can spend on a batch, messages reappear while they're still being processed, and you get duplicate work before your handler logic even runs.

Put a CloudWatch alarm on the DLQ's ApproximateNumberOfMessagesVisible so a poison event pages someone instead of sitting there for two weeks.

Idempotent handlers: the tax for at-least-once delivery

EventBridge and SQS guarantee at-least-once delivery, never exactly-once, combined with Lambda retries, any event can hit your handler two, three, or more times. If "process this order" means "charge this card," a duplicate delivery that isn't deduplicated is a duplicate charge.

The fix is a dedupe table keyed on the event's own idempotency key. I use the EventBridge event ID or a business key like order ID + status, whichever is stable across retries (the event ID survives EventBridge and SQS redelivery, but not a producer that re-publishes the same order):

hcl

resource "aws_dynamodb_table" "processed_events" {
  name         = "processed-events"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "eventId"
 
  attribute {
    name = "eventId"
    type = "S"
  }
 
  ttl {
    attribute_name = "expiresAt"
    enabled        = true
  }
}
 
resource "aws_iam_role_policy" "order_processor_dedupe" {
  name = "processed-events-table"
  role = aws_iam_role.order_processor.id
 
  policy = jsonencode({
    Version   = "2012-10-17"
    Statement = [{
      Effect   = "Allow"
      Action   = ["dynamodb:GetItem", "dynamodb:PutItem", "dynamodb:UpdateItem", "dynamodb:DeleteItem"]
      Resource = aws_dynamodb_table.processed_events.arn
    }]
  })
}

The naive version, "write the record, then do the work", has a hole. If the charge fails halfway, the record already says "done" and the retry skips it forever. So the record needs a state. Claim the key as IN_PROGRESS with a short lock expiry, run the side effect, then mark it COMPLETED. If the side effect throws, release the claim so the retry can run. If the function crashes or times out, the lock expires and the next delivery takes over. The handler reports failed messages back to SQS through batchItemFailures:

// index.mjs
import { DynamoDBClient, ConditionalCheckFailedException } from "@aws-sdk/client-dynamodb";
import {
  DynamoDBDocumentClient,
  PutCommand,
  GetCommand,
  UpdateCommand,
  DeleteCommand,
} from "@aws-sdk/lib-dynamodb";
 
const ddb = DynamoDBDocumentClient.from(new DynamoDBClient({}));
const TABLE = process.env.IDEMPOTENCY_TABLE;
const LOCK_SECONDS = 60; // longer than the function timeout
const RECORD_TTL_SECONDS = 7 * 24 * 3600; // longer than any redelivery window
 
async function claim(key) {
  const now = Math.floor(Date.now() / 1000);
  try {
    await ddb.send(
      new PutCommand({
        TableName: TABLE,
        Item: {
          eventId: key,
          status: "IN_PROGRESS",
          lockExpiresAt: now + LOCK_SECONDS,
          expiresAt: now + RECORD_TTL_SECONDS,
        },
        // New key, or a previous attempt died without finishing
        ConditionExpression:
          "attribute_not_exists(eventId) OR (#s = :inProgress AND lockExpiresAt < :now)",
        ExpressionAttributeNames: { "#s": "status" },
        ExpressionAttributeValues: { ":inProgress": "IN_PROGRESS", ":now": now },
      })
    );
    return true;
  } catch (err) {
    if (err instanceof ConditionalCheckFailedException) return false;
    throw err;
  }
}
 
async function processOnce(key, sideEffect) {
  if (!(await claim(key))) {
    const { Item } = await ddb.send(
      new GetCommand({ TableName: TABLE, Key: { eventId: key }, ConsistentRead: true })
    );
    if (Item?.status === "COMPLETED") return; // duplicate delivery: ack it
    throw new Error(`event ${key} is in progress elsewhere`); // retry later
  }
 
  try {
    await sideEffect();
  } catch (err) {
    // Release the claim so the next delivery can retry immediately
    await ddb.send(new DeleteCommand({ TableName: TABLE, Key: { eventId: key } }));
    throw err;
  }
 
  await ddb.send(
    new UpdateCommand({
      TableName: TABLE,
      Key: { eventId: key },
      UpdateExpression: "SET #s = :completed",
      ExpressionAttributeNames: { "#s": "status" },
      ExpressionAttributeValues: { ":completed": "COMPLETED" },
    })
  );
}
 
export const handler = async (event) => {
  const batchItemFailures = [];
 
  for (const record of event.Records) {
    try {
      const orderEvent = JSON.parse(record.body); // full EventBridge event
      await processOnce(orderEvent.id, () => chargeCustomer(orderEvent.detail));
    } catch (err) {
      console.error({ messageId: record.messageId, error: err.message });
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }
 
  // Only the failed messages go back to the queue (and eventually the DLQ)
  return { batchItemFailures };
};

chargeCustomer is your side effect. There's still a small window where the charge succeeds and the COMPLETED write fails, so I also pass the same key to the payment provider's own idempotency key where one exists. A rare re-run is then harmless end to end. TTL keeps the table from growing unbounded. DynamoDB deletes expired items lazily, though, which is why the lock uses its own lockExpiresAt attribute instead of relying on TTL.

In production I usually reach for the idempotency utility in Powertools for AWS Lambda (TypeScript) rather than maintaining this by hand. It implements the same in-progress/expiry/completed pattern on a DynamoDB table and also caches the result. The code above is what it does under the hood, and I'd want to understand that before trusting a library with payments. Idempotency keys matter most for anything with an external side effect, such as payments, emails or inventory decrements. Operations that only read, transform and write to your own database are often idempotent already and don't need the dedupe table.

The full path, end to end

EventBridge, SQS, Lambda concurrency limits, conditional writes, you've probably used each on its own already. What separates a pipeline that shrugs off a bad production week from one that pages someone at 2am is having all four in place before the first poison event shows up.

Event-Driven Apps with Lambda & EventBridge

EventBridge: route by pattern, not by function

Lambda concurrency: reserved and provisioned, deliberately

SQS as a buffer: backpressure

The DLQ: where poison events go to die quietly

Idempotent handlers: the tax for at-least-once delivery

The full path, end to end

Want to actually run this in production?

Related Tutorials

Provisioning a Secure AWS VPC with Terraform

Secure Static Website on AWS S3 & CloudFront

Multi-Region PostgreSQL Failover, Tested End to End