Skip to content

[BUG] Lambda: DeadLetterConfig is stored but never applied, and async invocations are never retried #4194

Description

@allensanborn

Service

Lambda

AWS API Action

Invoke (InvocationType: Event), CreateFunction / UpdateFunctionConfiguration (DeadLetterConfig), PutFunctionEventInvokeConfig

AWS Documentation

https://docs.aws.amazon.com/lambda/latest/dg/invocation-async.html#invocation-dlq

Expected behavior

An asynchronous invocation that fails is retried twice by default, and once the retries
are exhausted the event is sent to the function's DeadLetterConfig target.

Actual behavior

Neither half happens. DeadLetterConfig is stored and returned faithfully, so Terraform
and CloudFormation both see no drift, but nothing is ever delivered to the queue or topic.
The failing invocation is also never retried: it runs exactly once.

docs/services/lambda.md documents the event-invoke half ("Asynchronous invocations do
not yet apply its retry, event age or destination settings") but says nothing about
DeadLetterConfig, which is the older mechanism and a separate field on the function
itself. A search for DeadLetterConfig in src/main/java returns hits only under
services/eventbridge, none under services/lambda.

So the whole asynchronous failure path is inert: no retries, no event-age expiry, no DLQ.

This is closely related to #4193 (OnSuccess/OnFailure destinations), which AWS
documents as the successor to the DLQ. They are likely one fix in one place.

Reproduction

export AWS_ENDPOINT_URL=http://localhost:4566
QARN=$(aws sqs create-queue --queue-name dlq-probe >/dev/null && \
       aws sqs get-queue-attributes --queue-url "$(aws sqs get-queue-url --queue-name dlq-probe --query QueueUrl --output text)" \
       --attribute-names QueueArn --query Attributes.QueueArn --output text)

# handler is one line: exports.handler = async () => { throw new Error("always fails"); };
aws lambda create-function --function-name dlq-probe-fn \
  --runtime nodejs22.x --handler index.handler \
  --role arn:aws:iam::000000000000:role/lambda-role \
  --zip-file fileb://fn.zip --dead-letter-config "TargetArn=$QARN"

# stored and returned correctly
aws lambda get-function-configuration --function-name dlq-probe-fn --query DeadLetterConfig
# { "TargetArn": "arn:aws:sqs:us-east-1:000000000000:dlq-probe" }

aws lambda invoke --function-name dlq-probe-fn --invocation-type Event \
  --payload '{"probe":1}' --cli-binary-format raw-in-base64-out /dev/null   # 202

# the handler failed (CloudWatch Logs shows errorType Error, "always fails"), but:
aws sqs get-queue-attributes --queue-url .../dlq-probe \
  --attribute-names ApproximateNumberOfMessages
# { "ApproximateNumberOfMessages": "0" }

# and it ran once, where AWS would have attempted it three times:
aws logs filter-log-events --log-group-name /aws/lambda/dlq-probe-fn \
  --query 'events[].message' --output text | grep -c "always fails"
# 1

Environment

  • Floci version / image tag: 2.1.0 (floci/floci:latest)
  • Java SDK version (if applicable): n/a, reproduced with AWS CLI v2
  • How you're running Floci (Docker / native / mvn quarkus:dev): Docker

Additional context

Found by driving a private Terraform module against Floci. Its dead-letter-queue
example root exists to demonstrate
exactly this behaviour, and it applies and destroys cleanly while demonstrating
nothing, because every API call involved succeeds. The same root pairs the DLQ with an
aws_lambda_function_event_invoke_config setting maximum_event_age_in_seconds = 60 and
maximum_retry_attempts = 0, neither of which is applied either.

Are you willing to contribute a PR?

  • Yes
  • No

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinglambdaAWS Lambda

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions