bug: Step Functions Map iterations occasionally retry past MaxAttempts or restart the backoff when their first failures coincide

[amitkastel]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

We run a Step Functions definition in which a Map state's iterations each call a Task that always fails. The Task has this Retry:

"Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2 }]

Occasionally, one iteration ignores that policy while its sibling follows it. We have two sightings where we captured the per-iteration history, each with a different wrong field. Both came from the same execution shape: 2 items, MaxConcurrency: 10, and the Task answered by SFN_MOCK_CONFIG with a Throw for every invocation. The offsets below are measured from each iteration's TaskStateEntered.

Sighting A: the attempt count and the backoff both reset

The first iteration made 4 attempts, with intervals of 1s, 1s, 2s. Its sibling made 3 attempts at 1s and 2s, as expected.

iteration 0: 4 scheduled / 4 failed
  #51 +10ms   TaskScheduled → #53 +10ms   TaskFailed
  #57 +1015ms TaskScheduled → #59 +1015ms TaskFailed
  #60 +2018ms TaskScheduled → #62 +2019ms TaskFailed   ← only ~1s after #59
  #70 +4023ms TaskScheduled → #72 +4023ms TaskFailed
iteration 1: 3 scheduled / 3 failed
  #48 +3ms    TaskScheduled → #50 +3ms    TaskFailed
  #54 +1007ms TaskScheduled → #56 +1008ms TaskFailed
  #63 +3012ms TaskScheduled → #65 +3013ms TaskFailed

Sighting B: only the attempt count resets

The first iteration made 4 attempts, with intervals of 1s, 2s, 4s. The backoff progression is correct, but there is one retry past MaxAttempts. Its sibling again made 3 attempts at 1s and 2s.

iteration 0: 4 scheduled / 4 failed
  #48 +3ms    TaskScheduled → #52 +4ms    TaskFailed
  #54 +1006ms TaskScheduled → #58 +1007ms TaskFailed
  #63 +3010ms TaskScheduled → #67 +3011ms TaskFailed
  #70 +7016ms TaskScheduled → #72 +7016ms TaskFailed
iteration 1: 3 scheduled / 3 failed
  #50 +3ms    TaskScheduled → #53 +3ms    TaskFailed
  #56 +1006ms TaskScheduled → #59 +1007ms TaskFailed
  #60 +3008ms TaskScheduled → #62 +3008ms TaskFailed

In both sightings, every TaskScheduled has its own TaskStarted and TaskFailed, and the previousEventId chain of each iteration is clean. These are real extra attempts, not double-recorded events. The two iterations' first failures land within a few milliseconds of each other (in B their event ids are adjacent, #52 and #53).

Expected Behavior

Each Map iteration keeps its own retry state, as AWS Step Functions does. With MaxAttempts: 2 every iteration makes exactly 3 attempts, 1s then 2s apart.

How are you starting LocalStack?

Custom (please describe below)

Steps To Reproduce

LocalStack runs in Docker via testcontainers (localstack/localstack:2026.08.0 with an auth token; SERVICES=stepfunctions,dynamodb,sts,iam,…) and SFN_MOCK_CONFIG mounted. The execution shape is below. The mocked Task in our definition is an arn:aws:states:::aws-sdk:sfn:startSyncExecution state; the resource does not matter, since the Throw comes from the mock config.

{
  "StartAt": "Fanout",
  "States": {
    "Fanout": {
      "Type": "Map",
      "ItemsPath": "$.items",
      "MaxConcurrency": 10,
      "ItemProcessor": {
        "ProcessorConfig": { "Mode": "INLINE" },
        "StartAt": "AlwaysFails",
        "States": {
          "AlwaysFails": {
            "Type": "Task",
            "Resource": "arn:aws:states:::aws-sdk:sfn:describeExecution",
            "Parameters": { "ExecutionArn": "arn:aws:states:us-east-1:000000000000:execution:x:y" },
            "Retry": [{ "ErrorEquals": ["States.ALL"], "IntervalSeconds": 1, "MaxAttempts": 2, "BackoffRate": 2 }],
            "Catch": [{ "ErrorEquals": ["States.ALL"], "Next": "Caught" }],
            "End": true
          },
          "Caught": { "Type": "Pass", "End": true }
        }
      },
      "End": true
    }
  }
}

The mock config:

{
  "StateMachines": { "MapRetryRepro": { "TestCases": { "AllFail": { "AlwaysFails": "alwaysThrows" } } } },
  "MockedResponses": { "alwaysThrows": { "0-9999": { "Throw": { "Error": "Boom", "Cause": "always fails" } } } }
}

Start executions with input {"items": [0, 1]} against <stateMachineArn>#AllFail, then count each iteration's TaskScheduled events from GetExecutionHistory.

This is intermittent and load-dependent. We have seen it 6 times in CI over three days, where GitHub Actions ubuntu-latest (4 vCPU) runs 4 test files in parallel against one container. It has never happened on an idle laptop. A standalone script on an idle Apple-silicon machine did not reproduce it: 75 executions (up to 20 items each, 60 of them started at once, about 800 iterations in total) were all correct.

Environment

- OS: GitHub Actions ubuntu-latest runner (4 vCPU); also macOS 15 (Docker Desktop) locally, where it does not reproduce
- LocalStack:
  LocalStack version: 2026.08.0
  LocalStack Docker image sha: localstack/localstack:2026.08.0
  LocalStack build date: 2026-08-26
  LocalStack build git hash: n/a

Anything else?

We have a candidate mechanism from reading the source. It is not confirmed: we cannot trigger it on demand.

EvalComponent.heap_key (localstack-core/localstack/services/stepfunctions/asl/component/eval_component.py) is lazily initialised with an unguarded check-then-set:

@property
def heap_key(self) -> str:
    if self.__heap_key is None:
        self.__heap_key = long_uid()
    return self.__heap_key

The retrier's counters live in the frame's env.heap under that key. MaxAttemptsDecl._eval_body reads f"MaxAttemptsDecl-{self.heap_key}-attempt_number" and stores it with a second heap_key call; BackoffRateDecl does the same for next_multiplier. Every Map iteration evaluates the same component instances: JobPool hands every Job the same job_program, and each iteration gets its own frame whose heap starts empty.

Now suppose two iterations evaluate a retrier's MaxAttemptsDecl for the first time concurrently. long_uid() → uuid4() → os.urandom() can release the GIL, so both threads can see None:

1. Iteration 0 generates uid1, reads its counter with key uid1 (default -1 → 0), and stores it under uid1.
2. Iteration 1 then overwrites __heap_key with uid2.
3. On iteration 0's next failure it reads under uid2, finds nothing, and the counter restarts at -1, which buys one extra retry.

That accounts for both sightings exactly:

• B: only the MaxAttemptsDecl key was lost. One extra attempt; the backoff stays 1s → 2s → 4s.
• A: both the MaxAttemptsDecl and BackoffRateDecl keys were lost. One extra attempt, and the multiplier restarts: 1s → 1s → 2s.

It also fits the timing: only the iterations' first failures collide, and the effect shows up on the next failure.

The image we run is the licensed one, whose Step Functions sources are encrypted, so we could not check its copy directly. The community source on main has this pattern, and so does the moto-vendored copy of the interpreter inside that image (moto/stepfunctions/parser/asl/component/eval_component.py). Assigning heap_key eagerly in __init__, or guarding the lazy init with a lock, would close the window.

Environment.next_local_mock_invocation_number is also shared across frames and incremented without a lock. We don't think it causes this, because it only picks which mocked response is returned and every response here is the same Throw. It might be worth a look, though.

On our side we now tolerate one emulator-added attempt per execution in the affected test and pin the exact Retry policy in a unit test of the definition.