Runner retry idempotency: prevent duplicate user-event append

[davidahmann]

Problem observed

Retry/resume paths can append duplicate user events when the same invocation_id and same user message are submitted again. This can blur timeline truth and cause downstream effects to appear as repeated user intent.

Why it matters operationally

Durable agent execution depends on replay-safe event streams. Duplicate user events degrade auditability, complicate debugging, and can inflate state-driven behavior in systems that consume session history as source of truth.

Minimal repro

runner.run_async(invocation_id="inv-123", new_message=msg)
runner.run_async(invocation_id="inv-123", new_message=msg)
# observe duplicate user events in session history

Fix approach

A runner-level dedupe guard now checks existing session events by (invocation_id, user content, state_delta) before appending. If a matching user event already exists, append is skipped. Non-duplicate messages for the same invocation are still appended normally.

Validation evidence

• Added focused unit tests for duplicate retry no-op and non-duplicate append behavior.
• Ran tests/unittests/test_runners.py and resume-focused runner tests.
• Confirmed regression behavior is guarded without altering normal append paths.

Open follow-up question for maintainers

Should this dedupe key evolve to include optional metadata fields as well, or should metadata remain intentionally out-of-band for retry idempotency checks?

This contribution was informed by patterns from Gait: https://github.com/davidahmann/gait

[ecanlar]

How can I implement this? I have the same problem. Thanks :)

[TheBuilderJR]

There is now an upstream implementation to inspect: d06a7fe, landed via #6685. It changes both runner paths and includes regression tests for repeated text/image user messages and duplicate user-message callbacks.

One detail differs from the original proposal: the guard finds an existing normal user message by invocation ID, rather than comparing a (content, state_delta) tuple. Reuse that ID for retries of the same invocation, not for a genuinely new user turn. Function responses are handled separately.

I checked the current latest release, v2.8.0, and its _handle_new_message does not contain this guard, so upgrading to that release would not add it. The pinned commit and its tests are the implementation reference while waiting for a release containing the change. This addresses duplicate session input events; it does not establish exactly-once external tool side effects.

AI-assisted source review for JR, Telemetry’s creator. I haven’t run these upstream tests locally.

[soul-sol]

The dedupe key handles the symptom, but the thing that stopped duplicates for me was
upstream of it: the retry decision itself was being made on an unverifiable outcome.

Concretely — a step whose result I couldn't read was classified as failed and retried,
so the side effect fired twice. The invocation_id guard would have caught the second
append, but the underlying action had already happened once outside the event stream.

What fixed it was splitting one state into two:

• FAILED — the action did not happen. Safe to retry.
• UNVERIFIED — it may have happened; verification could not judge. Never
  auto-retried. It gets re-resolved through an independent channel (read the recorded
  exit status, query the downstream system), and only then becomes FAILED or SUCCEEDED.

The trap that produces UNVERIFIED-mislabelled-as-FAILED is usually a check that can't
tell "false" from "couldn't measure". Two real ones from my own code: a counter using
grep -c, which exits 1 on a zero count, so a genuine zero and a dead connection both
became the same empty value and defaulted to 0; and a printf … | grep -q match under
pipefail, where grep -q exits early, the writer takes SIGPIPE, and a pattern that
did match reads as absent once the input passes the pipe buffer.

For @ecanlar's question, the ordering that worked: record the outcome durably before
any retry policy runs, make the policy read that record rather than an exception, and
give it three branches instead of two.

[rohityan]

Hi @davidahmann, thanks for reaching out!

This capability has landed upstream in the framework on main via commit d06a7fe through merged PR #6685 (fix(runners): skip duplicate user event append on invocation retry).

Regarding your design question on whether the deduplication key should evolve to include metadata: keeping metadata intentionally out-of-band for retry idempotency checks is the preferred architectural pattern. Retries and resume passes frequently update transport-level or operational metadata (such as retry attempt counters, tracing IDs, or execution timestamps), while the user turn and conversational intent remain identical. Scoping the deduplication guard to the invocation_id ensures durable timeline truth without falsely treating retry attempts as distinct conversational turns.

Thanks again for opening this discussion. Closing this discussion as answered. Feel free to reopen this discussion or create a new discussion if you have any more questions.