Handle Duplicate Webhook Events Safely
Duplicate webhook delivery is normal in an at-least-once system. Verify authenticity, record the event atomically, and make every side effect safe to repeat.
Expect retries by design
A provider may retry when your endpoint times out, returns a non-success status, closes the connection early, or when its acknowledgement is lost in transit. The first request may have completed even though the provider did not observe the response. Read the provider's retry schedule, success codes, event identifiers, retention period, and manual replay behavior.
Verify before deduplicating
Authenticate the webhook using the raw request bytes and the provider's documented signature process before trusting an event ID. Enforce timestamp or replay limits where specified, compare signatures in constant time, and support secret rotation. Do not log signatures, secrets, or sensitive bodies. A forged request must not reserve an identifier that blocks the legitimate event.
Claim the event atomically
Prefer a stable provider event or delivery ID, scoped by provider or endpoint. Insert it into storage with a unique constraint in the same durable workflow that schedules processing. A check-then-insert sequence without atomicity lets two workers process the same event concurrently. Store a processing state and enough non-sensitive metadata to investigate failures and retry safely.
Make side effects idempotent
Use source object IDs and event versions when updating local state. Give outgoing payment, email, provisioning, or job operations their own idempotency keys. A processed-event table alone is insufficient if a crash occurs after a side effect but before the event is marked complete. Use a transaction, outbox, or downstream idempotency mechanism appropriate to the side effect.
Respond quickly, process separately
After verification and durable acceptance, return the documented success response promptly and process expensive work asynchronously. Slow database queries or remote API calls in the webhook request path invite retries. Bound queue attempts, preserve the original event identifier, and send terminal failures to an observable recovery path instead of acknowledging data that was never stored.
Treat ordering as a separate problem
Deduplication does not guarantee order. A later update may arrive before an earlier event or a replay may contain old state. Use a provider sequence, object version, event creation time with defined tie-breaking, or fetch the current source-of-truth object. Do not derive a deduplication key from the entire body or a short timestamp window; distinct legitimate events can look identical.
Test failure boundaries
Replay one signed event concurrently and after each simulated crash point. Confirm one business effect, safe recovery, quick acknowledgement, and useful redacted logs. Inspect payload structure locally with the JSON Tree Viewer after following the provider's data-handling rules.
Frequently Asked Questions
Why are events delivered twice?
At-least-once systems retry when acknowledgement is uncertain.
What is the best dedupe key?
The provider's stable event or delivery ID, properly scoped.
Does dedupe guarantee order?
No. Ordering needs separate version or reconciliation logic.