Fix Webhook Endpoint Timeout and Retry Loops
A webhook provider retries when acknowledgement is late or uncertain. Authenticate and durably accept the event quickly, then process it idempotently outside the request deadline.
Learn the provider deadline
Read the documented timeout, accepted success statuses, retry schedule, signature rules, event identifier, and replay behavior. Capture delivery ID, event type, request timestamp, response status, handler duration, and a correlation ID. Do not log signatures, secrets, authorization headers, or sensitive payload values.
Measure each handler phase
Separate connection and body read, raw-body signature verification, parsing, deduplication, durable storage, queue publication, and response write. Then measure asynchronous business work separately. A database lock, cold start, DNS lookup, remote API, email send, or large log operation can consume the entire acknowledgement budget.
Authenticate before accepting
Verify the signature over the exact raw bytes using the provider algorithm, timestamp tolerance, and constant-time comparison. Support secret rotation as documented. Reject invalid requests without creating a deduplication record. Bound body size and parsing work so an untrusted request cannot hold the endpoint indefinitely.
Persist before returning success
Return the provider's success status only after the authenticated event is durably recorded or atomically placed on a durable queue. Responding before durability can lose an event if the process crashes. Performing every business side effect before responding increases timeout and duplicate risk. The correct boundary is durable acceptance followed by asynchronous processing.
Make processing idempotent
Use the provider event or delivery ID with a database unique constraint. Track processing state and give downstream side effects their own idempotency keys. A timeout does not prove the first attempt failed; the provider may retry after local completion. Handle concurrent duplicates and crashes between the side effect and completion marker.
Remove request-path dependencies
Avoid fetching the full source object, calling another API, sending notifications, or running expensive transformations before acknowledgement unless required to authenticate or safely accept the event. Set strict timeouts on unavoidable storage and queue operations. Monitor pool exhaustion, queue latency, event age, and terminal processing failures.
Test the deadline boundary
Replay a signed fixture with delayed storage, concurrent duplicates, queue failure, process interruption, and a lost response. Confirm safe retry, one business effect, and prompt acknowledgement after recovery. Inspect non-secret delivery headers with the HTTP Header Analyzer.
Frequently Asked Questions
When should success be returned?
After authentication and durable acceptance.
Why retry after work completed?
The acknowledgement can be late or lost.
Should all logic run first?
No. Process expensive work asynchronously.