Webhook Retry Strategy: Backoff, Dead-Letter Queues, and Recovery

Webhook receivers need a plan for delivery failures and for events that arrive but fail during processing. Separate the provider’s retry policy from your internal retry queue, and keep a way to inspect and replay failed work.

What Webhook Senders Do When Your Endpoint Fails

Every major webhook provider has its own retry policy. Knowing what yours does changes how you design your receiver.

• Stripe: Retries up to 3 days in live mode, following an exponential schedule starting at a few minutes and growing to several hours. It stops retrying on 4xx responses (treating them as intentional rejections) but continues on 5xx and timeouts.
• Shopify: Retries 8 times over approximately 4 hours.
• GitHub: Retries 3 times at 5-minute intervals.
• Svix (used by many SaaS platforms): Approximately 8 retries over roughly 24 hours with exponential backoff.

Check the provider’s documented response-code and timeout behavior. Do not assume every sender uses the same retry rules.

Exponential Backoff Explained

Exponential backoff means the delay between retry attempts grows exponentially rather than staying fixed. A basic schedule might look like:

Attempt 1: immediate
Attempt 2: 1 minute later
Attempt 3: 4 minutes later
Attempt 4: 16 minutes later
Attempt 5: 64 minutes later

The motivation is practical: if your endpoint is down because your database is overloaded, hammering it with immediate retries makes the overload worse. Spreading retries out gives systems time to recover.

Jitter — adding a small random offset to each delay — prevents thundering herd problems when many events fail simultaneously and would otherwise all retry at the same moment.

If you are building a system that sends webhooks (not just receives them), this is the schedule to implement on your sender side.

What Your Receiver Should Do

When your endpoint cannot process an event, your job is to communicate that clearly through HTTP status codes and your own internal retry infrastructure.

Return 200/202 fast, process asynchronously. If your business logic might take more than a second or two, do not do it inline. Accept the event, write it to a queue or database, return 200, and process in the background. This prevents false timeouts from triggering unnecessary retries.

Return 5xx only when you cannot persist or handle the event right now. A 500 or 503 signals to the sender that you want a retry. Use this when you are experiencing a transient failure — a database connection error, a dependency timeout.

Return 4xx when the event is malformed or the request is bad. A 400 or 422 tells the sender not to retry. Use this carefully; most genuine delivery failures are transient and should not be permanently abandoned.

Building a Dead-Letter Queue on the Receiver Side

A sender can exhaust its delivery attempts. For events you have received and persisted, a failed-job queue provides a separate recovery path.

A simple dead-letter pattern:

1. When a background job fails after a set number of internal retries (say, 5 attempts), move the event payload to a dead_letter_events table instead of dropping it.
2. Record the failure reason, the timestamp, and the full raw payload.
3. Build an admin interface or script that lets you inspect and replay dead-letter events once the underlying problem is fixed.

The table schema can be minimal:

CREATE TABLE dead_letter_events (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  source      TEXT NOT NULL,         -- e.g. 'stripe', 'github'
  event_id    TEXT NOT NULL,
  payload     JSONB NOT NULL,
  failure_reason TEXT,
  failed_at   TIMESTAMPTZ DEFAULT NOW(),
  replayed_at TIMESTAMPTZ
);

When you fix a bug that was causing failures, you can replay the dead-letter queue selectively without losing business-critical events.

Receiver-Side Exponential Backoff

Your own background job queue should also implement exponential backoff for retries, separate from whatever the sender does. If processing fails because a third-party dependency was down, you want your own retry schedule — not a 24-hour gap waiting for the sender to try again.

Popular job queues handle this out of the box:

• Sidekiq (Ruby): sidekiq_options retry: 5 with exponential delays.
• BullMQ (Node.js): attempts and backoff options on each job.
• Celery (Python): autoretry_for with countdown and max_retries.

Set your retry ceiling low enough that the job fails quickly if a dependency is truly broken, and the dead-letter queue catches it for manual inspection.

Practical Checklist

• Know your provider's retry window and stop-conditions (4xx vs 5xx behaviour).
• Return 200/202 immediately; offload processing to a background queue.
• Use 5xx for transient failures, 4xx only for truly unrecoverable bad requests.
• Implement exponential backoff with jitter in your own job queue.
• Build a dead-letter table and a replay mechanism before you go to production.
• Alert on dead-letter growth; do not let it silently accumulate.

For help reviewing webhook delivery and replay behavior, talk to LinkIT.