7 Webhook Integration Mistakes That Cause Production Outages
Review webhook receipt, signature checks, duplicate handling, processing queues, ordering, and recovery for failed or missed deliveries.
A webhook handler must authenticate deliveries, record them durably, and handle duplicates or processing failures. Review these seven areas before relying on webhook events for orders, payments, or other operational state.
Mistake 1: Processing Events Synchronously in the Request Handler
Lengthy processing can exceed the sender’s response deadline. GitHub, for example, expects a response within 10 seconds and does not automatically redeliver failed deliveries. Read each provider’s timeout and recovery policy. GitHub webhook practices.
The fix: return 200 immediately after validating and storing the event, then hand it to a background job queue for processing.
Mistake 2: No Idempotency Handling
A provider retry or manual redelivery can repeat an event that your system already received. Make repeated delivery safe even if automatic retries are not enabled.
Without idempotency, a repeated event can repeat your handler’s side effects, such as creating a second fulfillment record.
The fix: use the provider’s documented event identifier, scoped to the relevant account and endpoint. Record receipt and processing state durably. A duplicate delivery should not repeat completed work, but pending or failed work still needs recovery.
Mistake 3: Skipping Signature Verification
If you are not verifying the HMAC signature on inbound webhooks, your endpoint accepts any HTTP POST from anyone who knows its URL. A fabricated payment.succeeded event is indistinguishable from a real one without signature verification.
The fix: implement the provider’s documented authentication scheme, preferably with its verification library. For Stripe signatures, preserve the raw request body and validate the signature and timestamp before processing. Stripe webhook guidance.
Mistake 4: Using the Wrong HTTP Status Codes
Returning 500 when you should return 200, or 200 when you should return 4xx, teaches the sender the wrong thing about what to retry.
A 500 tells the sender your endpoint is having a transient problem — retry later. A parsing bug may cause repeated failures during the provider’s retry window; retries alone will not fix it.
A 200 for a duplicate event that you silently discarded is correct. A 200 for an event you actually failed to process is wrong — it tells the sender you succeeded when you did not.
The fix: return 200/202 only when you have successfully received and queued the event. Return 5xx for genuine transient failures. Return 4xx sparingly and intentionally, knowing the sender may not retry.
Mistake 5: Trusting Webhook Order
Webhook providers do not guarantee delivery order. Under retry conditions, it is entirely possible to receive subscription.updated before subscription.created. Check the provider’s ordering guarantees before relying on event sequence.
The fix: design handlers to be order-tolerant. Use the event's embedded timestamp or sequence number (if the provider includes one) to decide whether to apply an update. Use upserts rather than inserts where possible.
Mistake 6: No Dead-Letter Strategy
Automatic retry limits and manual replay options vary by provider. An event your endpoint never received will not be present in your internal dead-letter store. Plan provider redelivery or reconciliation for missed deliveries as well as retries for received events.
The fix: build a dead-letter store. Any event that exhausts your internal retry attempts gets written to a dead_letter_events table with the full payload. Build a replay mechanism so you can reprocess events after you fix the underlying bug.
Mistake 7: Hardcoding Secrets in Code
Webhook signing secrets committed to a repository expose you to two risks: anyone with repo access can forge events, and rotating the secret requires a code change and a deploy.
The fix: load signing secrets from environment variables or a secrets manager. Verify that your CI pipeline does not log environment variables. Ensure the secret rotation path is tested and fast.