Don't take a webhook at its word: HMAC signatures, replay protection, duplicates and a queue
Your provider sends the same event three times, sometimes out of order, and sometimes it wasn't the provider at all. A step-by-step guide to receiving webhooks in Django and Celery safely and with exactly-once effects.
Picture this: a payment provider sends a payment.succeeded event to your /webhooks/acme/ endpoint. The handler updates the order, renders a PDF receipt and emails it. That takes a few seconds, and the provider's timeout is shorter. No response means "not delivered", so a minute later the provider sends the very same event again. The customer gets two receipts, the order history gets two entries.
Meanwhile someone found the endpoint URL in a log and manually posted {"type": "payment.succeeded", "order_id": 777}. If the signature check is weak, that order is now "paid". And a third problem: subscription.canceled and subscription.updated were sent almost simultaneously, the second arrived first, and a canceled subscription is active again.
These three problems — forged requests, repeated delivery and out-of-order arrival — come from the nature of webhooks. You can't dismiss them as rare: nearly every provider openly promises at-least-once delivery, not exactly-once. In this post we'll build the receiving side end to end, and at the end look at what you owe your consumers if you send webhooks yourself.
Why the naive handler breaks
The first version in many projects looks something like this:
# webhooks/views.py -- DON'T do this
import hashlib
import json
from django.conf import settings
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
from orders.models import Order
from orders.emails import send_receipt_email
@csrf_exempt
def acme_webhook(request):
data = json.loads(request.body)
expected = hashlib.sha256(
(settings.ACME_SECRET + json.dumps(data)).encode()
).hexdigest()
if expected != request.headers.get("X-Webhook-Signature"):
return HttpResponse(status=401)
order = Order.objects.get(pk=data["data"]["order_id"])
order.status = data["data"]["status"]
order.save()
send_receipt_email(order) # SMTP, PDF, a few seconds
return HttpResponse("ok")
There isn't one bug here, there are six:
sha256(secret + message)is not HMAC. For Merkle–Damgård hashes like SHA-256 this construction is open to length extension attacks. That's exactly why thehmacmodule exists.json.dumps(json.loads(body))doesn't give back the original bytes: whitespace, key order and unicode escaping differ. The provider signed the raw bytes, so you must verify the raw bytes.- Comparing with
!=stops at the first differing character. In theory that leaks information through timing;hmac.compare_digestruns in constant time. - No timestamp. A genuine request captured once (with a valid signature!) can be replayed at any time — a replay attack.
- No duplicate check on the event id, so every provider retry runs everything again.
- Heavy work (email, PDF) inside the request. The response is slow, it times out, a retry arrives — so bug 5 gets triggered precisely by this one.
Step 1: signature and timestamp
A solid scheme looks like this: you share a secret with the provider. The provider signs the string timestamp.body with HMAC-SHA256 and sends two headers, X-Webhook-Timestamp and X-Webhook-Signature. You repeat the computation and compare in constant time. Because the timestamp is inside the signature it can't be altered, and an old request falls outside the window and gets rejected.
# webhooks/signing.py
import hashlib
import hmac
import time
TOLERANCE_SECONDS = 300
class SignatureError(Exception):
pass
def compute_signature(secret: bytes, timestamp: str, body: bytes) -> str:
signed_payload = timestamp.encode() + b"." + body
return hmac.new(secret, signed_payload, hashlib.sha256).hexdigest()
def verify_signature(
secrets: list[bytes],
timestamp: str | None,
signature: str | None,
body: bytes,
) -> None:
if not timestamp or not signature:
raise SignatureError("missing headers")
try:
ts = int(timestamp)
except ValueError:
raise SignatureError("bad timestamp") from None
if abs(time.time() - ts) > TOLERANCE_SECONDS:
raise SignatureError("timestamp outside tolerance")
# Several secrets = rotation without downtime (old + new).
for secret in secrets:
expected = compute_signature(secret, timestamp, body)
if hmac.compare_digest(expected, signature):
return
raise SignatureError("signature mismatch")
The list of secrets is for rotation. Add the new secret, switch it in the provider's dashboard, then remove the old one after a while. No downtime window needed.
# settings.py
import os
ACME_WEBHOOK_SECRETS = [
s.encode() for s in os.environ["ACME_WEBHOOK_SECRETS"].split(",") if s
]
Header names and signature format vary by provider: some use hex, some base64, some pack several values into one header like t=...,v1=.... The mechanism is almost always the same: HMAC, raw body, timestamp. If the provider's official SDK ships a verification function, use it.
A signature says "the holder of the secret sent this request". It does not say "this request is arriving for the first time". That's why the timestamp window and dedup by event id are separate concerns, and you need both.
A five-minute window doesn't fully stop replay: a request captured inside the window can still be resent. The next step — the duplicate filter — closes that gap.
Step 2: fast 200, work goes to the queue
The receiving endpoint has a single job: verify, store, say "got it". Business logic lives in a Celery worker. First we write the event to the database:
# webhooks/models.py
from django.db import models
class WebhookEvent(models.Model):
class Status(models.TextChoices):
PENDING = "pending"
PROCESSED = "processed"
FAILED = "failed"
provider = models.CharField(max_length=32)
event_id = models.CharField(max_length=128)
event_type = models.CharField(max_length=64)
payload = models.JSONField()
status = models.CharField(
max_length=16, choices=Status, default=Status.PENDING
)
received_at = models.DateTimeField(auto_now_add=True)
processed_at = models.DateTimeField(null=True, blank=True)
class Meta:
constraints = [
models.UniqueConstraint(
fields=["provider", "event_id"], name="uniq_webhook_event"
),
]
indexes = [models.Index(fields=["status", "received_at"])]
UniqueConstraint(provider, event_id) is the foundation of dedup. The check lives in PostgreSQL, not in your code: even if two concurrent requests arrive with the same event, the unique index lets only one INSERT through.
# webhooks/views.py
import json
from django.conf import settings
from django.db import transaction
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST
from .models import WebhookEvent
from .signing import SignatureError, verify_signature
from .tasks import process_webhook_event
@csrf_exempt
@require_POST
def acme_webhook(request):
body = request.body # raw bytes, exactly what was signed
try:
verify_signature(
settings.ACME_WEBHOOK_SECRETS,
request.headers.get("X-Webhook-Timestamp"),
request.headers.get("X-Webhook-Signature"),
body,
)
except SignatureError:
return HttpResponse(status=400)
try:
payload = json.loads(body)
event_id = str(payload["id"])
event_type = str(payload["type"])
except (ValueError, KeyError, TypeError):
return HttpResponse(status=400)
with transaction.atomic():
event, created = WebhookEvent.objects.get_or_create(
provider="acme",
event_id=event_id,
defaults={"event_type": event_type, "payload": payload},
)
if created:
transaction.on_commit(
lambda: process_webhook_event.delay(event.pk)
)
# Duplicate or new: the answer is the same, 200.
return HttpResponse(status=200)
A few details matter here:
get_or_createbacked by a unique constraint is race-safe: the second request hitsIntegrityErroron INSERT, Django catches it and returns the existing row (created=False).- The task is enqueued via
transaction.on_commit. Call.delay()directly and the worker may start before the commit and not find the row. - Duplicates get a 200 too. Return 409 or 500 and the provider will treat it as a failed delivery and keep retrying.
- On a bad signature, don't explain why in the response — 400 is enough. Log the reason on your side.
Step 3: the worker must be idempotent
Dedup at the endpoint only solves "the same event isn't stored twice". Celery itself can also run a task twice (acks_late, a dying worker, visibility timeout). So the worker locks the row and checks the status too:
# webhooks/tasks.py
from datetime import timedelta
from celery import shared_task
from django.db import transaction
from django.utils import timezone
from .handlers import handle_event
from .models import WebhookEvent
@shared_task(acks_late=True)
def process_webhook_event(event_pk: int) -> None:
with transaction.atomic():
event = (
WebhookEvent.objects.select_for_update()
.filter(pk=event_pk)
.first()
)
if event is None or event.status == WebhookEvent.Status.PROCESSED:
return
handle_event(event) # DB work only, no HTTP calls under the lock
event.status = WebhookEvent.Status.PROCESSED
event.processed_at = timezone.now()
event.save(update_fields=["status", "processed_at"])
@shared_task
def requeue_stale_webhook_events() -> None:
# Safety net: broker was down when on_commit fired, worker died, etc.
cutoff = timezone.now() - timedelta(minutes=10)
stale = WebhookEvent.objects.filter(
status=WebhookEvent.Status.PENDING, received_at__lt=cutoff
).values_list("pk", flat=True)[:500]
for pk in stale:
process_webhook_event.delay(pk)
select_for_update() makes a second concurrent worker wait until the first commits; then it sees PROCESSED and returns. One condition: no external HTTP calls inside handle_event — waiting on the network while holding a lock ties up the connection and blocks other queries. If you need an email or an external API, push that into a separate task, again via on_commit.
requeue_stale_webhook_events is the safety net. If the .delay() inside on_commit lands at a moment when the broker is down, the event stays in the database as pending and nobody picks it up. Run this task from Celery beat every few minutes and it re-enqueues those orphans. Since the worker is idempotent, an extra enqueue does no harm.
Step 4: events arrive out of order
Providers send events from many workers, with retries, in parallel. A canceled created after an updated can reach you first. If your handler treats the last event to arrive as "the latest state", the state goes backwards.
There are two reliable approaches:
- Conditional update by version or time. Store the time of the last applied event on the object and only apply newer ones.
- Thin event + re-fetch. Treat the event as a signal: "
sub_42changed". Then fetch the object's current state from the provider's API. The ordering problem disappears, at the cost of one API call per event.
The first approach in code:
# webhooks/handlers.py
from datetime import datetime, timezone as dt_timezone
from django.db.models import Q
from billing.models import Subscription
def handle_event(event) -> None:
if event.event_type != "subscription.updated":
return # unknown types: store, acknowledge, ignore
data = event.payload["data"]
occurred_at = datetime.fromtimestamp(
event.payload["created"], tz=dt_timezone.utc
)
# Conditional UPDATE: an older event can never overwrite a newer state.
Subscription.objects.filter(
external_id=data["subscription_id"],
).filter(
Q(last_event_at__isnull=True) | Q(last_event_at__lt=occurred_at)
).update(
status=data["status"],
last_event_at=occurred_at,
)
This becomes a single atomic SQL statement — no race between read and write:
UPDATE billing_subscription
SET status = 'canceled', last_event_at = '2026-11-20 10:00:05+00'
WHERE external_id = 'sub_42'
AND (last_event_at IS NULL OR last_event_at < '2026-11-20 10:00:05+00');
Careful: if two events were created in the same second, < drops the second one. If the provider includes a sequence or an object version, use that instead of time. If there's no such field and the state is critical (payments, subscriptions), the second approach is safer.
The sending side: retries, backoff, jitter
If you send webhooks to your own customers, the roles flip: now you promise at-least-once and you make it possible for receivers to do everything above. The core rules:
- The event id is identical across all attempts, while the timestamp and signature are recomputed on every attempt (otherwise a retry an hour later won't pass the receiver's five-minute window).
- Retries use exponential backoff with jitter, so hundreds of customer endpoints that failed at the same moment don't all get hit again in the same second.
- 2xx is success. 5xx, 408, 429 and network errors are retried. Other 4xx are not retried, but logged.
- Always set a timeout.
requestshas no default timeout and can wait forever. - Create the delivery task in the same transaction as the business change (outbox or
on_commit), or events get lost.
# outgoing/tasks.py
import json
import time
import requests
from celery import shared_task
from django.utils import timezone
from webhooks.signing import compute_signature
from .models import WebhookDelivery
class RetryableDeliveryError(Exception):
pass
@shared_task(
autoretry_for=(
RetryableDeliveryError,
requests.ConnectionError,
requests.Timeout,
),
retry_backoff=5, # 5s, 10s, 20s, 40s ...
retry_backoff_max=3600, # never wait more than an hour
retry_jitter=True, # spread retries, avoid a thundering herd
max_retries=12,
acks_late=True,
)
def deliver_webhook(delivery_id: int) -> None:
delivery = WebhookDelivery.objects.select_related("endpoint").get(
pk=delivery_id
)
if delivery.delivered_at is not None:
return
body = json.dumps(
delivery.payload, separators=(",", ":"), ensure_ascii=False
).encode()
ts = str(int(time.time())) # fresh timestamp on every attempt
signature = compute_signature(
delivery.endpoint.secret.encode(), ts, body
)
resp = requests.post(
delivery.endpoint.url,
data=body,
headers={
"Content-Type": "application/json",
"X-Webhook-Id": str(delivery.event_id), # same on every retry
"X-Webhook-Timestamp": ts,
"X-Webhook-Signature": signature,
},
timeout=(3, 10),
allow_redirects=False,
)
delivery.last_status = resp.status_code
if 200 <= resp.status_code < 300:
delivery.delivered_at = timezone.now()
delivery.save(update_fields=["last_status", "delivered_at"])
return
delivery.save(update_fields=["last_status"])
if resp.status_code in (408, 429) or resp.status_code >= 500:
raise RetryableDeliveryError(f"HTTP {resp.status_code}")
# Other 4xx: the receiver rejected the request; retrying won't help.
In Celery, retry_backoff=5 produces delays of 5, 10, 20, 40... seconds, retry_backoff_max caps them, and retry_jitter=True picks each delay randomly between 0 and the computed value. Once max_retries is exhausted Celery re-raises the last exception — that's a good moment to mark the delivery failed, show it in the customer's dashboard and offer a manual "redeliver" button. If customers type in their own URLs, allow_redirects=False plus blocking internal IP ranges protects you from SSRF.
Testing locally
You can exercise the endpoint without the provider's sandbox — compute the signature with openssl:
SECRET='whsec_test'
BODY='{"id":"evt_1","type":"ping","created":1763632800,"data":{}}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $NF}')
curl -sS -i -X POST http://localhost:8000/webhooks/acme/ \
-H 'Content-Type: application/json' \
-H "X-Webhook-Timestamp: $TS" \
-H "X-Webhook-Signature: $SIG" \
--data-binary "$BODY"
Send the same command twice: the second must also return 200, but the database should have one row and one executed task. Replace TS with a value from ten minutes ago and you should get a 400.
Production pitfalls
| Problem | Symptom | Fix |
|---|---|---|
| Body altered in transit | Signature never matches | Verify only request.body, make sure no middleware rewrites it |
| Server clock drift | Every event is "too old" | NTP, don't make the window too narrow |
| Large payload | RequestDataTooBig, 400 |
DATA_UPLOAD_MAX_MEMORY_SIZE applies to request.body too |
| Broker down | Event stuck in pending |
Periodic requeue task |
| Table grows forever | Slow queries | Purge old processed rows or partition |
A few more:
- The dedup key is the provider's event id, not a payload hash. Some providers add an attempt counter to the payload, so the hash changes.
- Unknown event types still get a 200 and are stored. When the provider adds a new type, your endpoint shouldn't turn into a retry storm.
- Never log the secret, but do log signature failures — they're the first sign of an attack or a broken config.
- An IP allowlist doesn't replace the signature; at most it's an extra layer.
csrf_exemptgoes on the webhook view only. Don't disable CSRF for a whole URL prefix.
Checklist
- HMAC-SHA256, raw
request.body,hmac.compare_digest. - Timestamp inside the signature, ±5 minute window.
- Accept multiple secrets for rotation.
UniqueConstraint(provider, event_id)+get_or_create.- Store, enqueue with
on_commit, return 200 immediately (duplicates included). - In the worker:
select_for_update+ status check, no HTTP under the lock. - Periodic requeue for events stuck in
pending. - For ordering: conditional
UPDATEor re-fetch from the API. - If you send: stable event id, fresh signature, backoff + jitter, timeouts, manual redelivery.
Takeaway
A webhook isn't "a command that arrives once"; it's "a message that arrives at least once, in any order, and might be forged". The signature and timestamp answer who sent it, the unique constraint answers how many times it was processed, and the conditional update answers which state is the latest. Let the endpoint only accept and let the queue do the work — then retries stop being a threat and become mere noise.
// comments
Loading comments...