Webhook'ga ko'r-ko'rona ishonmang: HMAC imzo, replay himoyasi, dublikatlar va navbat
Provayder bitta eventni uch marta yuboradi, ba'zan noto'g'ri tartibda, ba'zan esa uni umuman provayder yubormagan bo'ladi. Django va Celery'da webhook'ni xavfsiz va bir martadan qabul qilishni qadamma-qadam ko'ramiz.
Tasavvur qiling: to'lov provayderi sizning /webhooks/acme/ endpoint'ingizga payment.succeeded event'ini yuboradi. Handler buyurtmani yangilaydi, keyin chek PDF'ini yasab email jo'natadi. Bu bir necha soniya oladi, provayderning timeout'i esa undan qisqa. Provayder javob olmadi, demak "yetkazilmadi" deb hisoblaydi va bir daqiqadan keyin xuddi shu event'ni qayta yuboradi. Mijozga ikkita chek boradi, buyurtma tarixida ikkita yozuv paydo bo'ladi.
Shu payt boshqa joyda kimdir endpoint URL'ini loglardan topib oldi va qo'lda {"type": "payment.succeeded", "order_id": 777} yubordi. Imzo tekshiruvi zaif bo'lsa, buyurtma "to'langan" bo'lib qoladi. Uchinchi muammo: obuna uchun subscription.canceled va subscription.updated deyarli bir vaqtda yuborildi, lekin ikkinchisi birinchi yetib keldi va bekor qilingan obuna yana "active" bo'lib qoldi.
Bu uchta muammo — soxta so'rov, takroriy yetkazish va tartibsiz kelish — webhook'ning tabiatidan kelib chiqadi. Ularni "kam uchraydi" deb chetlab o'tib bo'lmaydi, chunki deyarli barcha provayderlar ochiqchasiga "at-least-once" (kamida bir marta) yetkazishni va'da qiladi, "exactly-once" emas. Bu maqolada qabul qiluvchi tomonni to'liq quramiz, oxirida esa o'zingiz webhook yuborsangiz nima qilish kerakligini ko'ramiz.
Sodda handler nega buziladi
Ko'p loyihalarda birinchi versiya shunga o'xshaydi:
# webhooks/views.py -- DON'T do this
import hashlib
import json
from django.conf import settings
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
from orders.models import Order
from orders.emails import send_receipt_email
@csrf_exempt
def acme_webhook(request):
data = json.loads(request.body)
expected = hashlib.sha256(
(settings.ACME_SECRET + json.dumps(data)).encode()
).hexdigest()
if expected != request.headers.get("X-Webhook-Signature"):
return HttpResponse(status=401)
order = Order.objects.get(pk=data["data"]["order_id"])
order.status = data["data"]["status"]
order.save()
send_receipt_email(order) # SMTP, PDF, a few seconds
return HttpResponse("ok")
Bu yerda bitta emas, oltita xato bor:
sha256(secret + message)— bu HMAC emas. SHA-256 kabi Merkle–Damgård hash'lar uchun bunday konstruksiya length extension hujumiga ochiq. Kriptografik imzo uchunhmacmoduli bor, aynan shu sababli.json.dumps(json.loads(body))asl baytlarni qaytarmaydi: bo'shliqlar, kalitlar tartibi, unicode escape farq qiladi. Provayder xom baytlarni imzolagan, siz ham xom baytlarni tekshirishingiz kerak.!=bilan solishtirish birinchi farq qilgan belgida to'xtaydi. Nazariy jihatdan bu timing orqali ma'lumot sizdiradi;hmac.compare_digestdoimiy vaqtda ishlaydi.- Timestamp yo'q. Hujumchi bir marta ushlab olgan haqiqiy so'rovni (imzosi to'g'ri!) istalgan vaqt qayta yuborishi mumkin — bu replay hujumi.
- Event id bo'yicha dublikat tekshiruvi yo'q, shuning uchun provayder retry qilsa, hammasi qaytadan bajariladi.
- Og'ir ish (email, PDF) so'rov ichida. Javob sekinlashadi, timeout bo'ladi, retry keladi — ya'ni 5-xato aynan shu yerdan "yoqiladi".
1-qadam: imzo va timestamp
Ishonchli sxema shunday: provayder bilan umumiy secret bor. Provayder timestamp.body qatorini HMAC-SHA256 bilan imzolaydi va ikkita header yuboradi: X-Webhook-Timestamp va X-Webhook-Signature. Siz xuddi shu hisobni takrorlaysiz va natijani doimiy vaqtda solishtirasiz. Timestamp imzo ichida bo'lgani uchun uni o'zgartirib bo'lmaydi, eski so'rov esa vaqt oynasidan tashqarida qolib, rad etiladi.
# webhooks/signing.py
import hashlib
import hmac
import time
TOLERANCE_SECONDS = 300
class SignatureError(Exception):
pass
def compute_signature(secret: bytes, timestamp: str, body: bytes) -> str:
signed_payload = timestamp.encode() + b"." + body
return hmac.new(secret, signed_payload, hashlib.sha256).hexdigest()
def verify_signature(
secrets: list[bytes],
timestamp: str | None,
signature: str | None,
body: bytes,
) -> None:
if not timestamp or not signature:
raise SignatureError("missing headers")
try:
ts = int(timestamp)
except ValueError:
raise SignatureError("bad timestamp") from None
if abs(time.time() - ts) > TOLERANCE_SECONDS:
raise SignatureError("timestamp outside tolerance")
# Several secrets = rotation without downtime (old + new).
for secret in secrets:
expected = compute_signature(secret, timestamp, body)
if hmac.compare_digest(expected, signature):
return
raise SignatureError("signature mismatch")
Bir nechta secret ro'yxati — rotatsiya uchun. Yangi secret'ni qo'shasiz, provayder panelida almashtirasiz, bir muddatdan keyin eskisini ro'yxatdan olib tashlaysiz. Hech qanday "uzilish oynasi" kerak emas.
# settings.py
import os
ACME_WEBHOOK_SECRETS = [
s.encode() for s in os.environ["ACME_WEBHOOK_SECRETS"].split(",") if s
]
Header nomlari va imzo formati provayderga bog'liq: kimdir hex, kimdir base64 ishlatadi, kimdir header'ga t=...,v1=... ko'rinishida bir nechta qiymat joylaydi. Mexanizm esa deyarli har doim bir xil: HMAC, xom body, timestamp. Provayder rasmiy SDK'da tekshiruv funksiyasini bersa, o'shandan foydalaning.
Imzo "bu so'rovni secret egasi yuborgan" deydi. "Bu so'rov birinchi marta keldi" demaydi. Shuning uchun timestamp oynasi va event id bo'yicha dedup — alohida, ikkalasi ham kerak.
5 daqiqalik oyna replay'ni butunlay to'xtatmaydi: oyna ichida ushlangan so'rovni qayta yuborish mumkin. Uni keyingi qadam — dublikat filtri — yopadi.
2-qadam: tez 200, ish esa navbatga
Qabul qiluvchi endpoint'ning yagona vazifasi: tekshirish, saqlash, "qabul qildim" deyish. Biznes logika — Celery worker'da. Avval event'ni bazaga yozamiz:
# webhooks/models.py
from django.db import models
class WebhookEvent(models.Model):
class Status(models.TextChoices):
PENDING = "pending"
PROCESSED = "processed"
FAILED = "failed"
provider = models.CharField(max_length=32)
event_id = models.CharField(max_length=128)
event_type = models.CharField(max_length=64)
payload = models.JSONField()
status = models.CharField(
max_length=16, choices=Status, default=Status.PENDING
)
received_at = models.DateTimeField(auto_now_add=True)
processed_at = models.DateTimeField(null=True, blank=True)
class Meta:
constraints = [
models.UniqueConstraint(
fields=["provider", "event_id"], name="uniq_webhook_event"
),
]
indexes = [models.Index(fields=["status", "received_at"])]
UniqueConstraint(provider, event_id) — dedup'ning asosi. Tekshiruv kod darajasida emas, PostgreSQL darajasida: ikki parallel so'rov bir xil event bilan kelsa ham, unique index ulardan faqat bittasiga INSERT qilishga ruxsat beradi.
# webhooks/views.py
import json
from django.conf import settings
from django.db import transaction
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST
from .models import WebhookEvent
from .signing import SignatureError, verify_signature
from .tasks import process_webhook_event
@csrf_exempt
@require_POST
def acme_webhook(request):
body = request.body # raw bytes, exactly what was signed
try:
verify_signature(
settings.ACME_WEBHOOK_SECRETS,
request.headers.get("X-Webhook-Timestamp"),
request.headers.get("X-Webhook-Signature"),
body,
)
except SignatureError:
return HttpResponse(status=400)
try:
payload = json.loads(body)
event_id = str(payload["id"])
event_type = str(payload["type"])
except (ValueError, KeyError, TypeError):
return HttpResponse(status=400)
with transaction.atomic():
event, created = WebhookEvent.objects.get_or_create(
provider="acme",
event_id=event_id,
defaults={"event_type": event_type, "payload": payload},
)
if created:
transaction.on_commit(
lambda: process_webhook_event.delay(event.pk)
)
# Duplicate or new: the answer is the same, 200.
return HttpResponse(status=200)
Bu yerda bir necha muhim detal bor:
get_or_createunique constraint bilan birga race-safe: ikkinchi so'rov INSERT'daIntegrityErroroladi, Django uni ushlab, mavjud qatorni qaytaradi (created=False).- Vazifa
transaction.on_commitorqali yuboriladi. Agar.delay()ni to'g'ridan-to'g'ri chaqirsangiz, worker qator commit bo'lishidan oldin ishga tushib, uni topa olmasligi mumkin. - Dublikatga ham 200 qaytaramiz. 409 yoki 500 qaytarsangiz, provayder "yetkazilmadi" deb, retry'ni davom ettiradi.
- Imzo xato bo'lsa, javobda sababni aytmaymiz — 400 yetarli. Sababni o'z logimizga yozamiz.
3-qadam: worker idempotent bo'lishi kerak
Dedup endpoint'da faqat "bir xil event ikki marta saqlanmasin" degan muammoni hal qiladi. Lekin Celery'ning o'zi ham vazifani ikki marta bajarishi mumkin (acks_late, worker o'lishi, visibility timeout). Shuning uchun worker ham qatorni lock qilib, statusni tekshiradi:
# webhooks/tasks.py
from datetime import timedelta
from celery import shared_task
from django.db import transaction
from django.utils import timezone
from .handlers import handle_event
from .models import WebhookEvent
@shared_task(acks_late=True)
def process_webhook_event(event_pk: int) -> None:
with transaction.atomic():
event = (
WebhookEvent.objects.select_for_update()
.filter(pk=event_pk)
.first()
)
if event is None or event.status == WebhookEvent.Status.PROCESSED:
return
handle_event(event) # DB work only, no HTTP calls under the lock
event.status = WebhookEvent.Status.PROCESSED
event.processed_at = timezone.now()
event.save(update_fields=["status", "processed_at"])
@shared_task
def requeue_stale_webhook_events() -> None:
# Safety net: broker was down when on_commit fired, worker died, etc.
cutoff = timezone.now() - timedelta(minutes=10)
stale = WebhookEvent.objects.filter(
status=WebhookEvent.Status.PENDING, received_at__lt=cutoff
).values_list("pk", flat=True)[:500]
for pk in stale:
process_webhook_event.delay(pk)
select_for_update() ikkinchi parallel worker'ni birinchisi commit qilguncha kutishga majbur qiladi; keyin ikkinchisi PROCESSED statusni ko'radi va chiqib ketadi. Shart: handle_event ichida tashqi HTTP chaqiruvlar bo'lmasin — lock ostida tarmoqni kutish connection pool va boshqa so'rovlarni bloklaydi. Email yoki tashqi API kerak bo'lsa, ularni alohida vazifaga, yana on_commit orqali chiqaring.
requeue_stale_webhook_events — xavfsizlik to'ri. on_commit ichidagi .delay() broker ishlamay turgan paytga to'g'ri kelsa, event bazada pending bo'lib qoladi va hech kim uni olmaydi. Celery beat bilan har bir necha daqiqada ishga tushiriladigan bu vazifa shunday "yetim" event'larni qayta navbatga qo'yadi. Worker idempotent bo'lgani uchun ortiqcha yuborish zarar qilmaydi.
4-qadam: event'lar tartibsiz keladi
Provayder event'larni turli worker'lardan, retry bilan, parallel yuboradi. updated dan keyin yaratilgan canceled sizga birinchi kelishi mumkin. Agar handler oxirgi kelgan event'ni "eng yangi holat" deb qabul qilsa, holat orqaga qaytadi.
Ikki ishonchli yondashuv bor:
- Versiya yoki vaqt bo'yicha shartli yangilash. Obyektda oxirgi qo'llangan event vaqtini saqlaymiz va faqat yangiroq event'ni qo'llaymiz.
- "Thin event" + qayta o'qish. Event'ni faqat signal deb qabul qilasiz: "
sub_42o'zgardi". Keyin provayder API'sidan obyektning joriy holatini olasiz. Tartib muammosi yo'qoladi, lekin har bir event'ga bitta API chaqiruvi qo'shiladi.
Birinchi yondashuv kodda:
# webhooks/handlers.py
from datetime import datetime, timezone as dt_timezone
from django.db.models import Q
from billing.models import Subscription
def handle_event(event) -> None:
if event.event_type != "subscription.updated":
return # unknown types: store, acknowledge, ignore
data = event.payload["data"]
occurred_at = datetime.fromtimestamp(
event.payload["created"], tz=dt_timezone.utc
)
# Conditional UPDATE: an older event can never overwrite a newer state.
Subscription.objects.filter(
external_id=data["subscription_id"],
).filter(
Q(last_event_at__isnull=True) | Q(last_event_at__lt=occurred_at)
).update(
status=data["status"],
last_event_at=occurred_at,
)
Bu bitta atomar SQL'ga aylanadi — o'qib, keyin yozish oralig'ida race yo'q:
UPDATE billing_subscription
SET status = 'canceled', last_event_at = '2026-11-20 10:00:05+00'
WHERE external_id = 'sub_42'
AND (last_event_at IS NULL OR last_event_at < '2026-11-20 10:00:05+00');
Ehtiyot bo'ling: ikki event bir xil soniyada yaratilgan bo'lsa, < ikkinchisini tashlab yuboradi. Provayder event'da sequence yoki obyekt versiyasini bersa, vaqt o'rniga o'shandan foydalaning. Bunday maydon bo'lmasa va holat muhim bo'lsa (to'lov, obuna), ikkinchi yondashuv xavfsizroq.
Yuboruvchi tomon: retry, backoff, jitter
Agar o'zingiz mijozlarga webhook yuborsangiz, endi rollar almashadi: siz "at-least-once" va'da qilasiz va qabul qiluvchi yuqoridagi hamma narsani qila olishi uchun sharoit yaratasiz. Asosiy qoidalar:
- Event id barcha urinishlarda bir xil, timestamp va imzo esa har urinishda yangidan hisoblanadi (aks holda bir soatdan keyingi retry qabul qiluvchining 5 daqiqalik oynasidan o'tmaydi).
- Retry eksponensial backoff va jitter bilan: bir vaqtda yiqilgan yuzlab mijoz endpoint'iga bir soniyada qaytib "urilmaslik" uchun.
- 2xx — muvaffaqiyat. 5xx, 408, 429 va tarmoq xatolari — retry. Qolgan 4xx — retry qilinmaydi, lekin logga yoziladi.
- Har doim timeout.
requestsda standart timeout yo'q, u abadiy kutishi mumkin. - Event'ni yuborish vazifasini biznes tranzaksiyasi bilan birga yarating (outbox yoki
on_commit), aks holda event yo'qoladi.
# outgoing/tasks.py
import json
import time
import requests
from celery import shared_task
from django.utils import timezone
from webhooks.signing import compute_signature
from .models import WebhookDelivery
class RetryableDeliveryError(Exception):
pass
@shared_task(
autoretry_for=(
RetryableDeliveryError,
requests.ConnectionError,
requests.Timeout,
),
retry_backoff=5, # 5s, 10s, 20s, 40s ...
retry_backoff_max=3600, # never wait more than an hour
retry_jitter=True, # spread retries, avoid a thundering herd
max_retries=12,
acks_late=True,
)
def deliver_webhook(delivery_id: int) -> None:
delivery = WebhookDelivery.objects.select_related("endpoint").get(
pk=delivery_id
)
if delivery.delivered_at is not None:
return
body = json.dumps(
delivery.payload, separators=(",", ":"), ensure_ascii=False
).encode()
ts = str(int(time.time())) # fresh timestamp on every attempt
signature = compute_signature(
delivery.endpoint.secret.encode(), ts, body
)
resp = requests.post(
delivery.endpoint.url,
data=body,
headers={
"Content-Type": "application/json",
"X-Webhook-Id": str(delivery.event_id), # same on every retry
"X-Webhook-Timestamp": ts,
"X-Webhook-Signature": signature,
},
timeout=(3, 10),
allow_redirects=False,
)
delivery.last_status = resp.status_code
if 200 <= resp.status_code < 300:
delivery.delivered_at = timezone.now()
delivery.save(update_fields=["last_status", "delivered_at"])
return
delivery.save(update_fields=["last_status"])
if resp.status_code in (408, 429) or resp.status_code >= 500:
raise RetryableDeliveryError(f"HTTP {resp.status_code}")
# Other 4xx: the receiver rejected the request; retrying won't help.
Celery'da retry_backoff=5 kechikishni 5, 10, 20, 40... soniya qilib oshiradi, retry_backoff_max uni chegaralaydi, retry_jitter=True esa har bir kechikishni 0 va hisoblangan qiymat oralig'ida tasodifiy tanlaydi. max_retries tugagach, Celery oxirgi xatoni ko'taradi — shu paytda yetkazishni failed deb belgilab, mijozga dashboard'da ko'rsatish va qo'lda "qayta yuborish" tugmasini berish foydali. Mijozlar URL'ni o'zi kiritsa, allow_redirects=False va ichki IP manzillarni bloklash SSRF'dan himoya qiladi.
Mahalliy test
Endpoint'ni provayder sandbox'isiz ham tekshirish mumkin — imzoni openssl bilan hisoblaymiz:
SECRET='whsec_test'
BODY='{"id":"evt_1","type":"ping","created":1763632800,"data":{}}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $NF}')
curl -sS -i -X POST http://localhost:8000/webhooks/acme/ \
-H 'Content-Type: application/json' \
-H "X-Webhook-Timestamp: $TS" \
-H "X-Webhook-Signature: $SIG" \
--data-binary "$BODY"
Xuddi shu buyruqni ikki marta yuboring: ikkinchisi ham 200 qaytarishi, lekin bazada bitta qator va bitta bajarilgan vazifa bo'lishi kerak. TS ni 10 daqiqa oldingi qiymatga almashtirsangiz, 400 olishingiz kerak.
Production'da uchraydigan tuzoqlar
| Muammo | Belgisi | Yechim |
|---|---|---|
| Body o'zgartirilgan | Imzo doim mos kelmaydi | Faqat request.body ni tekshiring, middleware body'ni qayta yozmasin |
| Server soati og'ib ketgan | Barcha event'lar "eski" | NTP, oynani juda tor qilmang |
| Katta payload | RequestDataTooBig, 400 |
DATA_UPLOAD_MAX_MEMORY_SIZE request.body ga ham ta'sir qiladi |
| Broker ishlamadi | Event pending da qotib qoldi |
Periodik requeue vazifasi |
| Jadval cheksiz o'sadi | Sekin query'lar | Eski processed qatorlarni tozalash yoki partitioning |
Yana bir nechta narsa:
- Dedup kaliti — provayderning event id'si, payload hash'i emas. Retry'da ayrim provayderlar payload'ga urinish raqamini qo'shadi, hash o'zgaradi.
- Noma'lum event turlari uchun ham 200 qaytaring va saqlab qo'ying. Provayder yangi tur qo'shganda, endpoint'ingiz retry bo'roniga aylanmasin.
- Secret'ni logga yozmang, imzo xatosini esa yozing — bu hujum yoki noto'g'ri konfiguratsiyaning birinchi belgisi.
- IP allowlist imzoning o'rnini bosmaydi; qo'shimcha qatlam bo'lishi mumkin, xolos.
csrf_exemptfaqat webhook view'iga. Butun URL prefiksi uchun CSRF'ni o'chirib qo'ymang.
Checklist
- HMAC-SHA256, xom
request.body,hmac.compare_digest. - Imzo ichida timestamp, ±5 daqiqa oyna.
- Rotatsiya uchun bir nechta secret qabul qiling.
UniqueConstraint(provider, event_id)+get_or_create.- Saqlash,
on_commitbilan navbatga qo'yish, darhol 200 (dublikatga ham). - Worker'da
select_for_update+ status tekshiruvi, lock ostida HTTP yo'q. pendingda qotgan event'lar uchun periodik requeue.- Tartib uchun shartli
UPDATEyoki API'dan qayta o'qish. - Yuborsangiz: barqaror event id, yangi imzo, backoff + jitter, timeout, qo'lda qayta yuborish.
Xulosa
Webhook — bu "bir marta keladigan buyruq" emas, balki "kamida bir marta, istalgan tartibda keladigan, soxta bo'lishi ham mumkin bo'lgan xabar". Imzo va timestamp kim yuborganini, unique constraint necha marta qayta ishlanganini, shartli yangilash esa qaysi holat oxirgisi ekanini hal qiladi. Endpoint faqat qabul qilsin, ishni navbat qilsin — shunda retry'lar qo'rqinchli emas, balki shunchaki shovqin bo'lib qoladi.
// izohlar
Izohlar yuklanmoqda...