Barcha yozuvlar
Dasturlash

Webhook'ga ko'r-ko'rona ishonmang: HMAC imzo, replay himoyasi, dublikatlar va navbat

Provayder bitta eventni uch marta yuboradi, ba'zan noto'g'ri tartibda, ba'zan esa uni umuman provayder yubormagan bo'ladi. Django va Celery'da webhook'ni xavfsiz va bir martadan qabul qilishni qadamma-qadam ko'ramiz.

Tasavvur qiling: to'lov provayderi sizning /webhooks/acme/ endpoint'ingizga payment.succeeded event'ini yuboradi. Handler buyurtmani yangilaydi, keyin chek PDF'ini yasab email jo'natadi. Bu bir necha soniya oladi, provayderning timeout'i esa undan qisqa. Provayder javob olmadi, demak "yetkazilmadi" deb hisoblaydi va bir daqiqadan keyin xuddi shu event'ni qayta yuboradi. Mijozga ikkita chek boradi, buyurtma tarixida ikkita yozuv paydo bo'ladi.

Shu payt boshqa joyda kimdir endpoint URL'ini loglardan topib oldi va qo'lda {"type": "payment.succeeded", "order_id": 777} yubordi. Imzo tekshiruvi zaif bo'lsa, buyurtma "to'langan" bo'lib qoladi. Uchinchi muammo: obuna uchun subscription.canceled va subscription.updated deyarli bir vaqtda yuborildi, lekin ikkinchisi birinchi yetib keldi va bekor qilingan obuna yana "active" bo'lib qoldi.

Bu uchta muammo — soxta so'rov, takroriy yetkazish va tartibsiz kelish — webhook'ning tabiatidan kelib chiqadi. Ularni "kam uchraydi" deb chetlab o'tib bo'lmaydi, chunki deyarli barcha provayderlar ochiqchasiga "at-least-once" (kamida bir marta) yetkazishni va'da qiladi, "exactly-once" emas. Bu maqolada qabul qiluvchi tomonni to'liq quramiz, oxirida esa o'zingiz webhook yuborsangiz nima qilish kerakligini ko'ramiz.

Sodda handler nega buziladi

Ko'p loyihalarda birinchi versiya shunga o'xshaydi:

# webhooks/views.py  -- DON'T do this
import hashlib
import json

from django.conf import settings
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt

from orders.models import Order
from orders.emails import send_receipt_email


@csrf_exempt
def acme_webhook(request):
    data = json.loads(request.body)
    expected = hashlib.sha256(
        (settings.ACME_SECRET + json.dumps(data)).encode()
    ).hexdigest()
    if expected != request.headers.get("X-Webhook-Signature"):
        return HttpResponse(status=401)

    order = Order.objects.get(pk=data["data"]["order_id"])
    order.status = data["data"]["status"]
    order.save()
    send_receipt_email(order)  # SMTP, PDF, a few seconds
    return HttpResponse("ok")

Bu yerda bitta emas, oltita xato bor:

  1. sha256(secret + message) — bu HMAC emas. SHA-256 kabi Merkle–Damgård hash'lar uchun bunday konstruksiya length extension hujumiga ochiq. Kriptografik imzo uchun hmac moduli bor, aynan shu sababli.
  2. json.dumps(json.loads(body)) asl baytlarni qaytarmaydi: bo'shliqlar, kalitlar tartibi, unicode escape farq qiladi. Provayder xom baytlarni imzolagan, siz ham xom baytlarni tekshirishingiz kerak.
  3. != bilan solishtirish birinchi farq qilgan belgida to'xtaydi. Nazariy jihatdan bu timing orqali ma'lumot sizdiradi; hmac.compare_digest doimiy vaqtda ishlaydi.
  4. Timestamp yo'q. Hujumchi bir marta ushlab olgan haqiqiy so'rovni (imzosi to'g'ri!) istalgan vaqt qayta yuborishi mumkin — bu replay hujumi.
  5. Event id bo'yicha dublikat tekshiruvi yo'q, shuning uchun provayder retry qilsa, hammasi qaytadan bajariladi.
  6. Og'ir ish (email, PDF) so'rov ichida. Javob sekinlashadi, timeout bo'ladi, retry keladi — ya'ni 5-xato aynan shu yerdan "yoqiladi".

1-qadam: imzo va timestamp

Ishonchli sxema shunday: provayder bilan umumiy secret bor. Provayder timestamp.body qatorini HMAC-SHA256 bilan imzolaydi va ikkita header yuboradi: X-Webhook-Timestamp va X-Webhook-Signature. Siz xuddi shu hisobni takrorlaysiz va natijani doimiy vaqtda solishtirasiz. Timestamp imzo ichida bo'lgani uchun uni o'zgartirib bo'lmaydi, eski so'rov esa vaqt oynasidan tashqarida qolib, rad etiladi.

# webhooks/signing.py
import hashlib
import hmac
import time

TOLERANCE_SECONDS = 300


class SignatureError(Exception):
    pass


def compute_signature(secret: bytes, timestamp: str, body: bytes) -> str:
    signed_payload = timestamp.encode() + b"." + body
    return hmac.new(secret, signed_payload, hashlib.sha256).hexdigest()


def verify_signature(
    secrets: list[bytes],
    timestamp: str | None,
    signature: str | None,
    body: bytes,
) -> None:
    if not timestamp or not signature:
        raise SignatureError("missing headers")
    try:
        ts = int(timestamp)
    except ValueError:
        raise SignatureError("bad timestamp") from None
    if abs(time.time() - ts) > TOLERANCE_SECONDS:
        raise SignatureError("timestamp outside tolerance")

    # Several secrets = rotation without downtime (old + new).
    for secret in secrets:
        expected = compute_signature(secret, timestamp, body)
        if hmac.compare_digest(expected, signature):
            return
    raise SignatureError("signature mismatch")

Bir nechta secret ro'yxati — rotatsiya uchun. Yangi secret'ni qo'shasiz, provayder panelida almashtirasiz, bir muddatdan keyin eskisini ro'yxatdan olib tashlaysiz. Hech qanday "uzilish oynasi" kerak emas.

# settings.py
import os

ACME_WEBHOOK_SECRETS = [
    s.encode() for s in os.environ["ACME_WEBHOOK_SECRETS"].split(",") if s
]

Header nomlari va imzo formati provayderga bog'liq: kimdir hex, kimdir base64 ishlatadi, kimdir header'ga t=...,v1=... ko'rinishida bir nechta qiymat joylaydi. Mexanizm esa deyarli har doim bir xil: HMAC, xom body, timestamp. Provayder rasmiy SDK'da tekshiruv funksiyasini bersa, o'shandan foydalaning.

Imzo "bu so'rovni secret egasi yuborgan" deydi. "Bu so'rov birinchi marta keldi" demaydi. Shuning uchun timestamp oynasi va event id bo'yicha dedup — alohida, ikkalasi ham kerak.

5 daqiqalik oyna replay'ni butunlay to'xtatmaydi: oyna ichida ushlangan so'rovni qayta yuborish mumkin. Uni keyingi qadam — dublikat filtri — yopadi.

2-qadam: tez 200, ish esa navbatga

Qabul qiluvchi endpoint'ning yagona vazifasi: tekshirish, saqlash, "qabul qildim" deyish. Biznes logika — Celery worker'da. Avval event'ni bazaga yozamiz:

# webhooks/models.py
from django.db import models


class WebhookEvent(models.Model):
    class Status(models.TextChoices):
        PENDING = "pending"
        PROCESSED = "processed"
        FAILED = "failed"

    provider = models.CharField(max_length=32)
    event_id = models.CharField(max_length=128)
    event_type = models.CharField(max_length=64)
    payload = models.JSONField()
    status = models.CharField(
        max_length=16, choices=Status, default=Status.PENDING
    )
    received_at = models.DateTimeField(auto_now_add=True)
    processed_at = models.DateTimeField(null=True, blank=True)

    class Meta:
        constraints = [
            models.UniqueConstraint(
                fields=["provider", "event_id"], name="uniq_webhook_event"
            ),
        ]
        indexes = [models.Index(fields=["status", "received_at"])]

UniqueConstraint(provider, event_id) — dedup'ning asosi. Tekshiruv kod darajasida emas, PostgreSQL darajasida: ikki parallel so'rov bir xil event bilan kelsa ham, unique index ulardan faqat bittasiga INSERT qilishga ruxsat beradi.

# webhooks/views.py
import json

from django.conf import settings
from django.db import transaction
from django.http import HttpResponse
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST

from .models import WebhookEvent
from .signing import SignatureError, verify_signature
from .tasks import process_webhook_event


@csrf_exempt
@require_POST
def acme_webhook(request):
    body = request.body  # raw bytes, exactly what was signed
    try:
        verify_signature(
            settings.ACME_WEBHOOK_SECRETS,
            request.headers.get("X-Webhook-Timestamp"),
            request.headers.get("X-Webhook-Signature"),
            body,
        )
    except SignatureError:
        return HttpResponse(status=400)

    try:
        payload = json.loads(body)
        event_id = str(payload["id"])
        event_type = str(payload["type"])
    except (ValueError, KeyError, TypeError):
        return HttpResponse(status=400)

    with transaction.atomic():
        event, created = WebhookEvent.objects.get_or_create(
            provider="acme",
            event_id=event_id,
            defaults={"event_type": event_type, "payload": payload},
        )
        if created:
            transaction.on_commit(
                lambda: process_webhook_event.delay(event.pk)
            )

    # Duplicate or new: the answer is the same, 200.
    return HttpResponse(status=200)

Bu yerda bir necha muhim detal bor:

  • get_or_create unique constraint bilan birga race-safe: ikkinchi so'rov INSERT'da IntegrityError oladi, Django uni ushlab, mavjud qatorni qaytaradi (created=False).
  • Vazifa transaction.on_commit orqali yuboriladi. Agar .delay() ni to'g'ridan-to'g'ri chaqirsangiz, worker qator commit bo'lishidan oldin ishga tushib, uni topa olmasligi mumkin.
  • Dublikatga ham 200 qaytaramiz. 409 yoki 500 qaytarsangiz, provayder "yetkazilmadi" deb, retry'ni davom ettiradi.
  • Imzo xato bo'lsa, javobda sababni aytmaymiz — 400 yetarli. Sababni o'z logimizga yozamiz.

3-qadam: worker idempotent bo'lishi kerak

Dedup endpoint'da faqat "bir xil event ikki marta saqlanmasin" degan muammoni hal qiladi. Lekin Celery'ning o'zi ham vazifani ikki marta bajarishi mumkin (acks_late, worker o'lishi, visibility timeout). Shuning uchun worker ham qatorni lock qilib, statusni tekshiradi:

# webhooks/tasks.py
from datetime import timedelta

from celery import shared_task
from django.db import transaction
from django.utils import timezone

from .handlers import handle_event
from .models import WebhookEvent


@shared_task(acks_late=True)
def process_webhook_event(event_pk: int) -> None:
    with transaction.atomic():
        event = (
            WebhookEvent.objects.select_for_update()
            .filter(pk=event_pk)
            .first()
        )
        if event is None or event.status == WebhookEvent.Status.PROCESSED:
            return
        handle_event(event)  # DB work only, no HTTP calls under the lock
        event.status = WebhookEvent.Status.PROCESSED
        event.processed_at = timezone.now()
        event.save(update_fields=["status", "processed_at"])


@shared_task
def requeue_stale_webhook_events() -> None:
    # Safety net: broker was down when on_commit fired, worker died, etc.
    cutoff = timezone.now() - timedelta(minutes=10)
    stale = WebhookEvent.objects.filter(
        status=WebhookEvent.Status.PENDING, received_at__lt=cutoff
    ).values_list("pk", flat=True)[:500]
    for pk in stale:
        process_webhook_event.delay(pk)

select_for_update() ikkinchi parallel worker'ni birinchisi commit qilguncha kutishga majbur qiladi; keyin ikkinchisi PROCESSED statusni ko'radi va chiqib ketadi. Shart: handle_event ichida tashqi HTTP chaqiruvlar bo'lmasin — lock ostida tarmoqni kutish connection pool va boshqa so'rovlarni bloklaydi. Email yoki tashqi API kerak bo'lsa, ularni alohida vazifaga, yana on_commit orqali chiqaring.

requeue_stale_webhook_events — xavfsizlik to'ri. on_commit ichidagi .delay() broker ishlamay turgan paytga to'g'ri kelsa, event bazada pending bo'lib qoladi va hech kim uni olmaydi. Celery beat bilan har bir necha daqiqada ishga tushiriladigan bu vazifa shunday "yetim" event'larni qayta navbatga qo'yadi. Worker idempotent bo'lgani uchun ortiqcha yuborish zarar qilmaydi.

4-qadam: event'lar tartibsiz keladi

Provayder event'larni turli worker'lardan, retry bilan, parallel yuboradi. updated dan keyin yaratilgan canceled sizga birinchi kelishi mumkin. Agar handler oxirgi kelgan event'ni "eng yangi holat" deb qabul qilsa, holat orqaga qaytadi.

Ikki ishonchli yondashuv bor:

  1. Versiya yoki vaqt bo'yicha shartli yangilash. Obyektda oxirgi qo'llangan event vaqtini saqlaymiz va faqat yangiroq event'ni qo'llaymiz.
  2. "Thin event" + qayta o'qish. Event'ni faqat signal deb qabul qilasiz: "sub_42 o'zgardi". Keyin provayder API'sidan obyektning joriy holatini olasiz. Tartib muammosi yo'qoladi, lekin har bir event'ga bitta API chaqiruvi qo'shiladi.

Birinchi yondashuv kodda:

# webhooks/handlers.py
from datetime import datetime, timezone as dt_timezone

from django.db.models import Q

from billing.models import Subscription


def handle_event(event) -> None:
    if event.event_type != "subscription.updated":
        return  # unknown types: store, acknowledge, ignore

    data = event.payload["data"]
    occurred_at = datetime.fromtimestamp(
        event.payload["created"], tz=dt_timezone.utc
    )

    # Conditional UPDATE: an older event can never overwrite a newer state.
    Subscription.objects.filter(
        external_id=data["subscription_id"],
    ).filter(
        Q(last_event_at__isnull=True) | Q(last_event_at__lt=occurred_at)
    ).update(
        status=data["status"],
        last_event_at=occurred_at,
    )

Bu bitta atomar SQL'ga aylanadi — o'qib, keyin yozish oralig'ida race yo'q:

UPDATE billing_subscription
SET status = 'canceled', last_event_at = '2026-11-20 10:00:05+00'
WHERE external_id = 'sub_42'
  AND (last_event_at IS NULL OR last_event_at < '2026-11-20 10:00:05+00');

Ehtiyot bo'ling: ikki event bir xil soniyada yaratilgan bo'lsa, < ikkinchisini tashlab yuboradi. Provayder event'da sequence yoki obyekt versiyasini bersa, vaqt o'rniga o'shandan foydalaning. Bunday maydon bo'lmasa va holat muhim bo'lsa (to'lov, obuna), ikkinchi yondashuv xavfsizroq.

Yuboruvchi tomon: retry, backoff, jitter

Agar o'zingiz mijozlarga webhook yuborsangiz, endi rollar almashadi: siz "at-least-once" va'da qilasiz va qabul qiluvchi yuqoridagi hamma narsani qila olishi uchun sharoit yaratasiz. Asosiy qoidalar:

  • Event id barcha urinishlarda bir xil, timestamp va imzo esa har urinishda yangidan hisoblanadi (aks holda bir soatdan keyingi retry qabul qiluvchining 5 daqiqalik oynasidan o'tmaydi).
  • Retry eksponensial backoff va jitter bilan: bir vaqtda yiqilgan yuzlab mijoz endpoint'iga bir soniyada qaytib "urilmaslik" uchun.
  • 2xx — muvaffaqiyat. 5xx, 408, 429 va tarmoq xatolari — retry. Qolgan 4xx — retry qilinmaydi, lekin logga yoziladi.
  • Har doim timeout. requests da standart timeout yo'q, u abadiy kutishi mumkin.
  • Event'ni yuborish vazifasini biznes tranzaksiyasi bilan birga yarating (outbox yoki on_commit), aks holda event yo'qoladi.
# outgoing/tasks.py
import json
import time

import requests
from celery import shared_task
from django.utils import timezone

from webhooks.signing import compute_signature

from .models import WebhookDelivery


class RetryableDeliveryError(Exception):
    pass


@shared_task(
    autoretry_for=(
        RetryableDeliveryError,
        requests.ConnectionError,
        requests.Timeout,
    ),
    retry_backoff=5,        # 5s, 10s, 20s, 40s ...
    retry_backoff_max=3600,  # never wait more than an hour
    retry_jitter=True,       # spread retries, avoid a thundering herd
    max_retries=12,
    acks_late=True,
)
def deliver_webhook(delivery_id: int) -> None:
    delivery = WebhookDelivery.objects.select_related("endpoint").get(
        pk=delivery_id
    )
    if delivery.delivered_at is not None:
        return

    body = json.dumps(
        delivery.payload, separators=(",", ":"), ensure_ascii=False
    ).encode()
    ts = str(int(time.time()))  # fresh timestamp on every attempt
    signature = compute_signature(
        delivery.endpoint.secret.encode(), ts, body
    )

    resp = requests.post(
        delivery.endpoint.url,
        data=body,
        headers={
            "Content-Type": "application/json",
            "X-Webhook-Id": str(delivery.event_id),  # same on every retry
            "X-Webhook-Timestamp": ts,
            "X-Webhook-Signature": signature,
        },
        timeout=(3, 10),
        allow_redirects=False,
    )

    delivery.last_status = resp.status_code
    if 200 <= resp.status_code < 300:
        delivery.delivered_at = timezone.now()
        delivery.save(update_fields=["last_status", "delivered_at"])
        return

    delivery.save(update_fields=["last_status"])
    if resp.status_code in (408, 429) or resp.status_code >= 500:
        raise RetryableDeliveryError(f"HTTP {resp.status_code}")
    # Other 4xx: the receiver rejected the request; retrying won't help.

Celery'da retry_backoff=5 kechikishni 5, 10, 20, 40... soniya qilib oshiradi, retry_backoff_max uni chegaralaydi, retry_jitter=True esa har bir kechikishni 0 va hisoblangan qiymat oralig'ida tasodifiy tanlaydi. max_retries tugagach, Celery oxirgi xatoni ko'taradi — shu paytda yetkazishni failed deb belgilab, mijozga dashboard'da ko'rsatish va qo'lda "qayta yuborish" tugmasini berish foydali. Mijozlar URL'ni o'zi kiritsa, allow_redirects=False va ichki IP manzillarni bloklash SSRF'dan himoya qiladi.

Mahalliy test

Endpoint'ni provayder sandbox'isiz ham tekshirish mumkin — imzoni openssl bilan hisoblaymiz:

SECRET='whsec_test'
BODY='{"id":"evt_1","type":"ping","created":1763632800,"data":{}}'
TS=$(date +%s)
SIG=$(printf '%s.%s' "$TS" "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | awk '{print $NF}')

curl -sS -i -X POST http://localhost:8000/webhooks/acme/ \
  -H 'Content-Type: application/json' \
  -H "X-Webhook-Timestamp: $TS" \
  -H "X-Webhook-Signature: $SIG" \
  --data-binary "$BODY"

Xuddi shu buyruqni ikki marta yuboring: ikkinchisi ham 200 qaytarishi, lekin bazada bitta qator va bitta bajarilgan vazifa bo'lishi kerak. TS ni 10 daqiqa oldingi qiymatga almashtirsangiz, 400 olishingiz kerak.

Production'da uchraydigan tuzoqlar

Muammo Belgisi Yechim
Body o'zgartirilgan Imzo doim mos kelmaydi Faqat request.body ni tekshiring, middleware body'ni qayta yozmasin
Server soati og'ib ketgan Barcha event'lar "eski" NTP, oynani juda tor qilmang
Katta payload RequestDataTooBig, 400 DATA_UPLOAD_MAX_MEMORY_SIZE request.body ga ham ta'sir qiladi
Broker ishlamadi Event pending da qotib qoldi Periodik requeue vazifasi
Jadval cheksiz o'sadi Sekin query'lar Eski processed qatorlarni tozalash yoki partitioning

Yana bir nechta narsa:

  • Dedup kaliti — provayderning event id'si, payload hash'i emas. Retry'da ayrim provayderlar payload'ga urinish raqamini qo'shadi, hash o'zgaradi.
  • Noma'lum event turlari uchun ham 200 qaytaring va saqlab qo'ying. Provayder yangi tur qo'shganda, endpoint'ingiz retry bo'roniga aylanmasin.
  • Secret'ni logga yozmang, imzo xatosini esa yozing — bu hujum yoki noto'g'ri konfiguratsiyaning birinchi belgisi.
  • IP allowlist imzoning o'rnini bosmaydi; qo'shimcha qatlam bo'lishi mumkin, xolos.
  • csrf_exempt faqat webhook view'iga. Butun URL prefiksi uchun CSRF'ni o'chirib qo'ymang.

Checklist

  • HMAC-SHA256, xom request.body, hmac.compare_digest.
  • Imzo ichida timestamp, ±5 daqiqa oyna.
  • Rotatsiya uchun bir nechta secret qabul qiling.
  • UniqueConstraint(provider, event_id) + get_or_create.
  • Saqlash, on_commit bilan navbatga qo'yish, darhol 200 (dublikatga ham).
  • Worker'da select_for_update + status tekshiruvi, lock ostida HTTP yo'q.
  • pending da qotgan event'lar uchun periodik requeue.
  • Tartib uchun shartli UPDATE yoki API'dan qayta o'qish.
  • Yuborsangiz: barqaror event id, yangi imzo, backoff + jitter, timeout, qo'lda qayta yuborish.

Xulosa

Webhook — bu "bir marta keladigan buyruq" emas, balki "kamida bir marta, istalgan tartibda keladigan, soxta bo'lishi ham mumkin bo'lgan xabar". Imzo va timestamp kim yuborganini, unique constraint necha marta qayta ishlanganini, shartli yangilash esa qaysi holat oxirgisi ekanini hal qiladi. Endpoint faqat qabul qilsin, ishni navbat qilsin — shunda retry'lar qo'rqinchli emas, balki shunchaki shovqin bo'lib qoladi.

// izohlar

Izohlar yuklanmoqda...

    // keyingi qadam

    Keling, birga nimadir quramiz.

    Telegramda yozish
    aslbek@asldev: ~
    ❯