All notes
Fintech

Idempotency keys: how a lost response turns into a double charge, and how to stop it

The response got lost, the client retried, and the customer paid twice. Idempotency-Key, a PostgreSQL unique constraint and one transaction make retries safe: a full Django implementation, compared with Stripe's approach.

A user taps "Pay" in a mobile app. The request reaches the server, 150,000 som is debited from their wallet, a Payment row is created. Then the response gets lost on the way back: the phone goes into an elevator, the connection switches from mobile data to Wi-Fi, or the load balancer hits its 30-second timeout. The app shows "Network error". The user taps again. The app itself probably retries automatically too.

The server now runs the same payment twice, and the customer has paid 300,000. Nothing exotic happened here. It's just how networks behave: the client cannot tell whether the request never ran or ran and the response was lost. In both cases it sees the same thing: a timeout.

This post covers the standard fix, idempotency keys: the Idempotency-Key header, storing the key together with a hash of the request and the response, how a PostgreSQL unique constraint behaves under concurrent requests, key expiry, and how all of this compares to Stripe's approach. At the end you get a complete, working Django + PostgreSQL implementation.

What idempotency means and why POST doesn't have it

An operation is idempotent if running it once or ten times leaves the system in the same state. HTTP semantics define GET, PUT and DELETE as idempotent: send PUT /users/7 with the same body twice and nothing changes the second time. POST is not: every POST /payments/ creates a new payment.

To make a retry safe, the client has to tell the server "this is the same request, not a new one". It does that by generating a unique key per intent and sending it in a header:

KEY=$(uuidgen)

curl -X POST https://api.example.com/v1/payments/ \
  -H "Authorization: Bearer $TOKEN" \
  -H "Idempotency-Key: $KEY" \
  -H "Content-Type: application/json" \
  -d '{"order_id": 4812, "amount": 15000000}'

# Timeout? Send exactly the same request with the same $KEY again.

The first time the server sees a key, it runs the request and stores the response alongside the key. When a request with the same key arrives again, it runs nothing and returns the stored response. From the client's side retries are now safe: whatever happens, money moves once.

One crucial detail: the key is generated per intent, not per attempt. One UUID is created when the user taps "Pay", and every retry carries that same UUID. Generate a fresh key on each retry and the whole mechanism is useless.

Where the naive approach breaks

The first thing most of us write looks like this:

# WRONG: check-then-act
def create_payment(request):
    key = request.headers.get("Idempotency-Key")
    existing = IdempotencyKey.objects.filter(user=request.user, key=key).first()
    if existing:
        return JsonResponse(existing.response_body, status=existing.response_code)

    code, body = charge_wallet(request.user, amount, order_id)  # money moves here

    IdempotencyKey.objects.create(
        user=request.user, key=key, response_code=code, response_body=body
    )
    return JsonResponse(body, status=code)

There are three problems here.

1. A race condition. Two requests arrive milliseconds apart (say, the client retried after a timeout while the first request is still running). Both call filter().first(), both get None, both debit the wallet. Classic check-then-act: another process slips in between the check and the action. We talked about select_for_update in earlier posts, but it doesn't help here: you can't lock a row that doesn't exist yet.

2. Money moved, key not saved. charge_wallet and create run separately. If the process dies between them (OOM kill, worker restart during a deploy), the money is gone and there's no key. The next retry debits again.

3. The request body isn't checked. If a buggy client reuses a key with a different amount, the server replays the old response and the client believes it paid 100,000 when it actually paid 150,000. That mistake should be reported loudly, not swallowed.

Disabling the button on the frontend doesn't fix this either: it stops double clicks, but not network retries, proxies, or the app's own retry logic.

The mechanism: let a unique constraint be the referee

The most reliable way to kill the race is to stop asking application code "who was first" and let the database decide. With a unique constraint on (user_id, key), PostgreSQL will never store two rows with the same key.

What's interesting is how PostgreSQL handles this between concurrent transactions. If transaction A has inserted a key but not yet committed, transaction B's INSERT of the same key doesn't fail right away. It waits, because it needs to know whether A commits or rolls back. You can watch this in two psql windows:

-- psql session A
BEGIN;
INSERT INTO payments_idempotencykey
    (user_id, key, request_hash, status, locked_at, created_at)
VALUES (1, 'k-1', 'abc', 'started', now(), now());

-- psql session B: this statement hangs until A finishes
INSERT INTO payments_idempotencykey
    (user_id, key, request_hash, status, locked_at, created_at)
VALUES (1, 'k-1', 'abc', 'started', now(), now());

-- session A
COMMIT;

-- session B now fails:
-- ERROR:  duplicate key value violates unique constraint "uniq_idem_user_key"
-- (if A had done ROLLBACK instead, B's INSERT would succeed)

So the unique index guarantees that exactly one of two requests "wins" and the other gets an IntegrityError. No hand-rolled locking needed.

Don't check the key, claim it. If the INSERT succeeds, the request is yours; if you get an IntegrityError, someone got there first.

That gives us a three-step flow:

  1. Claim the key. Insert a row with status started in its own short transaction and commit immediately, so other requests see right away that the key is taken.
  2. Do the work. Debiting the wallet and writing the response into the key row happen in one transaction. Either both are saved, or neither.
  3. Answer repeats. Key is completed: return the stored status code and body. Key is started: 409 Conflict. Hash differs: 422.

Implementation: Django + PostgreSQL

Models first. Amounts are stored as integers in tiyin (there's a separate post on why):

# payments/models.py
from django.conf import settings
from django.db import models


class IdempotencyKey(models.Model):
    class Status(models.TextChoices):
        STARTED = "started"
        COMPLETED = "completed"

    user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
    key = models.CharField(max_length=255)
    request_hash = models.CharField(max_length=64)
    status = models.CharField(max_length=16, choices=Status, default=Status.STARTED)
    response_code = models.PositiveSmallIntegerField(null=True)
    response_body = models.JSONField(null=True)
    locked_at = models.DateTimeField(null=True)
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        constraints = [
            models.UniqueConstraint(fields=["user", "key"], name="uniq_idem_user_key"),
        ]
        indexes = [models.Index(fields=["created_at"])]


class Wallet(models.Model):
    user = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)
    balance = models.PositiveBigIntegerField(default=0)  # in tiyin


class Payment(models.Model):
    user = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.PROTECT)
    order_id = models.PositiveBigIntegerField(unique=True)  # second line of defense
    amount = models.PositiveBigIntegerField()  # in tiyin
    created_at = models.DateTimeField(auto_now_add=True)

Note the unique=True on Payment.order_id. If your domain has a natural key (an order is paid at most once), protect it with a constraint too. The idempotency key is the general mechanism; the natural unique constraint is the second line of defense.

Now the core: claiming and checking the key.

# payments/idempotency.py
import hashlib
import json
from datetime import timedelta

from django.db import IntegrityError, OperationalError, transaction
from django.utils import timezone

from .models import IdempotencyKey

LOCK_TIMEOUT = timedelta(seconds=60)  # must be longer than your worker timeout


class IdempotencyConflict(Exception):
    """The same key is being processed right now."""


class IdempotencyMismatch(Exception):
    """The same key was used with a different request."""


def request_fingerprint(method: str, path: str, data: dict) -> str:
    canonical = json.dumps(data, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
    return hashlib.sha256(f"{method}:{path}:{canonical}".encode()).hexdigest()


def acquire(user, key: str, fingerprint: str) -> tuple[IdempotencyKey, bool]:
    """Return (record, replay). replay=True means a stored response exists."""
    now = timezone.now()

    # Step 1: try to claim the key. Short transaction, committed immediately.
    try:
        with transaction.atomic():
            record = IdempotencyKey.objects.create(
                user=user, key=key, request_hash=fingerprint, locked_at=now
            )
        return record, False
    except IntegrityError:
        pass  # the key already exists (committed by someone else)

    # Step 2: the key exists. Inspect it without waiting on a busy row.
    try:
        with transaction.atomic():
            record = IdempotencyKey.objects.select_for_update(nowait=True).get(
                user=user, key=key
            )
            if record.request_hash != fingerprint:
                raise IdempotencyMismatch()
            if record.status == IdempotencyKey.Status.COMPLETED:
                return record, True
            if record.locked_at and now - record.locked_at < LOCK_TIMEOUT:
                raise IdempotencyConflict()
            # Stale or released lock: take it over.
            record.locked_at = now
            record.save(update_fields=["locked_at"])
            return record, False
    except OperationalError:
        # nowait=True: another transaction holds the row lock right now
        raise IdempotencyConflict()

A few details:

  • The hash is computed over canonical JSON (sort_keys=True), so {"a":1,"b":2} and {"b":2,"a":1} hash the same. Method and path are included, so reusing a key on a different endpoint is also an error.
  • nowait=True. If another transaction holds the row right now (meaning the first request is debiting the wallet at this very moment), we don't wait and return 409 immediately. Without it, the second request would tie up a worker and a connection while waiting for the first.
  • locked_at covers the case where a process died. If a key has been started for longer than LOCK_TIMEOUT, we assume its owner abandoned it and take it over.

The debit logic is simple, locking the wallet row with select_for_update:

# payments/services.py
from django.db.models import F

from .models import Payment, Wallet


def charge_wallet(user, amount: int, order_id: int) -> tuple[int, dict]:
    """Must be called inside transaction.atomic()."""
    wallet = Wallet.objects.select_for_update().get(user=user)
    if wallet.balance < amount:
        return 402, {"error": "insufficient_funds"}

    wallet.balance = F("balance") - amount
    wallet.save(update_fields=["balance"])
    payment = Payment.objects.create(user=user, order_id=order_id, amount=amount)
    return 201, {"payment_id": payment.pk, "order_id": order_id, "amount": amount}

Notice that "insufficient funds" is returned as a normal response, not raised. It's a deterministic business outcome and should be stored too: a retry with the same key must get the same 402 again.

And the view:

# payments/views.py
import json

from django.contrib.auth.decorators import login_required
from django.db import transaction
from django.http import JsonResponse
from django.views.decorators.http import require_POST

from .idempotency import (
    IdempotencyConflict, IdempotencyMismatch, acquire, request_fingerprint,
)
from .models import IdempotencyKey
from .services import charge_wallet


def replayed(record: IdempotencyKey) -> JsonResponse:
    response = JsonResponse(record.response_body, status=record.response_code)
    response["Idempotent-Replayed"] = "true"
    return response


@transaction.non_atomic_requests
@login_required
@require_POST
def create_payment(request):
    key = request.headers.get("Idempotency-Key", "")
    if not 1 <= len(key) <= 255:
        return JsonResponse({"error": "idempotency_key_required"}, status=400)

    try:
        data = json.loads(request.body)
        amount, order_id = int(data["amount"]), int(data["order_id"])
    except (ValueError, KeyError, TypeError):
        return JsonResponse({"error": "invalid_request"}, status=400)

    fingerprint = request_fingerprint(request.method, request.path, data)
    try:
        record, replay = acquire(request.user, key, fingerprint)
    except IdempotencyMismatch:
        return JsonResponse({"error": "idempotency_key_reused"}, status=422)
    except IdempotencyConflict:
        return JsonResponse({"error": "request_in_progress"}, status=409)

    if replay:
        return replayed(record)

    try:
        with transaction.atomic():
            # Holding this row lock until COMMIT prevents a double run
            # even if a slow request's lock was taken over as "stale".
            locked = IdempotencyKey.objects.select_for_update().get(pk=record.pk)
            if locked.status == IdempotencyKey.Status.COMPLETED:
                return replayed(locked)

            code, body = charge_wallet(request.user, amount, order_id)

            locked.status = IdempotencyKey.Status.COMPLETED
            locked.response_code = code
            locked.response_body = body
            locked.save(update_fields=["status", "response_code", "response_body"])
    except Exception:
        # Everything above was rolled back, so a retry is safe: release the key.
        IdempotencyKey.objects.filter(
            pk=record.pk, status=IdempotencyKey.Status.STARTED
        ).update(locked_at=None)
        raise

    return JsonResponse(body, status=code)

The key line here is the select_for_update() in the inner transaction. Suppose the first request is very slow, LOCK_TIMEOUT passes, and a second request takes the key over as "stale". The second request gets through acquire, but in the inner transaction it hits the row lock held by the first and waits. Once the first commits, the second sees completed and simply replays the stored response. No double debit.

Should you store 500s? Comparing with Stripe

Stripe's API is the usual reference here. According to their documentation, Stripe saves the status code and body of the first request, including 500 errors, and returns exactly that for every later request with the same key. Keys are pruned after they're at least 24 hours old. Reusing a key with different parameters returns an error. Requests that fail validation or conflict with a concurrent request don't get a saved result, because the endpoint never started executing, so they can be retried.

Why does Stripe store even 500s? Because internally an operation spans many systems, and when a 500 happens some part of it may already have run. Allowing a retry would be dangerous.

Our case is different: the debit and the stored response live in one PostgreSQL transaction. If anything raises, everything rolls back, nothing changed, and a retry is safe. That's why the except block releases the key (locked_at=None).

Stripe Our implementation
Key scope per account (user, key)
Retention at least 24 hours configurable, 24 hours by default
Reuse with different parameters error 422
Concurrent request conflict error, nothing saved 409, nothing saved
Validation error not saved not saved (400, key never claimed)
500 error saved and replayed not saved, rolled back, retry allowed

On status codes: there is an IETF draft standardizing the Idempotency-Key header (in the httpapi working group), and it suggests similar codes: 400 when the header is missing, 409 when the original request is still in progress, 422 when a key is reused with a different payload. The Idempotent-Replayed: true header is optional, but it's very handy when debugging: the client can see the response is a replay.

When an external payment provider is involved

The scheme above works because all changes live in one database. A real payment often involves an HTTP call to an external provider, and that can't go inside a transaction: if the provider takes 20 seconds to respond, you hold locks and a connection for 20 seconds, and a rollback won't undo money the provider already moved.

Here the operation has to be split into separately committed steps (Brandur Leach's post on Stripe-like idempotency keys in Postgres calls these "atomic phases" and "recovery points"):

  1. In one transaction, create Payment(status="pending") with a pre-generated unique provider_ref.
  2. Outside any transaction, call the provider and pass that provider_ref. Many providers accept their own idempotency key or a unique merchant-side transaction ID.
  3. Once the response is in, mark the Payment and the idempotency key completed in a separate transaction.

If the process dies between steps 2 and 3, the request that takes over the stale key must not blindly call the provider again. It first checks the payment's status by provider_ref, and only then continues.

Key expiry and cleanup

You don't need to keep keys forever, but retention must be longer than the longest window in which a client might retry. A mobile app may send a request from an offline queue hours later: if the key is deleted after 10 minutes, that request runs as a brand-new payment.

Cleanup is a periodic Celery beat task, deleting in batches so a huge DELETE doesn't hold the table for long:

# payments/tasks.py
from datetime import timedelta

from celery import shared_task
from django.utils import timezone

from .models import IdempotencyKey

RETENTION = timedelta(hours=24)


@shared_task
def purge_idempotency_keys(batch_size: int = 5000) -> int:
    cutoff = timezone.now() - RETENTION
    total = 0
    while True:
        ids = list(
            IdempotencyKey.objects.filter(created_at__lt=cutoff)
            .values_list("pk", flat=True)[:batch_size]
        )
        if not ids:
            return total
        deleted, _ = IdempotencyKey.objects.filter(pk__in=ids).delete()
        total += deleted

The index on created_at exists for exactly this query.

The client side

No matter how carefully the server is written, a client that generates a new key per retry defeats it. A correct client looks like this:

import random
import time
import uuid

import httpx

RETRYABLE = {409, 500, 502, 503, 504}


def pay(order_id: int, amount: int, token: str) -> httpx.Response:
    key = str(uuid.uuid4())  # one key per user intent, NOT per attempt
    for attempt in range(5):
        try:
            response = httpx.post(
                "https://api.example.com/v1/payments/",
                json={"order_id": order_id, "amount": amount},
                headers={"Authorization": f"Bearer {token}", "Idempotency-Key": key},
                timeout=10,
            )
        except httpx.TransportError:
            pass  # timeout, reset connection: the outcome is unknown, retry
        else:
            if response.status_code not in RETRYABLE:
                return response
        time.sleep(min(2 ** attempt, 30) + random.random())  # backoff + jitter
    raise RuntimeError(f"payment outcome unknown, retry later with key {key}")

Three rules: the key is created outside the loop, 409 and 5xx mean "try again", and other 4xx (like 402 or 422) are final. Jitter (random.random()) keeps many clients from retrying in lockstep and knocking the server over a second time.

Production pitfalls

  • ATOMIC_REQUESTS = True. This wraps the whole view in one transaction, so the claimed key isn't committed until the view returns and other requests can't see it. That's why the view has @transaction.non_atomic_requests, and it must be the outermost decorator.
  • Key scope. If the unique constraint is on key alone, one user's key can collide with another's, and in the worst case someone else's stored response is returned to the wrong person. Always (user, key) or (merchant, key).
  • LOCK_TIMEOUT shorter than the worker timeout. Say gunicorn's timeout is 120 seconds and LOCK_TIMEOUT is 60: a live request gets treated as stale. The inner select_for_update still prevents a double debit, but don't rely on that; align the timeouts.
  • Secrets in the response body. The stored response sits in the database for 24 hours. No card numbers, tokens or personal data in it.
  • Volatile fields in the hash. If the client puts a timestamp or request_id in the body, the hash changes on every retry and the client gets 422. Hash only fields with business meaning, or require clients to resend the body byte for byte.
  • Retries after cleanup. A retry that arrives after the key was purged runs as a new request. The unique constraint on Payment.order_id is the second line of defense for exactly that case.
  • Redis isn't enough. SET key value NX EX 86400 is fine as a lock, but Redis and PostgreSQL don't share a transaction: the money moved but the response never made it into Redis, or eviction or a failover dropped the key. When money is involved, keep the key and the result in the same database.
  • Celery tasks can run twice too. With acks_late, a task is redelivered if the worker dies. Same idea applies: give the task an ID and guard its result with a unique constraint.

Checklist

  • Every POST endpoint that moves money requires an Idempotency-Key.
  • The client creates the key once per intent and reuses it on every retry.
  • The key is protected by a unique constraint on (user, key) and claimed with an INSERT, not a check.
  • A canonical hash of the request is stored; a key reused with a different payload gets 422.
  • The business change and the stored response are in one transaction.
  • In-progress requests get 409; completed ones get the stored status and body.
  • With ATOMIC_REQUESTS on, the view is marked non_atomic_requests.
  • Stale locks have a timeout, and it's longer than the worker timeout.
  • External providers get your own unique reference; recovery checks status first.
  • Keys are kept for at least 24 hours and purged in batches.

Takeaway

The network will lose responses, so a payment API has to survive retries. Idempotency keys make that simple: the client gives each intent one key, the server claims it through a unique constraint and stores the response in the same transaction as the business change. The rest is detail: the hash, 409/422, expiry, and recovery for external providers. Get them right once and "why was I charged twice" stops being your problem.

// comments

Loading comments...

    // next step

    Let's build something together.

    Message me on Telegram
    aslbek@asldev: ~
    ❯