All notes
Programming

When a Celery task runs twice: acks_late, retries and idempotent tasks

A worker died mid-deploy and the customer got the cashback twice. When Celery sends the ack, how retries work, and what it takes for a task to survive being run again.

Picture this: a user pays for an order and the view hands Celery a task: "credit the cashback and send an SMS". Meanwhile a deploy is rolling out. Kubernetes sends the old worker pod SIGTERM, waits 30 seconds, then sends SIGKILL. The task dies halfway through. Two outcomes are possible from here and both are bad: the task vanishes (the customer never gets the cashback), or it runs again and the cashback lands twice.

Which one you get depends on your Celery settings. This post covers when Celery considers a task "done", how retries work, and how to make a task harmless to run more than once (idempotent). The previous post covered idempotency keys for HTTP requests; here we apply the same idea to background jobs.

The naive code and its four bugs

In plenty of projects the first version looks roughly like this:

# orders/tasks.py — the naive version
from decimal import Decimal

from celery import shared_task

from notifications.sms import send_sms


@shared_task
def credit_cashback(order):
    user = order.user
    user.balance += order.total * Decimal("0.01")
    user.save()
    send_sms(user.phone, f"Cashback for order #{order.pk} credited")
# orders/views.py — the naive call site
from django.db import transaction
from django.http import JsonResponse

from .models import Order
from .tasks import credit_cashback


def pay_order(request, order_id):
    with transaction.atomic():
        order = Order.objects.select_for_update().get(pk=order_id, user=request.user)
        order.status = Order.Status.PAID
        order.save(update_fields=["status"])
        credit_cashback.delay(order)  # sent BEFORE commit, with a whole object
    return JsonResponse({"ok": True})

It looks fine. It actually has four problems:

  1. The task is sent before the commit. A worker can pick the message up before the transaction commits and read the order's old state. If the transaction rolls back, cashback gets credited for an order that was never paid.
  2. A whole object is passed as the argument. The JSON serializer cannot serialize a model instance at all; with pickle, the worker gets a copy that is seconds or minutes stale.
  3. balance += ... is a read-modify-write. Two concurrent tasks will lose one of the writes.
  4. The task isn't idempotent. Run it again and the money is added again and the SMS goes out again.

You only see why the fourth one, the important one, can't be avoided once you look at Celery's delivery guarantees.

When the ack is sent: at-most-once vs at-least-once

The broker (Redis or RabbitMQ) delivers a message to a worker, and the worker has to acknowledge it, sending an ack that says "got it, you can delete it". What matters is when that ack is sent.

By default, Celery acks the message before running the task. If the worker dies mid-task, the message is already gone from the broker and the task is lost. That is at-most-once: one time at most, sometimes zero.

With acks_late=True the ack is sent after the task finishes. If the worker loses its broker connection or the machine goes down, the unacked message is redelivered. That is at-least-once: one time at least, sometimes two or more.

There's a subtlety, though. In Celery the task runs in a child process of the pool, not in the main worker process. If that child process is killed (by the OOM killer or SIGKILL), the main process acks the message anyway, even with acks_late, and marks the task as FAILURE. To get the message requeued you also need task_reject_on_worker_lost=True.

Setting Ack sent Pool process killed Whole worker / connection lost Guarantee
Default Before execution Task lost Task lost at-most-once
acks_late=True After execution Acked, task lost Redelivered partly at-least-once
acks_late + reject_on_worker_lost After execution Requeued Redelivered at-least-once

Note that even with acks_late, a task that raises an ordinary exception still gets acked (task_acks_on_failure_or_timeout=True by default). Trying again is the job of retries, not of acks.

There is no "exactly-once". You choose between losing a task and repeating it. When money is involved you almost always choose repeating, and make the task survive it.

For anything that touches money, this is the configuration I start from:

# config/celery.py
import os

from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")

app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
# config/settings.py
CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json"]

CELERY_TASK_ACKS_LATE = True              # ack after the task finishes
CELERY_TASK_REJECT_ON_WORKER_LOST = True  # requeue if the pool process dies
CELERY_WORKER_PREFETCH_MULTIPLIER = 1     # do not hoard unacked messages

# Redis: must exceed your longest task AND your longest countdown/ETA
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 6 * 60 * 60}

Why worker_prefetch_multiplier = 1? Workers fetch messages ahead of time (prefetch), and with acks_late those sit unacked. With long tasks this means one worker hoards a pile of messages, and they all get redelivered if it dies.

Retries: exponential backoff, jitter and max_retries

When an external API is temporarily down, retrying right away is a bad idea: hundreds of workers pile onto a service that is already struggling. The right move is to double the wait on each attempt (exponential backoff) and add randomness (jitter) so the retries spread out over time.

# notifications/tasks.py
import requests
from celery import shared_task

from .models import Notification

SMS_URL = "https://sms.example.com/v1/messages"


@shared_task(
    bind=True,
    autoretry_for=(requests.ConnectionError, requests.Timeout),
    retry_backoff=True,      # 1, 2, 4, 8 ... seconds
    retry_backoff_max=600,   # never wait more than 10 minutes
    retry_jitter=True,       # random delay in [0, backoff]
    max_retries=8,
)
def send_sms_notification(self, notification_id: int) -> None:
    n = Notification.objects.get(pk=notification_id)
    if n.status == Notification.Status.SENT:
        return  # a previous delivery already did the job

    resp = requests.post(
        SMS_URL,
        json={"phone": n.phone, "text": n.text},
        headers={"Idempotency-Key": str(n.uuid)},
        timeout=(3, 10),
    )
    if resp.status_code == 429 or resp.status_code >= 500:
        # temporary problem on their side: retry manually with a delay
        raise self.retry(countdown=min(60 * 2 ** self.request.retries, 900))
    resp.raise_for_status()  # other 4xx: a bug in our request, retrying won't help

    Notification.objects.filter(pk=n.pk).update(status=Notification.Status.SENT)

What's going on here:

  • autoretry_for retries automatically only for the listed exceptions. Don't write autoretry_for=(Exception,): retrying a DoesNotExist or a KeyError eight times achieves nothing.
  • retry_backoff=True computes the delay as 1 * 2 ** retries seconds, capped by retry_backoff_max (600 by default). retry_jitter=True (on by default) then picks the actual delay at random between 0 and that value.
  • max_retries defaults to 3; None means retry forever. Once the limit is reached, self.retry(exc=exc) re-raises the original exception, or raises MaxRetriesExceededError if no exc was given.
  • self.retry() raises, which is why the convention is raise self.retry(...): it makes obvious that nothing below it runs.
  • timeout=(3, 10) is mandatory. Without a timeout, requests can hang forever and the retry never starts.

Keep in mind that a retry is the same task sent again as a new message with a countdown. So retries are part of the at-least-once machinery too, and the task still has to cope with running twice. That's what the Idempotency-Key header is for: if the SMS provider accepted the request but the response never reached us, the second request doesn't become a second SMS. That only works if the provider supports the header.

Making the task idempotent

An idempotent task produces the same result whether it runs once or five times. In the database, the most reliable way to get there is to do the "claim" in one atomic UPDATE:

# orders/tasks.py — the fixed version
from decimal import ROUND_HALF_UP, Decimal

from celery import shared_task
from django.db import transaction
from django.db.models import F

from wallets.models import Wallet

from .models import Order


@shared_task(acks_late=True)
def credit_cashback(order_id: int) -> None:
    with transaction.atomic():
        # Atomic "claim": only one execution can flip the flag.
        claimed = Order.objects.filter(
            pk=order_id,
            status=Order.Status.PAID,
            cashback_credited=False,
        ).update(cashback_credited=True)
        if not claimed:
            return  # already credited, or the order is not paid

        order = Order.objects.only("total", "user_id").get(pk=order_id)
        amount = (order.total * Decimal("0.01")).quantize(
            Decimal("0.01"), rounding=ROUND_HALF_UP
        )
        Wallet.objects.filter(user_id=order.user_id).update(
            balance=F("balance") + amount
        )

Here the check and the write are not separate steps. If you write if order.cashback_credited: return followed by save(), two concurrent tasks both see False and both add money. A conditional UPDATE behaves differently in PostgreSQL:

-- What the "claim" compiles to (simplified)
UPDATE orders_order
   SET cashback_credited = true
 WHERE id = 42
   AND status = 'paid'
   AND cashback_credited = false;
-- First transaction:  UPDATE 1
-- Second, concurrent: waits for the row lock, re-checks WHERE
--                     after the first commits -> UPDATE 0

Under READ COMMITTED the second transaction waits for the row lock held by the first, re-evaluates the WHERE clause against the new row version after the commit, and updates nothing. .update() returns the number of rows updated, which is what we check. The balance is updated with F("balance") + amount as well: an atomic addition inside the database, not a read-then-write in Python.

An even stronger option is to record every operation as its own ledger row and let the database reject duplicates:

# wallets/models.py — a ledger row makes duplicates impossible at the DB level
from django.db import models


class CashbackEntry(models.Model):
    order = models.ForeignKey("orders.Order", on_delete=models.PROTECT)
    amount = models.DecimalField(max_digits=14, decimal_places=2)
    created_at = models.DateTimeField(auto_now_add=True)

    class Meta:
        constraints = [
            models.UniqueConstraint(fields=["order"], name="uniq_cashback_per_order"),
        ]

A second CashbackEntry.objects.create(order_id=...) raises IntegrityError and the transaction rolls back. No matter what the code gets wrong, the constraint won't let the second row in.

External side effects (SMS, email, a payment through a provider) are different: you can't put them inside a database transaction. You have two tools here. One is an idempotency key, if the provider supports it. The other is recording the state (pending → sent) and accepting the occasional duplicate. That's also why I pull the SMS out of the money-moving task into its own task: a retry of one must not re-run the other.

Pass an id, not an object

Use credit_cashback.delay(order.pk) instead of credit_cashback.delay(order). Here's why:

  • JSON (Celery's default serializer) can't serialize a model instance. Turning on pickle is a security problem, because it opens the door to code execution through a message.
  • The task runs seconds later, or minutes later with retries. The row may have changed by then. With an id, the worker reads the current state from the database.
  • The message stays small and is easy to read in the broker.

For the same reason, be careful about passing values like amounts: let the task read them from the database. Otherwise a stale value lives on "forever" through the retries.

transaction.on_commit: send the task after the commit

The sneakiest bug in the naive version is calling delay() inside atomic(). The message hits Redis immediately, and the commit happens later. A fast worker sees an order that isn't PAID yet, and if the transaction rolls back, the task runs for a payment that doesn't exist.

# orders/views.py — the fixed call site
from functools import partial

from django.db import transaction
from django.http import JsonResponse

from .models import Order
from .tasks import credit_cashback


def pay_order(request, order_id):
    with transaction.atomic():
        order = Order.objects.select_for_update().get(pk=order_id, user=request.user)
        order.status = Order.Status.PAID
        order.save(update_fields=["status"])
        transaction.on_commit(partial(credit_cashback.delay, order.pk))
        # Celery 5.4+ with Django: credit_cashback.delay_on_commit(order.pk)
    return JsonResponse({"ok": True})

transaction.on_commit() runs the callback after the outermost transaction commits successfully, and never runs it on rollback. Since Celery 5.4, when used with Django, the default task class is DjangoTask, which has delay_on_commit() and apply_async_on_commit() methods that do exactly this.

In tests, TestCase wraps each test in a transaction that never commits, so on_commit callbacks don't fire on their own. Run them with captureOnCommitCallbacks:

# orders/tests.py
from unittest import mock

from django.test import TestCase

from .models import Order


class PayOrderTests(TestCase):
    def test_task_is_sent_after_commit(self):
        order = Order.objects.create(...)  # fixture of your choice
        with mock.patch("orders.views.credit_cashback.delay") as delay:
            with self.captureOnCommitCallbacks(execute=True):
                self.client.post(f"/orders/{order.pk}/pay/")
        delay.assert_called_once_with(order.pk)

on_commit isn't a complete guarantee either. The commit succeeds, then the callback can't reach Redis or the process dies at that exact second, and the task is lost. For money, a simple remedy is a periodic "sweeper" task (via Celery beat) that finds unfinished work and re-sends it. That's safe precisely because the task is idempotent:

# orders/tasks.py — a safety net for messages lost between commit and broker
from datetime import timedelta

from celery import shared_task
from django.utils import timezone

from .models import Order


@shared_task
def resend_missing_cashback() -> int:
    cutoff = timezone.now() - timedelta(minutes=10)
    ids = list(
        Order.objects.filter(
            status=Order.Status.PAID,
            cashback_credited=False,
            paid_at__lt=cutoff,
        ).values_list("pk", flat=True)[:500]
    )
    for pk in ids:
        credit_cashback.delay(pk)  # safe: the task is idempotent
    return len(ids)

In larger systems the general form of this is the outbox pattern: the message is written to a table in the same transaction as the business data, and a separate process forwards it to the broker.

The visibility timeout trap

Redis has nothing like RabbitMQ's "connection dropped, so return the unacked messages". So Celery's Redis transport keeps delivered messages in a separate unacked structure and puts a message back in the queue if no ack arrives within the visibility timeout (1 hour by default). That rescues the message when a worker dies, but it has two surprising side effects:

  1. Long tasks. If an acks_late task runs longer than the visibility timeout, the message is handed to another worker too, and the task runs twice even though nothing crashed.
  2. countdown and ETA. Messages sent with apply_async(countdown=...), eta=... or self.retry(countdown=...) are taken by a worker immediately and held in memory until they're due. If the delay is longer than the visibility timeout, the message is redelivered and each new worker starts holding it as well, so a single task ends up running in several copies.

The fix: set visibility_timeout higher than both your longest task and your largest countdown (6 hours in the settings above). For work that should happen a day from now, don't use a countdown. Use a run_at column in the database and a periodic task. With retries, retry_backoff_max caps the delay, which matters here too.

RabbitMQ has no visibility timeout, but it has a similar limit: consumer_timeout, how long the broker waits for an ack. If it's exceeded, the broker closes the channel and the message is redelivered. Check it if you have long tasks.

You can inspect the state in Redis:

# Messages taken by workers but not yet acked (Redis transport)
redis-cli HLEN unacked
# Messages waiting in the default queue
redis-cli LLEN celery

Pitfalls in production

  • Not enough time for a graceful shutdown. After SIGTERM, Celery stops taking new tasks and tries to finish the running ones. In Kubernetes, terminationGracePeriodSeconds defaults to 30 seconds, and if a task takes longer, the pod gets SIGKILL. Set it above your longest task, or split tasks into short chunks.
  • Poison messages. With reject_on_worker_lost=True, a task that OOM-kills the worker every time gets requeued and loops forever. Watch memory and set a time_limit.
  • Half-done work on retry. If a task writes to the database, then calls an API and fails, the retry redoes the first part too. Either do it all in one transaction or make each step idempotent on its own.
  • Eager-mode tests. With task_always_eager=True, tests skip the broker, serialization and acks. Code that passes an object as an argument passes the tests and fails in production.
  • Waiting on results. Don't call result.get() inside another task: it makes workers wait on each other and can deadlock.

Checklist

  • Tasks that move money or change state: acks_late=True and task_reject_on_worker_lost=True.
  • Every task survives a repeat: conditional UPDATE, UniqueConstraint, or a status field.
  • Idempotency keys for external APIs, and always a timeout.
  • autoretry_for only for transient errors, with retry_backoff, retry_jitter and an explicit max_retries.
  • Only ids and plain values as arguments, JSON serializer.
  • delay() only through transaction.on_commit() or delay_on_commit().
  • A periodic check task for lost messages.
  • visibility_timeout > longest task and largest countdown.
  • Shutdown grace period > longest task.

Takeaway

Celery won't give you "exactly once". By default a task can be lost, and with acks_late it can be repeated. In a system that moves money, the right choice is at-least-once delivery plus an idempotent task. Send tasks after the commit, pass ids, let the database reject duplicates, and account for the visibility timeout. Then a SIGKILL during a deploy doesn't turn into double cashback in a customer's account.

// comments

Loading comments...

    // next step

    Let's build something together.

    Message me on Telegram
    aslbek@asldev: ~
    ❯