When a Celery task runs twice: acks_late, retries and idempotent tasks
A worker died mid-deploy and the customer got the cashback twice. When Celery sends the ack, how retries work, and what it takes for a task to survive being run again.
Picture this: a user pays for an order and the view hands Celery a task: "credit the cashback and send an SMS". Meanwhile a deploy is rolling out. Kubernetes sends the old worker pod SIGTERM, waits 30 seconds, then sends SIGKILL. The task dies halfway through. Two outcomes are possible from here and both are bad: the task vanishes (the customer never gets the cashback), or it runs again and the cashback lands twice.
Which one you get depends on your Celery settings. This post covers when Celery considers a task "done", how retries work, and how to make a task harmless to run more than once (idempotent). The previous post covered idempotency keys for HTTP requests; here we apply the same idea to background jobs.
The naive code and its four bugs
In plenty of projects the first version looks roughly like this:
# orders/tasks.py — the naive version
from decimal import Decimal
from celery import shared_task
from notifications.sms import send_sms
@shared_task
def credit_cashback(order):
user = order.user
user.balance += order.total * Decimal("0.01")
user.save()
send_sms(user.phone, f"Cashback for order #{order.pk} credited")
# orders/views.py — the naive call site
from django.db import transaction
from django.http import JsonResponse
from .models import Order
from .tasks import credit_cashback
def pay_order(request, order_id):
with transaction.atomic():
order = Order.objects.select_for_update().get(pk=order_id, user=request.user)
order.status = Order.Status.PAID
order.save(update_fields=["status"])
credit_cashback.delay(order) # sent BEFORE commit, with a whole object
return JsonResponse({"ok": True})
It looks fine. It actually has four problems:
- The task is sent before the commit. A worker can pick the message up before the transaction commits and read the order's old state. If the transaction rolls back, cashback gets credited for an order that was never paid.
- A whole object is passed as the argument. The JSON serializer cannot serialize a model instance at all; with pickle, the worker gets a copy that is seconds or minutes stale.
balance += ...is a read-modify-write. Two concurrent tasks will lose one of the writes.- The task isn't idempotent. Run it again and the money is added again and the SMS goes out again.
You only see why the fourth one, the important one, can't be avoided once you look at Celery's delivery guarantees.
When the ack is sent: at-most-once vs at-least-once
The broker (Redis or RabbitMQ) delivers a message to a worker, and the worker has to acknowledge it, sending an ack that says "got it, you can delete it". What matters is when that ack is sent.
By default, Celery acks the message before running the task. If the worker dies mid-task, the message is already gone from the broker and the task is lost. That is at-most-once: one time at most, sometimes zero.
With acks_late=True the ack is sent after the task finishes. If the worker loses its broker connection or the machine goes down, the unacked message is redelivered. That is at-least-once: one time at least, sometimes two or more.
There's a subtlety, though. In Celery the task runs in a child process of the pool, not in the main worker process. If that child process is killed (by the OOM killer or SIGKILL), the main process acks the message anyway, even with acks_late, and marks the task as FAILURE. To get the message requeued you also need task_reject_on_worker_lost=True.
| Setting | Ack sent | Pool process killed | Whole worker / connection lost | Guarantee |
|---|---|---|---|---|
| Default | Before execution | Task lost | Task lost | at-most-once |
acks_late=True |
After execution | Acked, task lost | Redelivered | partly at-least-once |
acks_late + reject_on_worker_lost |
After execution | Requeued | Redelivered | at-least-once |
Note that even with acks_late, a task that raises an ordinary exception still gets acked (task_acks_on_failure_or_timeout=True by default). Trying again is the job of retries, not of acks.
There is no "exactly-once". You choose between losing a task and repeating it. When money is involved you almost always choose repeating, and make the task survive it.
For anything that touches money, this is the configuration I start from:
# config/celery.py
import os
from celery import Celery
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "config.settings")
app = Celery("config")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()
# config/settings.py
CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_TASK_SERIALIZER = "json"
CELERY_ACCEPT_CONTENT = ["json"]
CELERY_TASK_ACKS_LATE = True # ack after the task finishes
CELERY_TASK_REJECT_ON_WORKER_LOST = True # requeue if the pool process dies
CELERY_WORKER_PREFETCH_MULTIPLIER = 1 # do not hoard unacked messages
# Redis: must exceed your longest task AND your longest countdown/ETA
CELERY_BROKER_TRANSPORT_OPTIONS = {"visibility_timeout": 6 * 60 * 60}
Why worker_prefetch_multiplier = 1? Workers fetch messages ahead of time (prefetch), and with acks_late those sit unacked. With long tasks this means one worker hoards a pile of messages, and they all get redelivered if it dies.
Retries: exponential backoff, jitter and max_retries
When an external API is temporarily down, retrying right away is a bad idea: hundreds of workers pile onto a service that is already struggling. The right move is to double the wait on each attempt (exponential backoff) and add randomness (jitter) so the retries spread out over time.
# notifications/tasks.py
import requests
from celery import shared_task
from .models import Notification
SMS_URL = "https://sms.example.com/v1/messages"
@shared_task(
bind=True,
autoretry_for=(requests.ConnectionError, requests.Timeout),
retry_backoff=True, # 1, 2, 4, 8 ... seconds
retry_backoff_max=600, # never wait more than 10 minutes
retry_jitter=True, # random delay in [0, backoff]
max_retries=8,
)
def send_sms_notification(self, notification_id: int) -> None:
n = Notification.objects.get(pk=notification_id)
if n.status == Notification.Status.SENT:
return # a previous delivery already did the job
resp = requests.post(
SMS_URL,
json={"phone": n.phone, "text": n.text},
headers={"Idempotency-Key": str(n.uuid)},
timeout=(3, 10),
)
if resp.status_code == 429 or resp.status_code >= 500:
# temporary problem on their side: retry manually with a delay
raise self.retry(countdown=min(60 * 2 ** self.request.retries, 900))
resp.raise_for_status() # other 4xx: a bug in our request, retrying won't help
Notification.objects.filter(pk=n.pk).update(status=Notification.Status.SENT)
What's going on here:
autoretry_forretries automatically only for the listed exceptions. Don't writeautoretry_for=(Exception,): retrying aDoesNotExistor aKeyErroreight times achieves nothing.retry_backoff=Truecomputes the delay as1 * 2 ** retriesseconds, capped byretry_backoff_max(600 by default).retry_jitter=True(on by default) then picks the actual delay at random between 0 and that value.max_retriesdefaults to 3;Nonemeans retry forever. Once the limit is reached,self.retry(exc=exc)re-raises the original exception, or raisesMaxRetriesExceededErrorif noexcwas given.self.retry()raises, which is why the convention israise self.retry(...): it makes obvious that nothing below it runs.timeout=(3, 10)is mandatory. Without a timeout,requestscan hang forever and the retry never starts.
Keep in mind that a retry is the same task sent again as a new message with a countdown. So retries are part of the at-least-once machinery too, and the task still has to cope with running twice. That's what the Idempotency-Key header is for: if the SMS provider accepted the request but the response never reached us, the second request doesn't become a second SMS. That only works if the provider supports the header.
Making the task idempotent
An idempotent task produces the same result whether it runs once or five times. In the database, the most reliable way to get there is to do the "claim" in one atomic UPDATE:
# orders/tasks.py — the fixed version
from decimal import ROUND_HALF_UP, Decimal
from celery import shared_task
from django.db import transaction
from django.db.models import F
from wallets.models import Wallet
from .models import Order
@shared_task(acks_late=True)
def credit_cashback(order_id: int) -> None:
with transaction.atomic():
# Atomic "claim": only one execution can flip the flag.
claimed = Order.objects.filter(
pk=order_id,
status=Order.Status.PAID,
cashback_credited=False,
).update(cashback_credited=True)
if not claimed:
return # already credited, or the order is not paid
order = Order.objects.only("total", "user_id").get(pk=order_id)
amount = (order.total * Decimal("0.01")).quantize(
Decimal("0.01"), rounding=ROUND_HALF_UP
)
Wallet.objects.filter(user_id=order.user_id).update(
balance=F("balance") + amount
)
Here the check and the write are not separate steps. If you write if order.cashback_credited: return followed by save(), two concurrent tasks both see False and both add money. A conditional UPDATE behaves differently in PostgreSQL:
-- What the "claim" compiles to (simplified)
UPDATE orders_order
SET cashback_credited = true
WHERE id = 42
AND status = 'paid'
AND cashback_credited = false;
-- First transaction: UPDATE 1
-- Second, concurrent: waits for the row lock, re-checks WHERE
-- after the first commits -> UPDATE 0
Under READ COMMITTED the second transaction waits for the row lock held by the first, re-evaluates the WHERE clause against the new row version after the commit, and updates nothing. .update() returns the number of rows updated, which is what we check. The balance is updated with F("balance") + amount as well: an atomic addition inside the database, not a read-then-write in Python.
An even stronger option is to record every operation as its own ledger row and let the database reject duplicates:
# wallets/models.py — a ledger row makes duplicates impossible at the DB level
from django.db import models
class CashbackEntry(models.Model):
order = models.ForeignKey("orders.Order", on_delete=models.PROTECT)
amount = models.DecimalField(max_digits=14, decimal_places=2)
created_at = models.DateTimeField(auto_now_add=True)
class Meta:
constraints = [
models.UniqueConstraint(fields=["order"], name="uniq_cashback_per_order"),
]
A second CashbackEntry.objects.create(order_id=...) raises IntegrityError and the transaction rolls back. No matter what the code gets wrong, the constraint won't let the second row in.
External side effects (SMS, email, a payment through a provider) are different: you can't put them inside a database transaction. You have two tools here. One is an idempotency key, if the provider supports it. The other is recording the state (pending → sent) and accepting the occasional duplicate. That's also why I pull the SMS out of the money-moving task into its own task: a retry of one must not re-run the other.
Pass an id, not an object
Use credit_cashback.delay(order.pk) instead of credit_cashback.delay(order). Here's why:
- JSON (Celery's default serializer) can't serialize a model instance. Turning on pickle is a security problem, because it opens the door to code execution through a message.
- The task runs seconds later, or minutes later with retries. The row may have changed by then. With an id, the worker reads the current state from the database.
- The message stays small and is easy to read in the broker.
For the same reason, be careful about passing values like amounts: let the task read them from the database. Otherwise a stale value lives on "forever" through the retries.
transaction.on_commit: send the task after the commit
The sneakiest bug in the naive version is calling delay() inside atomic(). The message hits Redis immediately, and the commit happens later. A fast worker sees an order that isn't PAID yet, and if the transaction rolls back, the task runs for a payment that doesn't exist.
# orders/views.py — the fixed call site
from functools import partial
from django.db import transaction
from django.http import JsonResponse
from .models import Order
from .tasks import credit_cashback
def pay_order(request, order_id):
with transaction.atomic():
order = Order.objects.select_for_update().get(pk=order_id, user=request.user)
order.status = Order.Status.PAID
order.save(update_fields=["status"])
transaction.on_commit(partial(credit_cashback.delay, order.pk))
# Celery 5.4+ with Django: credit_cashback.delay_on_commit(order.pk)
return JsonResponse({"ok": True})
transaction.on_commit() runs the callback after the outermost transaction commits successfully, and never runs it on rollback. Since Celery 5.4, when used with Django, the default task class is DjangoTask, which has delay_on_commit() and apply_async_on_commit() methods that do exactly this.
In tests, TestCase wraps each test in a transaction that never commits, so on_commit callbacks don't fire on their own. Run them with captureOnCommitCallbacks:
# orders/tests.py
from unittest import mock
from django.test import TestCase
from .models import Order
class PayOrderTests(TestCase):
def test_task_is_sent_after_commit(self):
order = Order.objects.create(...) # fixture of your choice
with mock.patch("orders.views.credit_cashback.delay") as delay:
with self.captureOnCommitCallbacks(execute=True):
self.client.post(f"/orders/{order.pk}/pay/")
delay.assert_called_once_with(order.pk)
on_commit isn't a complete guarantee either. The commit succeeds, then the callback can't reach Redis or the process dies at that exact second, and the task is lost. For money, a simple remedy is a periodic "sweeper" task (via Celery beat) that finds unfinished work and re-sends it. That's safe precisely because the task is idempotent:
# orders/tasks.py — a safety net for messages lost between commit and broker
from datetime import timedelta
from celery import shared_task
from django.utils import timezone
from .models import Order
@shared_task
def resend_missing_cashback() -> int:
cutoff = timezone.now() - timedelta(minutes=10)
ids = list(
Order.objects.filter(
status=Order.Status.PAID,
cashback_credited=False,
paid_at__lt=cutoff,
).values_list("pk", flat=True)[:500]
)
for pk in ids:
credit_cashback.delay(pk) # safe: the task is idempotent
return len(ids)
In larger systems the general form of this is the outbox pattern: the message is written to a table in the same transaction as the business data, and a separate process forwards it to the broker.
The visibility timeout trap
Redis has nothing like RabbitMQ's "connection dropped, so return the unacked messages". So Celery's Redis transport keeps delivered messages in a separate unacked structure and puts a message back in the queue if no ack arrives within the visibility timeout (1 hour by default). That rescues the message when a worker dies, but it has two surprising side effects:
- Long tasks. If an
acks_latetask runs longer than the visibility timeout, the message is handed to another worker too, and the task runs twice even though nothing crashed. - countdown and ETA. Messages sent with
apply_async(countdown=...),eta=...orself.retry(countdown=...)are taken by a worker immediately and held in memory until they're due. If the delay is longer than the visibility timeout, the message is redelivered and each new worker starts holding it as well, so a single task ends up running in several copies.
The fix: set visibility_timeout higher than both your longest task and your largest countdown (6 hours in the settings above). For work that should happen a day from now, don't use a countdown. Use a run_at column in the database and a periodic task. With retries, retry_backoff_max caps the delay, which matters here too.
RabbitMQ has no visibility timeout, but it has a similar limit: consumer_timeout, how long the broker waits for an ack. If it's exceeded, the broker closes the channel and the message is redelivered. Check it if you have long tasks.
You can inspect the state in Redis:
# Messages taken by workers but not yet acked (Redis transport)
redis-cli HLEN unacked
# Messages waiting in the default queue
redis-cli LLEN celery
Pitfalls in production
- Not enough time for a graceful shutdown. After SIGTERM, Celery stops taking new tasks and tries to finish the running ones. In Kubernetes,
terminationGracePeriodSecondsdefaults to 30 seconds, and if a task takes longer, the pod gets SIGKILL. Set it above your longest task, or split tasks into short chunks. - Poison messages. With
reject_on_worker_lost=True, a task that OOM-kills the worker every time gets requeued and loops forever. Watch memory and set atime_limit. - Half-done work on retry. If a task writes to the database, then calls an API and fails, the retry redoes the first part too. Either do it all in one transaction or make each step idempotent on its own.
- Eager-mode tests. With
task_always_eager=True, tests skip the broker, serialization and acks. Code that passes an object as an argument passes the tests and fails in production. - Waiting on results. Don't call
result.get()inside another task: it makes workers wait on each other and can deadlock.
Checklist
- Tasks that move money or change state:
acks_late=Trueandtask_reject_on_worker_lost=True. - Every task survives a repeat: conditional
UPDATE,UniqueConstraint, or a status field. - Idempotency keys for external APIs, and always a
timeout. autoretry_foronly for transient errors, withretry_backoff,retry_jitterand an explicitmax_retries.- Only ids and plain values as arguments, JSON serializer.
delay()only throughtransaction.on_commit()ordelay_on_commit().- A periodic check task for lost messages.
visibility_timeout> longest task and largest countdown.- Shutdown grace period > longest task.
Takeaway
Celery won't give you "exactly once". By default a task can be lost, and with acks_late it can be repeated. In a system that moves money, the right choice is at-least-once delivery plus an idempotent task. Send tasks after the commit, pass ids, let the database reject duplicates, and account for the visibility timeout. Then a SIGKILL during a deploy doesn't turn into double cashback in a customer's account.
// comments
Loading comments...