/skill

Django and Celery Tasks

Use Celery to run slow Django work outside web requests.

Use This Skill When

• Send email in the background.
• Run slow jobs.
• call an API without making the user wait.
• Run jobs at set times.
• Split a large job into small jobs.

Do not use Celery for work that must finish before the page loads.

Setup

1. Install Celery

For Redis:

pip install celery redis

For RabbitMQ:

pip install celery

2. Set the Broker

Add this to Django settings:

CELERY_BROKER_URL = "redis://localhost:6379/0"
CELERY_RESULT_BACKEND = "redis://localhost:6379/1"
CELERY_TASK_TIME_LIMIT = 300
CELERY_TASK_SOFT_TIME_LIMIT = 270

Keep the broker URL in an env file when it has a password.

3. Create the Celery App

Create myapp/celery.py:

import os

from celery import Celery

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "myapp.settings")

app = Celery("myapp")
app.config_from_object("django.conf:settings", namespace="CELERY")
app.autodiscover_tasks()

Add this to myapp/__init__.py:

from .celery import app as celery_app

__all__ = ("celery_app",)

Change myapp to the name of your Django project.

4. Write a Task

Create emails/tasks.py:

from celery import shared_task
from django.core.mail import send_mail


@shared_task(
    autoretry_for=(Exception,),
    retry_backoff=True,
    retry_jitter=True,
    retry_kwargs={"max_retries": 5},
)
def send_welcome_email(user_id):
    from django.contrib.auth import get_user_model

    user = get_user_model().objects.get(pk=user_id)

    send_mail(
        subject="Welcome",
        message="Thanks for joining.",
        from_email=None,
        recipient_list=[user.email],
    )

Pass IDs, text, numbers, lists, or maps to tasks. Do not pass Django model objects.

5. Start a Worker

celery -A myapp worker --loglevel=INFO

Run at least one worker in each live app setup.

Queue a Task

from emails.tasks import send_welcome_email

send_welcome_email.delay(user.id)

If the task depends on saved data, queue it after the database change is done:

from django.db import transaction

transaction.on_commit(
    lambda: send_welcome_email.delay(user.id)
)

This keeps the worker from reading data before it is saved.

Run Tasks on a Schedule

Add this to myapp/celery.py:

from celery.schedules import crontab

app.conf.beat_schedule = {
    "send-report-daily": {
        "task": "reports.tasks.send_report",
        "schedule": crontab(hour=9, minute=0),
    },
}

Start the scheduler:

celery -A myapp beat --loglevel=INFO

Run only one scheduler. Two schedulers may send the same task twice.

Set the time zone in Django settings:

TIME_ZONE = "America/Chicago"
USE_TZ = True
CELERY_TIMEZONE = TIME_ZONE

Concrete Example

Queue an image resize after an upload is saved:

# photos/tasks.py
from celery import shared_task


@shared_task(bind=True, max_retries=3)
def resize_photo(self, photo_id):
    from .models import Photo

    try:
        photo = Photo.objects.get(pk=photo_id)
        photo.make_thumbnail()
    except Photo.DoesNotExist:
        return
    except OSError as error:
        raise self.retry(exc=error, countdown=30)

# photos/views.py
from django.db import transaction
from django.shortcuts import redirect

from .forms import PhotoForm
from .tasks import resize_photo


def upload_photo(request):
    form = PhotoForm(request.POST, request.FILES)

    if form.is_valid():
        photo = form.save()
        transaction.on_commit(
            lambda: resize_photo.delay(photo.id)
        )
        return redirect("photo-detail", pk=photo.id)

The page can return at once. The worker resizes the photo later.

Error Rules

• Retry short network or broker errors.
• Do not retry bad input.
• Set a retry limit.
• Set a time limit for slow tasks.
• Log the task name and record ID.
• Keep failed tasks so they can be checked or run again.
• Use a dead-letter queue when the broker supports it.
• Do not log passwords, tokens, or private user data.

Safe Task Rules

A task may run more than once. Write it so a second run is safe.

Before sending money, email, or a web request, check if the work was already done. Use a unique key or a saved status when needed.

Keep tasks small. Pass file paths or record IDs instead of large files.

Do not wait for one task inside another task with .get(). Queue the next task instead.

Checks Before Release

• The broker is running.
• The worker can load Django settings.
• The task is found by Celery.
• Retries have a limit.
• Slow tasks have a time limit.
• Tasks are safe to run twice.
• Tasks are queued after database commit.
• Only one scheduler is running.
• Worker logs are saved.
• Failed jobs can be checked and run again.
• Workers stop in a safe way during release.