Celery Once allows you to specify and run unique tasks across your distributed Celery cluster. It can be used to prevent workers performing the same task when scheduled multiple times.
Imagine the scenario of generating and send a PDF report to a user. On our web app, a user could kick off this task by submitting a form to a web server, which then triggers our Celery task.
If generating the report is slow and our user hits submit multiple times, we don't want to queue up additional repeated tasks that end up spamming the user's inbox.
Here is how we could solve the scenario using Celery Once!
After setting up
celery_once installed, we can write a mutually exclusive task, like so...
# tasks.py import celery from reports import generate_report from celery_once import QueueOnce @celery.task(base=QueueOnce) def send_pdf_report(email): report = generate_report() report.send(email=email)
If we try to run the same task, while it's already queued, an
AlreadyQueued exception is raised.
# Run the initial task, not yet queued up... >>> result = send_pdf_report.delay("email@example.com") # Duplicate task run before previous one completes.. >>> send_pdf_report.delay("firstname.lastname@example.org") Traceback (most recent call last): .. AlreadyQueued() # Running for a different user has its own lock >>> send_pdf_report.delay("email@example.com")
That's Celery Once in a nutshell! More documentation on how to install, set up and tweak it to your needs can be found here.