What is Celery:
Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.The execution units, called tasks, are executed concurrently on a single or more worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute asynchronously (in the background) or synchronously (wait until ready). Celery is used in production systems to process millions of tasks a day. More here
What will we do:
We will build a demo celery project check github repo for details to get a basic understanding of it. So let us start. which should cover the using celery in a set up like mentioned in the diagram
make a virtualenvironment
mkviretualenv celerydemo
start from here: do go through the docs atleast once.As per docs in this tutorial you will learn the absolute basics of using Celery. You will learn about:
###Choosing and installing a message transport (broker):
- Installing Celery and creating your first task.
- Starting the worker and calling tasks.
- Keeping track of tasks as they transition through different states, and inspecting return values.
1.chosing a broker: rabbitmq for this post
sudo apt-get install rabbitmq-server more about rabbitmq here
.If you are installing it on webfaction then check this post
###Installing celery:
pip install celery
###Install django and getting project ready:
pip install Django==1.7
mkdir celerydemo
cd celerydemo
django-admin startproject mysite
pwd > ~/.virtualenvs/celerydemo/.project
python manage.py migrate
python manage.py runserver
Although with the latest version of celery you may not need this but we will still use it to play around , you can find details here
pip install django-celery
Add djcelery to INSTALLED_APPS.
python manage.py migrate djcelery
inside the mysite/celery.py add some code for initializing celery check file here Also add in mysite/init some code details here
plus then we can have a demoapp whcih will have tasks.py file from where we can add tasks and import and call delay on them.
###run the celery:
python manage.py celeryd -B -l info
###run the camera
python manage.py celery events --camera=djcelery.snapshot.Camera
###run the server:
python manage runserver
###To test out everything:
python manage.py shell
from demoapp.tasks import add
add.delay(4,4)
###check the celery console to see the tasks
for monitoring if the tasks are not showing up on the admin panel beacause of some issue with django-celery you can use flower
pip install flower
to start flower:
flower -A mysite --port=5555
then visit:
localhost:5555
then we can see the tasks in real time on the dash flower board
note naming the task(@task(name='demoapp.tasks.add')) and using snapshot(python manage.py celery events --camera=djcelery.snapshot.Camera) together could work just fine wihthout flower.
Moreover you can demonize celery with supervisor or circus. If you are interested in circus check this post