Many times in Django application development we reach a stage where background jobs like Scheduled emails, image processing etc. start to affect the application performance. These kinds of background jobs are mostly asynchronous and hence can be decoupled for easy maintenance & scaling using Celery. Which is basically a asynchronous task queue based on distributed message passing and can run millions of task in a day.
This blog does not explain how you can setup Celery for your application. For that there are plenty of good articles & posts available on the web. If you are looking for same, Celery official Doc will be a good place to kickstart. This blog is moreover focused on, which Message broker you can choose for your setup. A message broker is something that handles messaging (Informing about new tasks) for Celery. The key idea is to avoid variation in opinion available over the web on “use of different brokers” and suggest a selection approach based on application requirements.
Celery supports multiple message brokers ( Architecture diagram shown below ), among which Redis & RabbitMQ are most popular and deployed across many systems. Both of these systems have their pros and cons but choice becomes easy, if you know, what you need for your application, well.
RabbitMQ is recommended and very efficient (might be more than what you need) solution as message broker. It also supports reliable delivery using acknowledgement for messages But can give you a performance hit, If your application is task heavy and RabbitMQ is tuned for more reliability.
RabbitMQ is extremely efficient, widely deployed & tested message broker but can hit performance, if tuned for more durability and reliability. It’s more targeted towards serving enterprise messaging with advanced routing requirements.
Redis, on the other side is a easy to install & faster in processing but does Not guarantee the delivery, which is unlike RabbitMQ. Hence termination of server Or even Celery crash may result in loss of task information.
Redis is quick to start, lightweight & fast broker but does Not support reliable delivery Hence can be chosen for applications where in case system terminates, losing the information about tasks for a few minutes is Not critical.
Currently, Redis team is working towards bringing reliable delivery for Celery but there is a good chance that the feature will impact Redis performance significantly.
Other than Redis & RabbitMQ, there are other options also e.g. MongoDB, Django Database, SQLAlchemy & IronMQ. Most of these solutions are either experimental or 3rd party implementation. Except MongoDB, all of them lack support for monitoring and remote control. Here again, it makes sense to use any of these solutions if you are already experienced with that but most of these are meant for development environment.
Redis or RabbitMQ? Use Redis, if: You are looking for light and quickly deployable solution. Reliable message delivery is Not priority for you. Use RabbitMQ, if: Losing the information about the task is very critical. Want a solution which has better support and help over the web. The application has advanced Routing requirements.
Read more at: http://blog.langoor.mobi/django-celery-redis-vs-rabbitmq-message-broker/
