[tmarice]

Celery is the worst background task framework, except for all the others.

There are bugs and issues, but because so many people are using it, you’re rarely the first to stumble upon a problem. We processed double-digit millions of messages daily with Celery + RabbitMQ without major obstacles. Regardless of what people say, it should be your first go-to.

[formerly_proven]

Celery has way too much magic crammed into it, it is very annoying to debug, and produces interesting bugs. Celery is/was also a "pickle-first" API and this almost always turns out to be the wrong choice. As a rule of thumb, persisting pickles is a really bad idea. Trying to hide IPC / make-believe that it's not there tends to be a bad idea. Trying to hide interfaces between components tends to be a bad idea. Celery combines all of these bad ideas into one blob. The last time I looked the code was also a huge mess, even for old-guard-pythonic-code standards.

[rbanffy]

I think Celery has a lot of magic happening under it. When the abstractions are so high, it's important they never leak and you don't see anything below the turtles you are supposed to see.

I often prefer designing around explicit queues and building workers/dispatchers. One queuing system I miss is the old Google App Engine one - you set up the queue, the URL it calls with the payload (in your own app), the rate it should use, and that's it.