It's okay till it's not. Everyone I know who had Celery in production was looking for a substitution (custom or third-party) on a regular basis. Too many moving pieces and nuances (config × logic × backend), too many unresolved problems deep in its core (we've seen some ghosts you can't debug), too much of a codebase to understand or hack. At some point we were able to stabilize it (a bunch of magic tricks and patches) and froze every related piece; it worked well under pressure (thanks, RabbitMQ).
Because it’s a seducer. It does what you need to do and you two are happy together. So you shower more tasks on Celery and it becomes cold and non-responsive at random times.
And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues.
Although we could say the same thing about Kafka, couldn't we? It's made for much higher throughput and has usually other use cases, but it's also great until it's not great.
Celery is great and awful at the same time. In particular, because it is many Python folks' first introduction to distributed task processing and all the things that can go wrong with it. Not to mention, debugging can be a nightmare. Some examples:
- your function arguments aren't serializable
- your side effects (e.g. database writes) aren't idempotent
- discovering what backpressure is and that you need it
- losing queued tasks during deployment / non-compatible code changes
There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff.
> your side effects (e.g. database writes) aren't idempotent
What does idempotent mean in this context, or did you mean atomic/rollback on error?
I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips
JimDabell|2 months ago
— https://steve.dignam.xyz/2023/05/20/many-problems-with-celer...
> The problems with (Python’s) Celery:
— https://docs.hatchet.run/blog/problems-with-celery
> Dramatiq motivation:
— https://dramatiq.io/motivation.html
Here are some alternatives:
Dramatiq: https://github.com/Bogdanp/dramatiq
RQ: https://github.com/rq/rq
Huey: https://github.com/coleifer/huey
Hatchet: https://github.com/hatchet-dev/hatchet
hintoftime|2 months ago
https://temporal.io/
https://docs.dbos.dev/
https://news.ycombinator.com/item?id=45797228
https://python-absurd-client.readthedocs.io/en/latest/quicks...
https://pgqueuer.readthedocs.io/en/latest/
hda111|2 months ago
leobuskin|2 months ago
tclancy|2 months ago
And debugging is a pain in the ass. Most places I’ve been that have it, I’ve tried to sell them on adding Flower to give better insight and everyone thinks that’s a very good idea but there isn’t time because we need to debug these inscrutable Celery issues.
https://flower.readthedocs.io/en/latest/
sgt|2 months ago
akoumjian|2 months ago
- your function arguments aren't serializable - your side effects (e.g. database writes) aren't idempotent - discovering what backpressure is and that you need it - losing queued tasks during deployment / non-compatible code changes
There's also some stuff particular to celery's runtime model that makes it incredibly prone to memory leaks and other fun stuff.
Honestly, it's a great education.
ffsm8|2 months ago
What does idempotent mean in this context, or did you mean atomic/rollback on error?
I'm confused because how could a database write be idempotent in Django? Maybe if it introduced a version on each entity and used that for crdt on writes? But that'd be a significant performance impact, as it couldn't just be a single write anymore, instead they'd have to do it via multiple round trips
saaspirant|2 months ago