[TkTech]

Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.

[hangonhn]

Yeah that list right there. That's exactly it.

We don't hate Celery at all. It's just a bit harder to get it to do certain things and requires a bit more coding and understanding of celery than what we want to invest time and effort in.

Again, no hate towards Celery. It's not bad. We just want to see if there are better options out there.

[brianwawok]

But if you are too small for celery, it seems a hard sell to buy a premium message queue?

My top problem with my celery setup has always been visibility. AI and I spent and afternoon setting up a Prometheus / grafana server, and wiring celery into it. Has been a game changer. When things go crazy in prod, I can usually single it down to a specific task for a specific user. Has made my troubleshooting go from days to minutes. The actual queue and execute part has always been easy / worked well.

[xmcqdpt2]

For an even lighter system than Celery, I'm a big fan of

https://python-rq.org/

It's super low on the dependencies and integrates nicely as a library into python applications. It's very bare bones.

[TkTech]

I kinda have my own https://github.com/tktech/chancy :P