[hangonhn]

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue.

I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is.

Does anyone have experience with both and is able to give a quick comparison?

Thanks!

[BowBun]

Very, very different tools, though they cover similar areas.

Temporal - if you have strict workflow requirements, want _guarantees_ that things complete, and are willing to take on extra complexity to achieve that. If you're a bank or something, probably a great choice.

Oban - DB-backed worker queue, which processes tasks off-thread. It does not give you the guarantees that Temporal can because it has not abstracted every push/pull into a first-class citizen. While it offers some similar features with workflows, to multiple 9's of reliability you will be hardening that yourself (based on my experience with Celery+Sidekiq)

Based on my heavy experience with both, I'd be happy to have both available to me in a system I'm working on. At my current job we are forced to use Temporal for all background processing, which for small tasks is just a lot of boilerplate.

[kfajdsl]

I’ll say that, I think this varies by language/SDK - at least with the Temporal TypeScript SDK, a simple single idempotent step background task is however many lines of code to do the actual work in an activity, and then the only boilerplate is like 3 lines of code for a simple workflow function to call the activity.

[tecoholic]

We migrated from Celery to Prefect a couple of years back and have been very happy. But ours is a small op which handles tasks in 1000s and not millions. It’s been night and day in terms of visibility and tracking. I would definitely recommend it.

It’s a heavy weight that covers a lot of use cases. But we just run simple ProcessWorkers for our regular needs and ECS worker for heavier ML tasks.

[owaislone]

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

[TkTech]

Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.

[kukkeliskuu]

Most Django projects just need a basic way to execute timed and background tasks. Celery requires seperate containers or nodes, which complicates things unnecessarily. Django 6.0 luckily has tasks framework -- which is backported to earlier django versions has well, which can use the database. https://docs.djangoproject.com/en/6.0/topics/tasks/

[alanwreath]

I like celery but I started to try other things when I had projects doing work from languages in addition to python. Also I prefer the code work without having to think about queues as much as possible. In my case that was Argo workflows (not to be confused with Argo CD)