[nakovet]

> Forget Celery

How do you do background job processing without Celery? Every app/website I ever developed required some sort of background job processing, for Python I use Celery, Rails I use Sidekiq, Node.js I use Faktory. One of the biggest drawbacks (at least until a while ago) was that setting this up had to be a hacky webhook call to your own app that allowed up to N seconds request/response.

[stavros]

If you can tolerate 1 sec latency, Django-background-tasks is pretty good.

[williamdclt]

DB-based queues are pretty common (see outbox pattern). You actually don't have much choice but an outbox in your relational DB anyway, if you want the job to be enqueued transactionally.

[marcosdumay]

You write the demand to the database, and have a worker process execute it. Exactly like you'd do with celery.

Celery is a solution to scalability, not to any business problem in particular.

[hyllos]

Years ago I ditched Celery in favour of using BLPOP with redis. It was multiple times faster than Celery. Is this still the case today? So with vanilla Redis you need less workers for same throughput; if you need more throughput, spawn multiple workers.

[tempay]

I've had success with a boring event loop which looks at the state of the system and does all the background work. It's simple to implement and much easier to manage failing tasks that need manual intervention to stop failing. It also makes it easy to implement batching of work as you always know everything that needs to be done in each loop.

I also have multiple celery applications, but I wouldn't recommend it for smaller or simplier apps.

[porridgeraisin]

> boring event loop ... state of the system ... background work

Can you explain what you mean?

[kukkeliskuu]

Django background workers, backported as django-tasks library, the tasks being stored in my database (Postgres). Everything is started by honcho, which starts Superchronic and Django, and Superchronic runs django-tasks. All this (except the database) is running inside single fly.io app.