[danielstocks]

I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism to scheduled tasks. Eg. “Run this task 12 hours before departure”. Maybe AMQP is the wrong place to solve that problem.

[modal-soul]

I've been using something like this for exponential backoffs, but I think it'd work for this case as well.

Let's say you've got one exchange and one main queue for processing: jobs.exchange and jobs.queue respectively.

If you need to schedule something for later, you'd assert a new queue with a TTL for the target amount of time (scheduled-jobs-<time>.queue). Also set an expiry of some amount of time, so it'd get cleaned up if nothing had been scheduled for that particular time in a while. Finally, have its dead-letter-exchange set to jobs.exchange.

This could lead to a bunch of temporary queues, but the expiration should clean them up when they haven't been used for a bit.

[jpalomaki]

Haven’t used this myself, but there seems to be a plugin for delaying message delivery: https://www.rabbitmq.com/blog/2015/04/16/scheduling-messages...

[danthemanvsqz]

You schedule tasks very easily with the Celery Beat Scheduler. I've used it in production system to kick off big jobs and notifications.

https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-...

[spamizbad]

You're usually stuck polling for stuff like that. I'm a big fan of using things like Advanced Python Scheduler for those sorts of tasks: https://apscheduler.readthedocs.io/en/stable/

[bvm]

Yeh I would argue don't use a message queue for this, they're really best processing many messages quickly, there are plenty of scheduling libraries that have various persistence layers to handle this depending on your ecosystem.

[0xkd]

Celery has eta/countdown params that allow for running tasks at a specific time

[polote]

you can use a redis sortedset https://redis.io/commands/zadd