[sischoel]

Genuine question, not a comment on OPs implementation: What is the reason that people so often use some external service for creating task queues (often backed by some kind of db), instead of just implementing it themselves in whatever language they are using for the rest of their application?

Is it: - They want persistence in case the system goes down, or the task queue service is restarted.

- They want to replicate their task queue service and want to be sure that the data is properly synchronized.

- The amount of tasks/time if simply too much for some standard library queues

- They think is is easier to add another dependency than implement it themselves

- something else

?

[nhumrich]

Distributed queuing, with horizontal scaling workers and retries, without ever loosing a message/job, is surprisingly difficult to get right. It's simply not worth your time in a lot of cases. Similarly, why people don't just write their own web framework.

It sounds like your question is focused on "why not just use an in memory queue". Two reasons: 1. You loose messages/jobs, and it's not scalable 2. If the task is CPU intensive, it will lock up or slow down your web server.

[tibanne]

I can't say, I implemented my own queuing system in Python. It seemed simpler.