[darkoob12]

If you're working on machine learning the most economic choice is Python.

But weiting a processing pipeline with Python is frustrating if you have worked with C# concurrency.

I figured the best option is Celery and you cannot do it without an external broker. Celery is a mess. I really hate it.

[nijave]

Agree. I think it's improved a bit but Celery is frustrating as the defacto job/queue solution. A lot of the defaults make it unreliable (it can lose jobs if workers crash or don't shutdown cleanly)

I'm hoping the existence of free-threading will push for more first-class concurrency primitives. Concurrent Futures is nice until you need a concurrent-safe data structure besides a queue

[machiaweliczny]

Agree that celery is a mess and it doesn't work well with async (Asyncio) python. I think version 6 maybe will support it sometime.

I also had a lot of problem due to async primitives with sqlalchemy - there's some tricky stuff with asyncio.gather vs TaskGroup and how sqlalchemy session works with it to be able to compose code easily.