Writing a chat application in Django 4.2 using async StreamingHttpResponse

[samwillis]

All the new asyncIO stuff in Django is awesome, they are doing a phenomenal job retrofitting it all to an inherently sync designed framework.

One thing to note, it's been possible to build these sort of things with Django by using Gevent and Gunicorn for over a decade, and it works well.

In many ways I wish Gevent had been adopted by Python rather than AsyncIO.

[btown]

Can second the Django/Gevent/Gunicorn stack - we use it in production and IMO it's much easier to reason about and code in than asyncio. Just write synchronous Python, including long-running `requests` calls, and the system will yield control whenever you're blocked on network or disk I/O, no matter how deep that is in your stack. The entire ecosystem of Python libraries just works. I also wish more people knew about gevent - it's truly magical, especially if you need to work with APIs with unpredictable latency!

[jononomo]

Where can I learn more about this? I've been thinking of trying to integrate Supabase Realtime (https://github.com/supabase/realtime) into my Django app (without the rest of Supabase), but I'd also like to keep things even simpler if possible.

Also, what was the reason not to go with Gevent?

[jwtnb]

Hi, thank you for your comment. I have been looking for correct ways to integrate websockets on my django app. Currently I'm using gevent gunicorn setup. Can you elaborate a bit more on how to make websockets work with an existing app? Thanks.

[Ralfp]

Just a heads up that currently Django is not cleaning up open PostgreSQL connections when ran in ASGI mode, leading to too many open connections error: https://code.djangoproject.com/ticket/33497

[pdhborges]

I'll have zero trust running async Django until all sync_to_async and async_to_sync calls are removed from the code base.

[kolanos]

Reading that bug report it looks like Django hasn't been isolated as the source of the problem? Looks like Carlton Gibson closed it until it is reproducible. Problem could be in uvicorn, psycopg2, etc

[etimberg]

Yeah, I've definitely seen odd things in Django ASGI. Had at least one instance where it looked like it leaked data between requests but couldn't get a clear enough reproduce. Switching back to WSGI worked fine though

[baq]

Running Postgres without a connection bouncer is a huuuge no-no already, but if this breaks the default mode of pgbouncer it’s basically a showstopper.

[whalesalad]

that’s kind of a dealbreaker

[pmontra]

This implementation is probably a little different but we were using long poll in the late 90s and early 2000s. The problem was that you were committing one thread (or worse, one process) to each client. That obviously doesn't scale unless threads are extremely light on RAM and either the OS or the runtime support a large number of them. I remember that a way out was using continuations. Jetty was a Java application server that supported them (random link [1]) One thread -> many connections. I didn't investigate how Django is implementing this now but CPUs and RAM are still CPUs and RAM.

[1] https://stackoverflow.com/questions/10587660/how-does-jetty-...

[klabb3]

> The problem was that you were committing one thread (or worse, one process) to each client.

This is mostly true today as well, although we do have beefier machines and more efficient thread pooling.

I did some personal research for infrequent real-time notifications. My conclusion was that many stateful connections are poorly memory-optimized by language runtimes and reverse proxies. Even with lightweight tech like Golang, gRPC, nginx, etc, it’s hard to push anything less than 30 kB for an idle conn, mostly from thread/goroutine stacks, user space buffers and (easy to overlook) a bunch of HTTP headers and TLS handshake state that often remain after establishing the conn. That’s without any middleware or business logic wants their share of the cake.

The only mature project I found that really took this stuff seriously is uWebSockets. It’s extremely lightweight, around an OOM better than most alternatives. Highly recommend – they also have a socket library so you can implement other protocols if needed.

Anyway, it’s important to be aware that massive amounts of long running connections is not just about adding a websocket/SSE library. Chances are you’ll need a mostly-separate serving stack for that purpose.

[Waterluvian]

“ The idea is that the client "subscribes" to an HTTP endpoint, and the server can then issue data to the client as long as the connection is open.”

To those who have been around longer than me: isn’t this just long polling, that predates websocket?

[kiraaa]

https://github.com/sysid/sse-starlette makes token streaming so much easier in python

[dnadler]

I’m using this for an internal ChatGPT UI clone and it’s working great.

The actual biggest pain for me has been the front end handling of it with the fetch api. But that’s likely just due my inexperience with it.

[Thaxll]

Do modern chat actually use websocket and the like? Discord / Slack on the web ( browser ) what do they use?

[vorticalbox]

The slack_bolt library has support for web sockets.

https://api.slack.com/apis/connections/socket

[jhgg]

Discord uses websockets on all clients/platforms.

[holler]

yes they use websockets