This is all true, but it also serves to remind us that Rails gives developers so much out of the box, even if you're not aware of it.
ActionCable is Rails' WebSockets wrapper library, and it addresses basically every pain point in the post. However, it does so in a way that all Rails developers are using the same battle-tested solution. There's no need for every project to hack together its own proprietary approach.
Thundering herds, heartbeat monitoring are both covered.
If you need a messaging schema, I strongly recommend that you check out CableReady. It's a powerful library for triggering outcomes on the client. It ships with a large set of operations, but adding custom operations is trivial.
Elixir’s lightweight processes are also a good fit. Though I’ve seen some benchmarks that claim that goroutines can hit even lower overhead per connection.
My SaaS has been using WebSockets for the last 9 years. I plan to stop using them and move to very simple HTTP-based polling.
I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections with plenty of data going over them.
At a certain scale, this results in support requests, and frustratingly, I can't do anything about the problems my customers encounter.
The other problems are smaller, but still annoying, for example it isn't easy to compress content transmitted through websockets.
I always recommend looking at Server-Sent Events [0] and EventSource [1]. It's a standardization of old style long-polling, mapping very well to the HTTP paradigm and is built in to the web standard.
It's so much easier to reason about than websockets, and a naive server side implementation is very simple.
A caveat is to only use them with HTTP 2 and/or client side logic to only have one connection open to the server, because of browser limits on simultaneous requests to the same origin.
The last project I worked on went in the same direction.
Everything works great in local/qa/test, and then once we move to production we inevitably have customers with super weird network security arrangements. Users in branch offices on WiFi hardware installed in 2007. That kind of thing.
When you are building software for other businesses to use, you need to keep it simple or the customer will make your life absolutely miserable.
This is likely a misconfiguration or bugs on your end. Our products use WebSockets extensively for both business logic and for media delivery. We have significant traffic, including from 3rd world countries with extremely poor networks. When the server and the browser are proper, the reliability of the WebSocket protocol and the software stack is basically not different from raw TCP. When we have issues, the bugs are always on our end due to
1) ingress software (firewalls, WAPs, reverse proxies, TLS termination).
2) HTTP server protocol parsing and processing of the WebSocket data stream.
3) Web/HTTP framework issues.
We never have any issues due to networks on the users end, apart from the quality of the connection itself.
When seen from the equipment side, the WebSocket connection is an opaque stream of data. No different from a fps gameplay or a livestream. The equipment can break it but then the underlying HTTP breaks as well, giving very clear errors in browsers. Reconnection and keepalive for WebSockets in browsers are very robust which you can actually prove by tests...
What are the typical payload sizes in your WebSocket messages? Could you share the median and p99 values?
I've also discovered similar networking issues in my own application while traveling. For example, in Vietnam right now, I was facing recurring issues like long connection establishment times and loss of responsiveness mid-operation.
I thought I was losing my mind - I even configured Caddy to not use HTTP3/QUIC (some networks don't like UDP).
I moved some chunkier messages in my app to HTTP requests, and it has become much more stable (though still iffy at times).
This is surprising to me as I would expect network equipment to just see a TCP connection given both HTTP and Websockets are an application layer protocol and that long lived TCP connections are quite ubiquitous (databases, streaming services, SSH, etc).
The key to managing this complexity is to avoid mixing transport-level state with application-level state. The same approach for scaling HTTP requests also works for scaling WebSocket connections:
* Read, write and track all application-level state in a persistent data store.
* Identify sessions with a session token so that application-level sessions can span multiple WebSocket connections.
It's a lot easier to do this if your application-level protocol consists of a single discrete request and response (a la RPC). But you can also handle unidirectional/bidirectional streaming, as long as the stream states are tracked in your data store and on the client side.
Functional core, imperative shell makes testing and this fast iteration a lot easier. It’s best if your business logic knows very little about transport mechanisms.
I think part of the problem is that early systems wanted to eagerly process requests while they are still coming in. But in a system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines on attempting to make forward progress on incomplete data. Which means you can divorce the concept of a payload entirely from how you acquired it.
Many years ago, we used to start a streaming session with an http request, then upgrading to websockets after obtaining a response (this was our original "StreamSense" mechanism). In recent years, we changed StreamSense to go websocket first and fallback to http streaming or http long polling in case of issues.
At Lightstreamer, we started streaming data 25 years ago over http, then moving to websockets. We've seen so many different behaviors in the wild internet and got some much feedback from the fieldsl in these decades that we believe our current version of Lightstreamer includes heuristics and mechanisms for virtually every possible aspect of websockets that could go wrong. From massive disconnections and reconnections, to enterprise proxies with deep inspections, to mobile users continuously switching networks.
I recall when a big customer required us to support one million live websocket connections for each server (mid-sized) keeping low latency. It was challenging but forced us to come up with a brand new internal architecture. So many stories to tell covering 25 years of evolution...
I am really unsure why devs around the world keep defaulting to websockets for things that are made for server sent events. In 90% of the usecases i see, websockets are just not the right fit. Everything is simpler and easier with SSE. Some exceptions are high throughput >BI<directional data streams. But even if eg. your synced multiplayer cursors in something like figma use websockets don't use it for everything else eg. your notification updates.
I wrote about the way we handle WebSocket connections at Canva a while ago [1]. Even though some small things have changed here and there since the post was published, the overall approach has held up pretty well handling many millions of concurrent connections.
That said, even with great framework-level support, it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session".
I recall another complication with websockets: IIRC it's with proxy load balancers, like binding a connection to a single connection server, even if the backend connection is using HTTP/2. I probably have the details wrong. I'm sure someone will correct my statement.
I think there is a way to do it, but it likely involves custom headers on the initial connection that the load balancer can read to route to the correct origin server.
I imagine the way it might go is that the client would first send an HTTP request to an endpoint that returns routing instructions, and then use that in the custom headers it sends when initiating the WebSocket connection.
I think it's more that WebSockets are held open for a long time, so if you're not careful, you can get "hot" backends with a lot of connections that you can't shift to a different instance.
It can also be harder to rotate backends since you know you are disrupting a large number of active clients.
Horizontal scaling is certainly a challenge. With traditional load balancers, you don't control which instance your clients get routed to, so you end up needing to use message brokers or stateful routing to ensure message broadcasts work correctly with multiple websocket server instances.
The only complexity I have found with regards to scaling WebSockets is knowing the minimum delay between flush event completion and actual message completion to destination. It takes longer to process a message, even on IPC routing, than it does to kill a socket. That has upstream consequences with consideration of redirection and message pipes between multiple sockets. If you kill a socket too early after a message is flushed from the socket there is a good chance the destination sees the socket collapse before it has processed the final message off the socket and that processing delay is not something a remote location is easily aware of.
I have found for safety you need to allow an arbitrary delay of 100ms before killing sockets to ensure message completion which is likely why the protocol imposes a round trip of control frame opcode 8 before closing the connection the right way.
> WebSocket connections can be unexpectedly blocked, especially on restrictive public networks.
What? How would public network even know you’re running a websocket if you’re using TLS? I dont think it’s really possible in general case
> Since SSE is HTTP-based, it's much less likely to be blocked, providing a reliable alternative in restricted environments.
And websockets are not http-based?
What article describes as challenges seems like very pedestrian things that any rpc-based backend needs to solve.
The real reason websockets are hard to scale is because they pin state to a particular backend replica so if the whole bunch of them disconnect at scale the system might run out of resources trying to re-load all that state
I agree here. I have had an experience of scaling WebSockets server to 20M connections on a single server (with this one https://github.com/ITpC/LAppS.git). However there are several issues with scaling WebSockets, on the backends as well: mutex locking, non-parallel XOR of input stream, utf8 validation.
I do not know the state of the above repository code, it seems that it was never updated for at least 5 years. There were bugs in HTTP parsing in the client part for some cases. Though vertical scalability was excellent. Sad this thing never reached production state.
The comment about Render/Railway gracefully tranferring connections seems weird? I am pretty sure it just kills the service after the new one is alive which will kill the connections. Not some fancy zero downtime reconnect.
for me, the most important lesson i've learned when using websockets is to not use them whenever possible.
i don't hate them, they're great for what they are, but they're for realtime push of small messages only. trying to use them for the rest of your API as well just throws out all the great things about http - like caching and load balancing, and just normal request/response architecture. while you can use websockets for that it's only going to cause you headaches that are already solved by simply using a normal http api for the vast majority of your api.
> Elixir will get you pretty far along this scaling journey without too many problems:
been running a phoenix app in prod for 5 years. 1000+ paying customers. heavy use of websockets. never had an issue with the channels systems. it does what it says on the tin and works great right out of the box
Websockets are bidirectional while SSE is unidirectional (server to client). That said, there's nothing stopping you from facilitating client to server communication separately from SSE, you just don't have to build that channel with websockets.
I have been working on an idea/Node.js library called vramework.dev recently, and a big part of it focuses on addressing the main complexities mentioned below.
For a bit of background, in order to tackle scalability, the initial approach was to explore serverless architecture. While there are both advantages and disadvantages to serverless, a notable issue with WebSockets on AWS* is that every time a message is received, it invokes a function. Similarly, sending a message to a WebSocket requires invoking an HTTP call to their gateway with the websocket / channel id.
The upside of this approach is that you get out-of-the-box scalability by dividing your code into functions and building things in a distributed fashion. The downside is latency, due to all the extra network hops.
This is where vramework comes in. It allows you to define a few functions (e.g., onConnect, onDisconnect, onCertainMessage) and provides the flexibility to run them locally using libraries like uws, ws, or socket.io, or deploy them in the cloud via AWS or Cloudflare (currently supported).
When running locally, the event bus operates locally as well, eliminating latency issues. If you apply the same framework to serverless, latency increases, but you gain scalability for free.
Additionally, vramework provides the following features:
- Standard Tooling
Each message is validated against its typescript signature at runtime. Any errors are caught and sent to the client. (Note: The error-handling mechanism has not yet been given much thought into as an API). Rate limiting is also incorporated as part of the permissioning system (each message can have permissions checked, one of them could rate limiting)
- Per-Message Authentication
It guards against abuse by ensuring that each message is valid for the user before processing it.
For example, you can configure the framework to allow unauthenticated messages for certain actions like authentication or ping/pong, while requiring authentication for others.
- User Sessions
Another key feature is the ability to associate each message with a user session. This is essential not only for authentication but also for the actual functionality of the application. This is done by doing a call to a cache (optionally) which returns the user session associated with the websocket. This session can be updated during the websocket lifetime if needed (if your protocol deals with auth as part of it's messages and not on connection)
And one last thing, it also produces a fully typed websocket client, so if using routes (where a property in your message indicates which function to use, the approach AWS uses serverless).
Would love to get thoughts and feedback on this!
edit: *and potentially Cloudflare, though I’m not entirely sure of its internal workings, just the Hibernation server and optimising for cost saving
const onConnect: ChannelConnection<'hello!'> = async (services, channel) => {
// On connection (like onOpen)
channel.send('hello') // This is checked against the input type
}
const onDisconnect: ChannelDisconnection = async (services, channel) => {
// On close
// This can't send anything since channel closed
}
const onMessage: ChannelMessage<'hello!' | { name: string }, 'hey'> = async (services, channel) => {
channel.send('hey')
}
export const subscribeToLikes: ChannelMessage<
{ talkId: string; action: 'subscribeToLikes' },
{ action: string; likes: number }
> = async (services, channel, { action, talkId }) => {
const channelName = services.talks.getChannelName(talkId)
// This is a service that implements a pubsub/eventhub interface
await services.eventHub.subscribe(channelName, channel.channelId)
// we return the action since the frontend can use it to route to specific listeners as well (this could be absorbed by vrameworks runtime in future)
return { action, likes: await services.talks.getLikes(talkId) }
}
addChannel({
name: 'talks',
route: '/',
auth: true,
onConnect,
onDisconnect,
// Default message handler
onMessage,
// This will route the message to the correct function if a property action exists with the value subscribeToLikes (or otherwise)
onMessageRoute: {
action: {
subscribeToLikes: {
func: subscribeToLikes,
permissions: {
isTalkMember: [isTalkMember, isNotPresenter],
isAdmin
},
},
},
},
})
A code example.
Worth noting you can share functions across websockets as well, which allows you to compose logic across different ones if needed
peteforde|1 year ago
ActionCable is Rails' WebSockets wrapper library, and it addresses basically every pain point in the post. However, it does so in a way that all Rails developers are using the same battle-tested solution. There's no need for every project to hack together its own proprietary approach.
Thundering herds, heartbeat monitoring are both covered.
If you need a messaging schema, I strongly recommend that you check out CableReady. It's a powerful library for triggering outcomes on the client. It ships with a large set of operations, but adding custom operations is trivial.
https://cableready.stimulusreflex.com/hello-world/
While both ActionCable and CableReady are Rails libraries, other frameworks would score huge wins if they adopted their client libraries.
hinkley|1 year ago
atul-jalan|1 year ago
jwr|1 year ago
I found that scalability isn't a problem (it rarely is these days). The real problem is crappy network equipment all over the world that will sometimes break websockets in strange and mysterious ways. I guess not all network equipment vendors test with long-lived HTTP websocket connections with plenty of data going over them.
At a certain scale, this results in support requests, and frustratingly, I can't do anything about the problems my customers encounter.
The other problems are smaller, but still annoying, for example it isn't easy to compress content transmitted through websockets.
yesbabyyes|1 year ago
It's so much easier to reason about than websockets, and a naive server side implementation is very simple.
A caveat is to only use them with HTTP 2 and/or client side logic to only have one connection open to the server, because of browser limits on simultaneous requests to the same origin.
[0] https://developer.mozilla.org/en-US/docs/Web/API/Server-sent... [1] https://developer.mozilla.org/en-US/docs/Web/API/EventSource
bob1029|1 year ago
Everything works great in local/qa/test, and then once we move to production we inevitably have customers with super weird network security arrangements. Users in branch offices on WiFi hardware installed in 2007. That kind of thing.
When you are building software for other businesses to use, you need to keep it simple or the customer will make your life absolutely miserable.
il-b|1 year ago
akshayKMR|1 year ago
I've also discovered similar networking issues in my own application while traveling. For example, in Vietnam right now, I was facing recurring issues like long connection establishment times and loss of responsiveness mid-operation. I thought I was losing my mind - I even configured Caddy to not use HTTP3/QUIC (some networks don't like UDP).
I moved some chunkier messages in my app to HTTP requests, and it has become much more stable (though still iffy at times).
catlifeonmars|1 year ago
tomrod|1 year ago
10000truths|1 year ago
* Read, write and track all application-level state in a persistent data store.
* Identify sessions with a session token so that application-level sessions can span multiple WebSocket connections.
It's a lot easier to do this if your application-level protocol consists of a single discrete request and response (a la RPC). But you can also handle unidirectional/bidirectional streaming, as long as the stream states are tracked in your data store and on the client side.
hinkley|1 year ago
I think part of the problem is that early systems wanted to eagerly process requests while they are still coming in. But in a system getting 100s of requests per second you get better concurrency if you wait for entire payloads before you waste cache lines on attempting to make forward progress on incomplete data. Which means you can divorce the concept of a payload entirely from how you acquired it.
magicalhippo|1 year ago
Then again, the frontend and backend are a distributed system, so not that weird one comes to similar conclusions.
[1]: https://news.ycombinator.com/item?id=42813049 Every System is a Log: Avoiding coordination in distributed applications
Rldm840|1 year ago
beng-nl|1 year ago
Eg “The 1M challenge”
jFriedensreich|1 year ago
SerCe|1 year ago
That said, even with great framework-level support, it's much, much harder to build a streaming functionality compared to plain request/response if you've got some notion of a "session".
[1]: https://www.canva.dev/blog/engineering/enabling-real-time-co...
crabmusket|1 year ago
This touches something that I think is starting to become understood- the concept of a "session backend" to address this kind of use case.
See the complexity of disaggregation a live session backend on AWS versus CloudFlare: https://digest.browsertech.com/archive/browsertech-digest-cl...
I wrote about session backends as distinct from durable execution: https://crabmusket.net/2024/durable-execution-versus-session...
exabrial|1 year ago
atul-jalan|1 year ago
I imagine the way it might go is that the client would first send an HTTP request to an endpoint that returns routing instructions, and then use that in the custom headers it sends when initiating the WebSocket connection.
Haven't tried this myself though.
arccy|1 year ago
hpx7|1 year ago
austin-cheney|1 year ago
I have found for safety you need to allow an arbitrary delay of 100ms before killing sockets to ensure message completion which is likely why the protocol imposes a round trip of control frame opcode 8 before closing the connection the right way.
dilyevsky|1 year ago
What? How would public network even know you’re running a websocket if you’re using TLS? I dont think it’s really possible in general case
> Since SSE is HTTP-based, it's much less likely to be blocked, providing a reliable alternative in restricted environments.
And websockets are not http-based?
What article describes as challenges seems like very pedestrian things that any rpc-based backend needs to solve.
The real reason websockets are hard to scale is because they pin state to a particular backend replica so if the whole bunch of them disconnect at scale the system might run out of resources trying to re-load all that state
pk-protect-ai|1 year ago
atul-jalan|1 year ago
Sytten|1 year ago
atul-jalan|1 year ago
It's a good idea for short-lived HTTP requests, but will cause problems for a persistent connection.
unknown|1 year ago
[deleted]
notatoad|1 year ago
i don't hate them, they're great for what they are, but they're for realtime push of small messages only. trying to use them for the rest of your API as well just throws out all the great things about http - like caching and load balancing, and just normal request/response architecture. while you can use websockets for that it's only going to cause you headaches that are already solved by simply using a normal http api for the vast majority of your api.
andy_ppp|1 year ago
https://hexdocs.pm/phoenix/channels.html
cultofmetatron|1 year ago
been running a phoenix app in prod for 5 years. 1000+ paying customers. heavy use of websockets. never had an issue with the channels systems. it does what it says on the tin and works great right out of the box
dartos|1 year ago
Why would I use websockets over SSE?
sisk|1 year ago
yasserf|1 year ago
For a bit of background, in order to tackle scalability, the initial approach was to explore serverless architecture. While there are both advantages and disadvantages to serverless, a notable issue with WebSockets on AWS* is that every time a message is received, it invokes a function. Similarly, sending a message to a WebSocket requires invoking an HTTP call to their gateway with the websocket / channel id.
The upside of this approach is that you get out-of-the-box scalability by dividing your code into functions and building things in a distributed fashion. The downside is latency, due to all the extra network hops.
This is where vramework comes in. It allows you to define a few functions (e.g., onConnect, onDisconnect, onCertainMessage) and provides the flexibility to run them locally using libraries like uws, ws, or socket.io, or deploy them in the cloud via AWS or Cloudflare (currently supported).
When running locally, the event bus operates locally as well, eliminating latency issues. If you apply the same framework to serverless, latency increases, but you gain scalability for free.
Additionally, vramework provides the following features:
- Standard Tooling
Each message is validated against its typescript signature at runtime. Any errors are caught and sent to the client. (Note: The error-handling mechanism has not yet been given much thought into as an API). Rate limiting is also incorporated as part of the permissioning system (each message can have permissions checked, one of them could rate limiting)
- Per-Message Authentication
It guards against abuse by ensuring that each message is valid for the user before processing it. For example, you can configure the framework to allow unauthenticated messages for certain actions like authentication or ping/pong, while requiring authentication for others.
- User Sessions
Another key feature is the ability to associate each message with a user session. This is essential not only for authentication but also for the actual functionality of the application. This is done by doing a call to a cache (optionally) which returns the user session associated with the websocket. This session can be updated during the websocket lifetime if needed (if your protocol deals with auth as part of it's messages and not on connection)
Some doc links:
https://vramework.dev/docs/channels/channel-intro
A post that explains vramework.dev a bit more in depth (linked directly to a code example for websockets):
https://presentation.vramework.dev/#/33/0/5
And one last thing, it also produces a fully typed websocket client, so if using routes (where a property in your message indicates which function to use, the approach AWS uses serverless).
Would love to get thoughts and feedback on this!
edit: *and potentially Cloudflare, though I’m not entirely sure of its internal workings, just the Hibernation server and optimising for cost saving
yasserf|1 year ago
Worth noting you can share functions across websockets as well, which allows you to compose logic across different ones if needed
tbarbugli|1 year ago
some of the complexity is self-inflected by ignoring KISS principle
atul-jalan|1 year ago