top | item 37931373

Reflect – Multiplayer web app framework with game-style synchronization

462 points| aboodman | 2 years ago |rocicorp.dev

150 comments

order
[+] linux2647|2 years ago|reply
The demo at the top of the homepage (https://reflect.net/) is a lot of fun. While I was watching, every time the puzzle got completed, people would shake their cursor in excitement, like saying, "yay we did it!!"
[+] white_dragon88|2 years ago|reply
The first thing I did was start making the word 'FUCK' with the pieces. As soon as others cottoned on and joined in, it was a very heart warming experience.
[+] automatoney|2 years ago|reply
There's a visual bug where you can hide pieces of the 'e' behind the completed portions of the 'e' - it doesn't work for other letters because they still show the outline.
[+] zebracanevra|2 years ago|reply
I don't think this demo is a good demonstration of the features this library can offer. Once you grab a piece, you seem to lock it for only you to use? This means there are no conflicts to resolve, the only conflict is when two people pick a piece at the same time, from which the only resolution is to give it to the user that picked the piece first.

I'd want a demo that shows a much better example of conflicts occuring.

[+] namelosw|2 years ago|reply
Man, they nailed it with this page!
[+] FrostKiwi|2 years ago|reply
Seems to run worse on Firefox than on Chromium / Edge / Chrome. Rather choppy, mouse cursor lagging behind and laptop fans spinning up. Smooth on Chromium and friends though.
[+] qup|2 years ago|reply
Granollers!!!
[+] tablatom|2 years ago|reply
People might remember this showing up once or twice before as Replicache. Reflect adds a fully managed, incredibly fast sync server.

The local-first/realtime space is getting busy these days. Replicache/Reflect is well worth checking out for the beautifully simple data model / coding model. I'm very happy with it.

One plus point compared to CRDTs is that you get to decide how you want to deal with conflicts yourself, with simple, sequential code. I'm not up to date with the latest in CRDT-land but I believe it can be complicated to add application specific conflict resolution if the built-in rules don't fit your needs.

[+] ochiba|2 years ago|reply
> One plus point compared to CRDTs is that you get to decide how you want to deal with conflicts yourself, with simple, sequential code. I'm not up to date with the latest in CRDT-land but I believe it can be complicated to add application specific conflict resolution if the built-in rules don't fit your needs.

I agree wholeheartedly — we took the same approach for PowerSync with a server reconciliation architecture [1] over CRDTs, which is also mentioned elsewhere in the comments here. For applications that have a central server, I think the simplicity of this kind of architecture is very appealing.

Props to Aaron for bringing awareness to and evangelizing server reconciliation.

[1] https://www.gabrielgambetta.com/client-side-prediction-serve...

[+] moralestapia|2 years ago|reply
I worked extensively on this (also implemented it on JS) about 15 years ago, to escape boredom while on my grandma's place. I wish I had open sourced something, but back then I was a young boy with no ulterior motivations :^).

Anyway, I want to comment on this:

>For example, any kind of arithmetic just works:

Of course, it is extremely trivial to set those up as idempotent operations.

>List operations just work:

Nope ... or at least, I'd have to take a closer look at your implementation, consider:

Some sort of array/list that looks like: [1, 2, 3, 4, 5]

* User A deletes the range from 2-4.

* User B deletes the range from 3-5.

* User C inserts (uh-oh) something between 3 and 4.

All updates reach the server at the same time. What is the solution to that? Timestamps? Fallback to LLW? This breaks the transactional model.

Pick any of those updates as "the winner", what do the other users see? How do you communicate this to them?

If your answer is like "we just send them the whole array again", this breaks the transactional model (x2).

[+] aboodman|2 years ago|reply
Hi! Thanks for the thoughtful question.

>List operations just work:

Fair point. Will change. What I really meant here was "(Many) list operations just work" (just like above I said "all kinds of things just work".

> All updates reach the server at the same time. What is the solution to that? Timestamps? Fallback to LLW?

There are a number of issues you might be pointing out, and I'm not sure which one you mean.

In general, you can't use list indexes as identifiers for edit/delete in Reflect because they aren't stable. That's why in this example I commented to use the item itself (if atomic) or more likely a stable ID: https://i.imgur.com/IKzmf0q.png

There is also the potential issue of User C's inserts getting deleted. This seems like a problem at first, but in a realtime collaborative context, nothing can fix this. User C's insert can land just before User A or B's delete. In this case, sync would be correct to delete user C's insert, no matter the protocol. But User C might still be sad. From her perspective, she just wrote something and it got deleted. This is the nature of two humans working in the same place at the same time and potentially having different intentions. For these problems, undo and presence indicators go a long way to avoid these problems through human/social mechanisms.

[+] philsnow|2 years ago|reply
You don’t even need to involve deletes; if all of these happen “simultaneously”:

  - client A appends “a” to list L
  - client B appends “b” to list L
  - client C appends “c” to list L
the server will arbitrarily apply the appends in some order, and then send the result back to the clients. But, each of the clients has applied its append locally and is showing those results until it gets different information from the server, so the next time step might be

  - client A thinks L is [“a”]
  - client B thinks L is [“b”]
  - client C thinks L is [“c”]
  - server sends state event to clients saying that L is [“c”,”b”,”a”]

Then when the clients receive the state from the server, I guess they all discard their pending mutations and use the state from the server as their state of the world. But what if each of the three appends results in that client winning the game if their mutation gets applied first? Do all the clients display “you win” for 300ms while waiting for the server’s update?
[+] gizmo|2 years ago|reply
Most of these systems do deletes with tombstones. Then you can safely insert after/inbetween deleted items.
[+] pmontra|2 years ago|reply
In a collaborative application the server could resolve the three operations in any order. The three users observe the result, realize that they are operating on the same data at the same moment and messed with the state. Then they fix it. Think about editing a document.

If they can't see each other working on the data together they could be surprised but they could eventually fix the data to the desired state.

If they don't check the result or can't check the result, the state won't be OK. However it's not what probably happens in interactive apps like collaborative editing or games.

[+] aboodman|2 years ago|reply
Hey hacker news! I'm one of the people behind this, happy to answer any questions.
[+] coreyh14444|2 years ago|reply
Aaron is humble, so I'll do it for him! He created Greasemonkey and has had a hand in several groundbreaking browser innovations over the years.
[+] jayunit|2 years ago|reply
Congrats! I've been watching this space for a while, having built a couple multiplayer sync systems in the past in private codebases, including a "redux-pubsub" library with rebasing and server canonicity that is (IIUC?) TCR-like. There's a lot to like about this model, and I find the linked article quite clear - thank you for writing and releasing this!

1. You wrote "For example, schema validation and migrations just sort of fall out of the design for free." - very curious to read about what you've found to work well for migrations! I feel like there's a lot of nice stuff you can build here (tracking schema version as part of the doc, pushing migration functions into clients, and then clients can live-update) but I never got the chance to build that.

2. Do you have a recommendation for use-cases that involve substantial shared text editing in a TCR system? I'd usually default to Yjs and Tiptap/Prosemirror here (and am watching Automerge Prosemirror with interest). The best idea I've come up with is running two data stores in parallel: a CRDT doc that is a flat key/value identifying a set of text docs keyed by UUID, and a TCR doc representing the data, which occasionally mentions CRDT text UUIDs.

[+] nodoodles|2 years ago|reply
What does this mean for Replicache development, is the client-side codebase mostly shared and can expect continued updates, or more likely to replace the primary focus for you?
[+] woggy|2 years ago|reply
Curious what your thoughts are on ElectricSQL?
[+] chrisco255|2 years ago|reply
What is the maximum number of users that can be concurrently in a room?
[+] hgs3|2 years ago|reply
This is a pedantic comment, but the terminology is confusing. Games have "players" so their sync systems are called "multiplayer," but regular software has "users" so their sync systems should be called "multiuser." The page reads strangely mixing the terms "user" with "multiplayer."
[+] nwenzel|2 years ago|reply
Maybe. But user is like a consumer. So HN is multiuser, but seems weird to call HN multi-player. Live concurrent interactions feels like something more than multiuser. Just my 2-cents, but multiplayer as in multiple users acting and interacting is a good use of the term multiplayer.
[+] mrkeen|2 years ago|reply
Its logic is based on the logic that's been in "multiplayer games" for decades.

On the other hand all web software is "multiuser", which doesn't tell you anything.

[+] itjustisthough|2 years ago|reply
That’s just the terminology these days. Multiplayer == “live” multi-user including visualizing what other users are doing at every moment.
[+] lgessler|2 years ago|reply
As someone who's casually been eyeing up CRDTs for two years I've wondered continually what the story for authorization is, and this article seems to suggest (in line with my own understanding) that a CRDT library on its own like Y.js can't really handle applications where mutations need to be checked for authorization because it lacks a central authority (i.e., the server) while Reflect assumes that a server will be mediating client interactions. Is this all correct?
[+] aboodman|2 years ago|reply
"Can't" is a strong word. You can get auth with a CRDT with effort, for example you can put a server in the middle of everything and have the server reverse any changes it sees which are unauthorized.

This ends up being a lot of work to maintain and easy to break as the application gets bigger, and it also defeats some of the benefits of the CRDT in the first place (now the server has to mediate everything and you can't have peer-to-peer sync).

Also if there are side-effects of any actions which require auth which aren't undoable, then that gets more complicated.

If you already have a server in the middle, it's a lot simpler to just use a protocol that allows the server to reject messages in the first place.

[+] foota|2 years ago|reply
CRDTs can still involve a server, just not necessarily in the middle of everything. Consider you could have connections authenticated by the server, for one.

E.g., someone connects peer to peer to you and claims some privilege etc., you could use the server to verify their claim.

Also, I don't think CRDT necessarily implies peer to peer, as you can use a central server for message passing, but keep the CRDT model for resolving the current state on both server and client.

[+] hem777|2 years ago|reply
One way to do authorization is to sign each operation/message and then verify the signature to match a public key in an access control list. This also enables CRDTs to work in a peer to peer context.

But as aboodman says in a sibling comment, if there’s a server as an authority, it can simply reject messages from unauthorized clients.

[+] stu_k|2 years ago|reply
How does one handle an upgrade to mutators? If a client is running old code then the operation will differ to the server. Obvious answer would be to version them independently: `increment_v1`, `increment_v2`, but wondering if there is a better answer?
[+] erikarvidsson|2 years ago|reply
At the moment there is no better answer than do not remove your old mutators until you know there are no more clients out there that might have pending changes.

Right now that window is pretty small because we haven't turned on persistence yet.

[+] joloooo|2 years ago|reply
Looks great, congrats on launching. Is this in a similar vein to https://partykit.io/?
[+] aboodman|2 years ago|reply
Yes they are in the same space. The key difference is in how opinionated each is.

PartyKit is extremely unopinionated. It's essentially lightweight javascript server, that launches fast and autoscales (I don't say this as as a bad thing, it's a useful primitive). Most people seem to run yjs in PartyKit, but you can also run automerge or even Replicache – my company's other project.

Reflect is entirely focused on providing the best possible multiplayer experience. We make a lot of choices up and down the stack to tightly integrate everything so that multiplayer just works and you can focus on building your app.

That's the plan anyway :).

[+] wheybags|2 years ago|reply
For reference, this strategy is called "deterministic lockstep" in the gamedev world, and is used especially often in games which have many entities which need their state synced. The canonical example is rtses, which pretty much all use it.
[+] Oberdiah|2 years ago|reply
Specifically "Deterministic Rollback", as the clients don't wait for the server's response before displaying the change locally.

For an in-depth explanation, see this excellent GDC talk on its implementation in Mortal Kombat and Injustice 2: https://youtu.be/7jb0FOcImdg

[+] DecoPerson|2 years ago|reply
Hey wheybags, this is not deterministic lockstep.

That is an algorithm for peer-to-peer games, where each peer waits for the input from all other players before advancing the game simulation. The deterministic part is because players share inputs (not simulation results) AND the simulation is deterministic for the same inputs. The lockstep part is because all clients advance at a coordinated pace. The Age of Empires series use this approach, and that’s why units don’t move immediately when you click. Starcraft uses this too, but it has some tricks to smooth the gameplay experience. Both are peer-to-peer with no single “server”.

In the case of Reflect, we have a server-authorative simulation (not peer-to-peer). Clients send their inputs to the server, but they do not wait for the result, they instead predict the result locally without confirmation from the server. The server also rolls back time and then replays inputs to compensate for individual client latency. And the client corrects/reconciles their local simulation once the server sends a simulation result that included one of their inputs.

The keywords for this algorithm are:

- Server-authoritative

- Predicted

- Lag compensated (simulation rollback / server rewind / input replay)

- Prediction reconciliation (local simulation rollback / local input replay / misprediction smoothing)

I’m not sure if Reflect has it, but client-side interpolation is also a common feature in FPS games, where upon receiving a world update, the client will tween entities to their new positions/rotations over a fixed interval (such as 0.1s). This allows you to send updates to clients at only 10Hz but have entities move smoothly (without needing the client to locally simulate the physics of every entity).

There is only one authority, the server, so determinism isn’t super important. However, it is nice to have for client predictions to reduce the occurrences of mispredictions. Mispredictions occur when the server state did not advance the way the client predicted. These can happen when:

(1) Another client’s input changed the world state in an important way,

(2) The simulation is not deterministic (e.g. the random number generation is not synced).

In FPS games, (1) is impossible to eliminate. These mispredictions are often smoothed using interpolation. (2) should be minimised, but may actually be desirable. For example, Counter Strike does not sync the RNG for bullet spread randomness, to prevent “nospread” cheat programs from predicting where each bullet will go and instantly adjusting the player’s aim so the bullet lines up perfectly.

https://www.gabrielgambetta.com/client-side-prediction-live-...

https://developer.valvesoftware.com/wiki/Latency_Compensatin...

https://developer.valvesoftware.com/wiki/Source_Multiplayer_...

[+] antidnan|2 years ago|reply
Reflect is awesome. We're currently using an alpha version in production, and I've been really happy with the system as well as Aaron and team!

Happy to answer any questions as a customer :)

[+] anonzzzies|2 years ago|reply
Besides trystero which is great but depends on things I cannot get into our company, anything open source? I cannot get this into our company either as it's vital infra and as such needs to be replaceable with open source at will. We have that as a rule. The only things that are exceptions are necessary evil; payments & cloudflare (bot-fight specifically).
[+] matlin|2 years ago|reply
While I do love the simplicity of this approach on the surface. Some of the complexity gets brushed under the rug.

Firstly is rebasing these operations in this style of system takes a bigger hit on performance than I expected in my experience. Sure, managing counters and doing integer math can consistently run at 120FPS but more complex operations can really bog down the system when a lot of users are interacting with it once.

The second challenge is that when operations fail (e.g. for authorization) undoing that operation isn't always straightforward. You either pop the operation and replay ops from a earlier snapshot (which necessitates storing all ops in order), writing an inverse operation manually, or having the server send the entire state back to rebase your unconfirmed ops on which can also end being expensive.

In general, the challenges of this approach are true for all operation-based CRDTs too. State-based approaches are a good deal simpler just tend to over-send data over the network.

[+] aboodman|2 years ago|reply
Hey Matlin, it's true that this approach is harder to implement, but that's a one-time cost, and it's one we take on for our users and they don't have to worry about.
[+] cco|2 years ago|reply
Wow this looks incredible. I had the most fun I've had on a computer in a long time this last weekend using Partykit.io to build https://is-sf-back.cconeill.partykit.dev/ and this looks so much smoother.

Going to have to check this out.

[+] holoduke|2 years ago|reply
When will it finally be possible to use UDP protocol within browsers? TCP got way too big overhead. A multiplayer engine in which you send 30 messages per second for 128 users is what i need. Not possible with websockets withput having super computers.
[+] cornfutes|2 years ago|reply
Looks great for a hobby project.

Has anyone done the cost + risk assessment of building a for-profit product on top of this? Would love to know, as I am working on a web IDE with collaboration. There’s also the matter of obscuring client data from 3rd and even 1st party.

[+] reactordev|2 years ago|reply
Don’t call it multiplayer, call it multiuser or concurrent user. Player assumes it’s a game. It’s not a game. It’s very very real. We are the agents. Cool project. Collaboration is hard. Synchronicity even harder. This makes steel become butter.
[+] wim|2 years ago|reply
Very elegant solution, and neat demo page!

I'm curious what trade-offs you looked at regarding sending state vs mutations from the server back to clients. As the server is authoritative, I imagine it could also send mutations to the clients in the order in which the server has determined parallel operations need to be resolved (and adjust mutations as needed to resolve conflicts or permissions issues). Clients would then need to rebase any pending local mutations, so that would involve more logic vs the data sent for mutations most likely being smaller than the "full" state of certain objects being updated?

[+] jauntywundrkind|2 years ago|reply
Very allagmatic. Instead of storing & transferring structures transfer operations.

Iw as wondering what kind of checkingpoint or snapshotting there might be. Such that one doesn't have to "replay-the-world" if something goes awry (localstack persistence did/l (does?) this, storing state by just recording/replaying API calls). It sort of seems like having a globally accepted state is enough & makes sense, but it still seems like work to rollback & do again, still seems to require snapshotting.

Something like this feels like a 100% fit for having immutable data structures. Where you can snapshot the world very very quickly/at low cost.