[quyse]

I've been working on a cloud gaming service (like Stadia). Wanted to see how far I can get it done using open source, without ready-to-use solutions like Parsec/Moonlight/Sunshine.

It works by running a game in Linux (I use NixOS btw) under Wayland (sway), capturing the frames via Pipewire in form of DMAbufs and passing them to ffmpeg's VA-API encoder (so frames don't leave GPU memory and are encoded on GPU right away), and finally sending encoded packets through WebRTC media stream to a web client. Inputs from a client are sent back to the server via WebRTC data channel and injected into Wayland.

Running the prototype over local network displays zero perceivable latency. (Of course when playing on a remote AWS server the latency is visible as expected). Pleased with the result so far, although it's my first experience with Pipewire, VA-API, and WebRTC, so my implementation is probably far from optimal.

Overall, very impressed by WebRTC - such a powerful thing right in every browser. Continued to be amazed by NixOS - my AWS AMI is NixOS-based and can be built and rebuilt with granular caching, with a single `nix build` command. Also Terraform/OpenTofu - just makes it all possible deploy-wise. So much good stuff exists!

[not_a_dane]

I think you also need to exchange some metadata like screen resolution etc, right ?

[quyse]

Currently resolution is hardcoded, but sure, can be done. Also need to figure out things like changing language/keyboard layout or using IME, to be able to type text properly. Sending mouse cursor picture from server to client may be nice too, in case a game uses hardware cursor and changes it's appearance, like some RTS games - currently mouse cursor is just being baked into video frames. Many possible improvements!

[xyzsparetimexyz]

Wow that's dope! Is this on GitHub anywhere?

[quyse]

Not yet, but I'll probably open source it eventually! Still need to clean things up a lot, and implement missing functionality, for example I haven't even bothered to implement audio capture yet, because I wanted to try video first