I work on a game engine (see profile) - I added a input recording/replay system that tracks a hash for each graphics frame for regression testing. We have ~17M frames of tests (80 hours of real playtime), and when something fails it generates a custom HTML report[1] where you can navigate the failing frames and some frames before/after for context. It's just graphical, so if the error isn't obvious you must delve into a debugger (and maybe wait ~10m for the engine to reach that point...), so I'd love to get something more powerful to assist in debugging.
rerun looks very interesting, I'll check it out to improve our debugging, thanks for sharing!
What's the performance of this like? It seems really appealing. I would love to be able to use it to debug https://github.com/MeoMix/symbiants because I use RNG heavily to add variance to the world and that, combined with indeterminate execution order of systems, can really leave me scratching my head sometimes.
However, I'm building using a tilemap that's 144x144. So I've got ~21000 entities to log. It seems impractical to snapshot the world every tick, but maybe if it were able to snapshot deltas or something?
Revy already works with snapshot deltas (see other comments scattered around this section for more details, but basically we only sync components that changed during the previous frame -- Rerun stitches everything back together at runtime)... but at 21k entities, I'm afraid you'll be facing much bigger issues on the Rerun-side of things :D
Rerun was originally designed for few (i.e. dozens up to hundreds) massive entities (e.g. it's common for a single entity to have a few million 3D points and color values attached to it).
While we're slowly working towards improving the many-entities use-case, the correct thing to do in this case would probably be for Revy to identify that all these entities are really just different instances of the same batch (either automatically, or by exposing a marker component or something).
So, say, you'd set a marker component on all your tiles, Revy would then snapshot them as a single batch of 144^2 instances, and then in Rerun you'd see a single entity `/tiles` which would be a batch of 144^2 instances (each with their own set of components, that's fine!).
From Rerun's point-of-view, this would be similar to a point cloud, and at 21k instances you'd be easily running at your monitor refresh rate with a lot of margin.
But by any means, try it! Not the web version though, you're definitely going to need multithreading :D
Nice project btw; I'll keep an eye on it and probably use it as a benchmark for the many-entities use-case!
Is a generic time travel debugging solution too much overhead? A good multithreaded time travel debugger (not deterministic replay based) should only incur ~100% overhead in the memory bandwidth bound case. If you are not saturating your memory bus without instrumentation then the overhead should be proportionally less.
Where basically the ECS boils down to what is essentially a relational database, and here it looks like that's being leveraged to do snapshotting and point-in-time queries!
Oh for sure, there's a lot of overlap between traditional relational databases and ECS designs. As always, in the end the hard part is to match the performance requirements.
If you squint enough, most ECS out there are pretty much very specialized relational databases that trade off flexibility in favor of performance for common gamedev use cases (very wide joins, very deep hierarchies (e.g. transform trees), full-table filters, etc).
Rerun's ECS goes one step further and makes time a first-class citizen, allowing for efficient joins across different components across different timestamps.
This is what makes it possible to only log diffs in Revy (we only snapshot the components that were modified during the last frame), rather than having to full snapshots every frame, which would be prohibitively expensive (both time and space). Rerun then stitches back everything together during visualization, in real-time!
Given that Bevy’s systems scheduler is nondeterministic (for everything not explicitly ordered), do you foresee issues coming from that? Or does this approach sidestep that as an issue?
Basing the recording off it's 24.6s runtime and 3.58MB download size it comes out about 145KB/s which is honestly really decent.
What format is it stored as (eg protobuf etc?) Is Rerun doing compression on the "raw" game data to achieve that?
Also had a good laugh at the cranky job in bacon.toml, might steal that :D
I also noticed the elements in the recording were all clickable, is that a Rerun feature, and did you have to manually reconstruct all the Bevy elements in a Rerun specific format?
Just curious how long this took you to make?
Super cool demo btw :) do you have more demos/PoC/examples listed somewhere I could peruse? Cheers
EDIT: Can your twitter post this so I can retweet :D
This is really awesome! I recently picked up Bevy and Rust to resume my attempt at making games and hopefully publishing something worthwhile. This is something that I felt was missing since day 2 of learning Bevy.
My own personal workaround have been to dump "user actions" to a ndjson file, which I can load at runtime when I want a "replay" but obviously missing being able to move forward/backwards, it just plays the actions.
Would love to see it working with bevy_xpdb, although I'm not sure how deterministic it is and if that gets in the way (I assume so?), it does have a `enhanced-determinism` flag that says "Enables increased determinism", but the lack of "complete/full determinism" terms doesn't give me a lot of hope.
Whether the physics engine is deterministic or not doesn't matter here -- Revy (and more importantly, Rerun) doesn't replay anything: it just stores state, every single frame, and then visualizes that state at every timestamp available.
Check out e.g. the live demo of the breakout example for example [1]: if you click on the pallet and then go to its parent node, you'll see that we just store that node's final transform (i.e. post-physics) every frame.
Well, floating point isn't deterministic in its rounding on a per-op scale. And doing tonnes and tonnes of floating point operations isn't going to help that. The only way to get deterministic floating point is to start tracking state at boot and not have a scheduler switch away and mess with FPU state while your program isn't in control.
'TIme travel'. Ah, capturing the state and rolling back. Something a Z-Machine interpreter had 40 years ago I think, if not more, with the 'undo' command at the prompt :D.
One day the OS' shells will have an undo command for everything, but they will waste tons of CPU cycles. And not by virtualizing. Altough if you run your OS under a light hypervisor such as xen, that funcionality might be able to be called from the userland and some kernel driver/hardware hook. Who knows.
Does Revy depend just on the ECS crate, or does it bring in other parts of Bevy? I see a blanket dep onto bevy, but is it really using more than the ECS? I like the idea. I might try it out.
I've been playing with Bevy the last couple weeks, and in general from my first impression I'd have to say that the bevy_ecs crate seems more mature than the rest of it. It's not a bad ECS framework, and actually quite useful independent of Bevy itself. I'd like it if they cleaned up their crates deps a bit, but it's pretty good standalone and not just for games, but for any concurrent data driven application.
ECS has weird nomenclature when viewed outside of the games industry. What it really has if you pan out, is queries and binary relations/tables/facts/properties, but calls them 'systems' and 'components'. "Components" outside of games & ECS usually means something else, so it's a bit of a head scratcher at first.
I think if you dig past the surface what you actually have is a high performance version of what we used to call "tuple spaces", a good model for managing state in parallel data-driven applications, esp where there's lots and lots of bits of state (e.g. vehicle autonomy with vision detection, or robotics, etc.)
I think that impression reflects on the priorities/ordering that the Bevy team has. They seem to have focused on building a very solid foundation of an ECS framework before focusing elsewhere, so the lower-level stuff is going to be more polished than the higher-level stuff (which I think they have now moved their focus onto)
[+] [-] hoten|2 years ago|reply
rerun looks very interesting, I'll check it out to improve our debugging, thanks for sharing!
[1]
https://hoten.cc/tmp/compare-report-example/ , an artificial failure where I commented out the player drawing code
https://hoten.cc/tmp/compare-report-screen-draw-refactor/ , a report for reviewing a subtle refactor and intentional modification of the drawing order
[+] [-] SeanAnderson|2 years ago|reply
However, I'm building using a tilemap that's 144x144. So I've got ~21000 entities to log. It seems impractical to snapshot the world every tick, but maybe if it were able to snapshot deltas or something?
[+] [-] teh_cmc|2 years ago|reply
Rerun was originally designed for few (i.e. dozens up to hundreds) massive entities (e.g. it's common for a single entity to have a few million 3D points and color values attached to it).
While we're slowly working towards improving the many-entities use-case, the correct thing to do in this case would probably be for Revy to identify that all these entities are really just different instances of the same batch (either automatically, or by exposing a marker component or something).
So, say, you'd set a marker component on all your tiles, Revy would then snapshot them as a single batch of 144^2 instances, and then in Rerun you'd see a single entity `/tiles` which would be a batch of 144^2 instances (each with their own set of components, that's fine!). From Rerun's point-of-view, this would be similar to a point cloud, and at 21k instances you'd be easily running at your monitor refresh rate with a lot of margin.
But by any means, try it! Not the web version though, you're definitely going to need multithreading :D
Nice project btw; I'll keep an eye on it and probably use it as a benchmark for the many-entities use-case!
[+] [-] Veserv|2 years ago|reply
[+] [-] LarsDu88|2 years ago|reply
Where basically the ECS boils down to what is essentially a relational database, and here it looks like that's being leveraged to do snapshotting and point-in-time queries!
[+] [-] teh_cmc|2 years ago|reply
If you squint enough, most ECS out there are pretty much very specialized relational databases that trade off flexibility in favor of performance for common gamedev use cases (very wide joins, very deep hierarchies (e.g. transform trees), full-table filters, etc).
Rerun's ECS goes one step further and makes time a first-class citizen, allowing for efficient joins across different components across different timestamps.
This is what makes it possible to only log diffs in Revy (we only snapshot the components that were modified during the last frame), rather than having to full snapshots every frame, which would be prohibitively expensive (both time and space). Rerun then stitches back everything together during visualization, in real-time!
[+] [-] tbillington|2 years ago|reply
[+] [-] teh_cmc|2 years ago|reply
Feel free to ask me anything!
[+] [-] ordinaryradical|2 years ago|reply
[+] [-] tbillington|2 years ago|reply
What format is it stored as (eg protobuf etc?) Is Rerun doing compression on the "raw" game data to achieve that?
Also had a good laugh at the cranky job in bacon.toml, might steal that :D
I also noticed the elements in the recording were all clickable, is that a Rerun feature, and did you have to manually reconstruct all the Bevy elements in a Rerun specific format?
Just curious how long this took you to make?
Super cool demo btw :) do you have more demos/PoC/examples listed somewhere I could peruse? Cheers
EDIT: Can your twitter post this so I can retweet :D
[+] [-] mysterydip|2 years ago|reply
[+] [-] diggan|2 years ago|reply
My own personal workaround have been to dump "user actions" to a ndjson file, which I can load at runtime when I want a "replay" but obviously missing being able to move forward/backwards, it just plays the actions.
Would love to see it working with bevy_xpdb, although I'm not sure how deterministic it is and if that gets in the way (I assume so?), it does have a `enhanced-determinism` flag that says "Enables increased determinism", but the lack of "complete/full determinism" terms doesn't give me a lot of hope.
[+] [-] teh_cmc|2 years ago|reply
Check out e.g. the live demo of the breakout example for example [1]: if you click on the pallet and then go to its parent node, you'll see that we just store that node's final transform (i.e. post-physics) every frame.
Happy gamedev!
[1] https://app.rerun.io/version/0.14.1/index.html?url=https://s...
[+] [-] 3836293648|2 years ago|reply
[+] [-] anthk|2 years ago|reply
One day the OS' shells will have an undo command for everything, but they will waste tons of CPU cycles. And not by virtualizing. Altough if you run your OS under a light hypervisor such as xen, that funcionality might be able to be called from the userland and some kernel driver/hardware hook. Who knows.
[+] [-] mathteddybear|2 years ago|reply
https://arxiv.org/abs/cs/0310016
https://www.youtube.com/watch?v=xpI8hIgOyko
Soon later, "debugging backwards in time" morphed into "time-travel debugging"
[+] [-] cmrdporcupine|2 years ago|reply
I've been playing with Bevy the last couple weeks, and in general from my first impression I'd have to say that the bevy_ecs crate seems more mature than the rest of it. It's not a bad ECS framework, and actually quite useful independent of Bevy itself. I'd like it if they cleaned up their crates deps a bit, but it's pretty good standalone and not just for games, but for any concurrent data driven application.
ECS has weird nomenclature when viewed outside of the games industry. What it really has if you pan out, is queries and binary relations/tables/facts/properties, but calls them 'systems' and 'components'. "Components" outside of games & ECS usually means something else, so it's a bit of a head scratcher at first.
I think if you dig past the surface what you actually have is a high performance version of what we used to call "tuple spaces", a good model for managing state in parallel data-driven applications, esp where there's lots and lots of bits of state (e.g. vehicle autonomy with vision detection, or robotics, etc.)
[+] [-] rcxdude|2 years ago|reply
[+] [-] tbillington|2 years ago|reply
What specifically do you mean by that, as in it includes unnecessary dependencies?
Flecs is also a great library to check out if you want more relations/power.
[+] [-] jasonjmcghee|2 years ago|reply
Awesome stuff.
[+] [-] teh_cmc|2 years ago|reply
[+] [-] poderosoTOTO|2 years ago|reply
[deleted]