Rendering a game in real time with AI

[sjsdaiuasgdia]

The "real-time" version looks awful with constantly shifting colors, inconsistently sized objects, and changing interpretations of the underlying data, resulting in what I would consider an unplayable game vs the original ASCII rendering.

The "better" version renders at a whopping 4 seconds per frame (not frames per second) and still doesn't consistently represent the underlying data, with shifting interpretations of what each color / region represents.

[harph]

It seems it's because OP is generating the whole screen every frame / every move. Of course that will give inconsistent results.

I wonder if this approach would work better:

1. generate the whole screen once

2. on update, create a mask for all changed elements of the underlying data

3. do an inpainting pass with this mask, with regional prompting to specify which parts have changed how

4. when moving the camera, do outpainting

This might not be possible with cloud based solutions, but I can see it being possible locally.

[johnfn]

> The "real-time" version looks awful, etc

Dang man it's just a guy showing off a neat thing he did for fun. This reaction seems excessive.

[actuallyalys]

Yeah, as interesting as the concept is, the lack of frame to frame consistency is a real problem. It also seems like the computing requirements would be immense—the article mentions burning through $10 in seconds.

[ozmodiar]

I like the idea behind https://oasis-ai.org/ where you can actually try to take advantage of the 'dream logic' inconsistency of each frame being procedurally generated based on the last one. For example, instead of building a house, build the corner of a house, look at that, then look back up and check if it hallucinated the rest of your ephemeral house for you. Of course that uses AI as the entire gameplay loop and not just a graphics filter. It's also... not great, but an interesting concept that I could see producing a fun dream logic game in the future.

[jschomay]

OP here. Thanks for the feedback. I agree that frame to frame consistency is quite bad currently. I did address that in the post, hinting at some of the techniques others have mentioned here, like in/out-painting and masking previous frames. For me, the exciting parts of this experiment was finding the opportunities and limits of realtime generation, and exploring ways of grounding generated content in a solid yet player controlled world layer.

[faeyanpiraat]

Yeah, but I find this fascinating regardless.

This is getting into the direction of a kind of simulation where stuff is not determined by code but a kind of "real" physics.

[gavmor]

This is awesome! I can definitely see it delivering value, especially over with procedurally generated terrain with 100s or 1000s of different terrain types and combinations—especially if user-defined properties get involved, ie what artists can't predict or prepare for, producing materials via something like https://infinite-craft.gg/

[panki27]

I'm pretty sure the generation could easily run locally on a low-to-mid tier graphics card.

While it might take a bit longer to generate, you're still saving network and authentication latency.

[stego-tech]

…but then you just have a graphics card, built to render graphics, that you could tap instead through traditional tooling that’s already widely known and which produces consistent output via local assets.

While the results of the experiment here are interesting from an academic standpoint, it’s the same issue as remote game streaming: the amount of time you have to process input from the player, render visuals and sound, and transmit it back to the player precludes remote rendering for all but the most latency-insensitive games and framerates. It’s publishers and IP owners trying to solve the problem of ownership (in that they don’t want anyone to own anything, ever) rather than tackling any actually important issues (such as inefficient rendering pipelines, improving asset compression and delivery methods, improving the sandboxing of game code, etc).

Trying to make AI render real-time visuals is the wrongest use of the technology.

[johnisgood]

I do not see how it replaces or substitutes network and authentication latency, especially in terms of a single player game in which neither is necessary.

[echelon]

This was built in September 2022, and it's still pretty mind-blowing today:

https://madebyoll.in/posts/game_emulation_via_dnn/demo/

https://madebyoll.in/posts/game_emulation_via_dnn/

Hook world state up to a server and you have multiplayer.

2025 update:

https://madebyoll.in/posts/world_emulation_via_dnn/

https://madebyoll.in/posts/world_emulation_via_dnn/demo/

[steveruizok]

We did a similar thing at tldraw with Draw Fast (https://drawfast.tldraw.com/) and it was very fun. Inspired a few knock offs too. We had to shut it down because it was getting popular on Russian Reddit. A related project Lens (https://lens.tldraw.com) also used the same technique, but in a collaborative drawing app.

At the peak, when we were streaming back video from Fal and getting <100ms of lag, the setup produced one of the most original creative experiences I’d ever had. I wish these sorts of ultra-fast image generators received more attention and research because they do open up some crazy UX.

[jschomay]

OP here, I remember both of the draw fast and lens demos! I'm pretty sure those were in the back of my subconscious, inspiring me to explore my take on real time rendering. Thanks for sharing your similar experience. I agree, this was a lot of fun to work on, and like one of the other commenters pointed out, experiencing it viscerally is a whole new kind of feeling, even with the consistency issues. I'd also like to see more experiments on what new kinds of UX could be possible with this tech.

[hiatus]

Is there any chance you'd open up the source for those projects so others can play with them?

[echelon]

LCM is what Krea used to gain massive momentum and raise their first $30M.

The tactile reaction to playing with this tech is that it feels utterly sci-fi. It's so freaking cool. Watching videos does not do it justice.

Not enough companies or teams are trying this stuff. This is really cool tech, and I doubt we've seen the peak of what real time rendering can do.

The rendering artifacts and quality make the utility for production use cases a little questionable, but it can certainly do "art therapy" and dreaming / ideation.

[g105b]

I've been trying to achieve the opposite of this project: render scenes in ASCII/ANSI in the style of old BBS terminal games. I've had terrible success so far. All the AI models I've tried only understand the concept of "pixel art" and not ASCII/ANSI graphics such as what can be seen on https://www.bbsing.com/ , https://16colo.rs , or on Reddit's r/ANSIart/ .

If anyone has any tips for how I could achieve this, I would love to hear your ideas.

[pjob]

I suspect you'll have much better luck directly implementing the style you want rather than trying to get a (current gen) AI to reliably generate ANSI art. This video is a nice overview of implementing a variation of the style in a post-processing shader:

https://www.youtube.com/watch?v=gg40RWiaHRY

[elpocko]

Do you mean you want to use AI to generate new scenes in ANSI-art style, or do you mean you want to use AI to render pre-existing scenes as ANSI art?

[ticulatedspline]

like this https://asciiart.club/ ? you don't really need AI for that.

if you have to use AI diffusion models are a poor choice for the target. might be easier to train an LLM to output actual text art.

[sovietmudkipz]

Sci-fi readers who’ve read Ender’s Game will recognize this style of software as similar in concept to the Mind Game Ender Wiggins plays. In the book, the Mind Game renders a game world based on the subject’s mind (conscious, subconscious) in a mechanically similar way to how dreams work for us IRL.

I’m excited for AI rendered games.

[bob1029]

The power consumption of modern gaming is getting a bit out of hand. This AI stuff is taking it to the next level.

Ray tracing and other forms of real time global illumination are extremely resource intensive approaches to lighting a scene. Every client machine has to figure out how to light everything every single frame. Contrast this with baked global illumination where the cost is incurred exactly once and is amortized across potentially millions of machines.

We need more things like baked GI in gaming. This class of techniques makes the development iterations slower and more deliberate, but it also produces a far more efficient and refined product experience. I'd be very interested in quantifying the carbon impact of realtime vs baked lighting in gaming. It is likely a non-trivial figure at scale. Also bear in mind that baked GI is why games like the batman series still look so good in 2025, even when running on period hardware. You cannot replace your art team by consuming more electricity.

[nh23423fefe]

It doesn't seem like moralizing about resources matters for velocity or capability.

> You cannot replace your art team by consuming more electricity

This isn't true.

[Martin_Silenus]

This kind of lazy lamer behaviour is so unlike Vulkan and hackers...

Wait a minute. Where am I?

[johnnyanmac]

Vibe News, of course!

[theknarf]

This is like playing a boardgame while exclusively looking at the board through a snapchat filter. Who wants this?

[cbm-vic-20]

...where "ASCII" means an image made up of a grid of elements from a limited set of glyphs.

[tantalor]

And those glyphs are not ASCII

This is ASCII: https://commons.wikimedia.org/wiki/File:ASCII-Table-wide.svg

[VagabundoP]

How far are we from Ender's Game?

[mason_mpls]

Now dwarf fortress can eat your CPU, Memory, and GPU. Exciting news.

[beacon294]

I thought so too, but looks like the OP wrote their own game that just has a similar ascii view.

However, no reason this can't translate to df!

[devinprater]

Cool, if the game is promptable, one may even be able to make the game accessible.

[babush]

Beautiful

[_1tem]

This is a pre-cursor to a dystopian future where reality will be a game generated in realtime at 60 FPS and streamed to your brain over Neuralink.

[lm28469]

Some people on HN will surely cheer for it, it's the peak of efficiency, you don't even have to move anymore!

[jebarker]

In some ways that’s a lot like how consciousness works isn’t it?

[mason_mpls]

I think that’ll be one of the few good parts of it imho

[coolKid721]

I do not get the point of this at all, why not just generate game assets and run them in an engine? With this format there would be no regularity that the thing you saw before will look the same (and that is not a fixable problem).

Actually figuring out and improving AI approaches for generating consistent and decent quality game assets is actually something that will be useful, this I have no idea the point of past a tech demo (and for some reason all the "ai game" people do this approach).

[sho_hn]

> I do not get the point of this at all

Dunno, this seems like an avenue definitely worth exploring.

Plenty of game applications today already have a render path of input -> pass through AI model -> final image. That's what the AI-based scaling and frame interpolation features like DLSS and FSR are.

In those cases, you have a very high-fidelity input doing most of the heavy lifting, and the AI pass filling in gaps.

Experiments like the OP's are about moving that boundary and "prompting" with a lower-fidelity input and having the model do more.

Depending on how well you can tune and steer the model, and where you place the boundary line, this might well be a compelling and efficient compute path for some applications, especially as HW acceleration for model workloads improves.

No doubt we will see games do variations of this theme, just like games have thoroughly explored other technology to generate assets from lower-fidelity seeds, e.g. classical proc gen. This is all super in the wheelhouse of game development.

Some kind of AI-first demoscene would be a pretty cool thing too. What's a trained model if not another fancy compressor?

[corysama]

Here’s a recent demo made by researchers at Nvidia trying to render the most detailed, realistic scene possible using AI tech to a small degree -mostly as asset compression, but also in material modeling beyond what direct approaches can currently handle in real time: https://m.youtube.com/watch?v=0_eGq38V1hk

Here’s a video from a rando on Reddit conveniently posted today after playing around for an afternoon: https://www.reddit.com/r/aivideo/comments/1n1piz4/enjoy_nano...

The Nvidia team carefully selected a scene to be feasible for their tech. The rando happened to select a similar-ish scene that is largely well suited for game tech.

Which scene looks more realistic? The rando’s by far. And, adding stylization is some AIs are very well established as being excellent at.

Yes, there are still lots of issues to overcome. But, I find the negativity in so many comments in here to be highly imbalanced.

[dagi3d]

hack, learn and have fun, that's it.

[abbycurtis33]

The tech will improve to far exceed the capabilities of a game engine. Real time improvisation and infinite choices, scope, etc.

It makes no sense when people say AI can't do this or that. It will do it next week.

[Workaccount2]

> (and that is not a fixable problem)

Genie 3 is incredible (relative to previous models) in this regard. Not a solve, but it is doing what it is not supposed to be able to do.[1]

[1]https://deepmind.google/discover/blog/genie-3-a-new-frontier...

[turnsout]

It's an interesting tech demo—I think one interesting use case for AI rendering is changing the style on the fly. For example, a certain power-up could change the look to a hyper-saturated comic book style. Definitely achievable with traditional methods, but because AI is prompt-based, you could combine or extend styles dynamically.

[127]

Visually speaking, there's always visual issues in tying disparate assets together in a seamless fashion. I can see how AI could be easily used to "hide the seams" so to speak. I think a hybrid approach would be an improvement definitely.

[elestor]

> I do not get the point of this at all

I think the point of this is just because it's cool. So, as you said, it only serves as a tech demo, but why not? Many things have no point. It's unreasonable, but it's cool.

[doawoo]

[flagged]

[mediaman]

It's interesting to see software engineers realize that AI can be useful in the hands of competent engineers, but that LLMs tend to produce a mess in the hands of those with little software engineering knowledge. Then they're confused why generative AI asset creation doesn't look that good in the hands of people who have no art training.

I know someone creating a game, and she is using AI in asset creation. But she's also a highly skilled environment technical artist from the industry, and so the use of AI looks totally different than all these demos from non-artists: different types of narrow AI get injected into different little parts of the asset creation pipeline, and the result is a mix of traditional tools (Substance Designer + Painter, Blender, Maya) with AI support in moodboarding, geometry creation and some parts of texture creation. The result is a 2-5x speedup, but instead of looking like slop it looks like a stylistically distinctive, cohesive world with consistent art direction.

The common pattern is that people think AI will automate "other people," because they see its shortcomings in their own field. But because they don't understand the technical skill required in other fields, they assume AI will just "do it." Instead, it seems like AI can be a force multiplier for technically skilled people, but that it begins showing its weakness when asked to take over entire pipelines normally produced by technically skilled people, whether they be engineers or artists.