[JonathanFly]

Creating 3D spaces from inconsistent source images! Super fun idea.

I tried a crude and terrible version of something like this a few years ago, but not just inconsistent spaces without a clear ground truth - purely abstract non-space images which aren't supposed to represent a 3D space at all. Transform an abstract art painting (Kandinsky or Pollock for example) into a explorable virtual reality space. Obviously there is no 'ground truth' for whatever 'walking around inside a Pollock painting' means - the goal was just to see what happens if you try to do it anyway. The workflow was:

1. Start From Single Abstract Art Source Image

2. SinGan to Create Alternative 'viewpoints' of the 'scene'

3. 3d-photo-inpainting (or Ken Burns, similar project) on original and SinGan'd images (monocular depth mapping, outputs a zoom/rotate/pan video)

4. Throw 3d-photo-inpainting frames into photogrammetry app (Nerf didn't exist yet) and dial up all the knobs to allow for the maximum amount of errors and inconsistency

5. Pray the photogrammetry process doesn't explode (9 times out of 10 it crashed after 24 hours, brutal)

I must have posted an example on Twitter but I can't find the right search term to find it. But for example, even 2019 tier depth mapping produced pretty fun videos from abstract art: https://x.com/jonathanfly/status/1174033265524690949 The closest thing I can find is photogrammetry of an NVIDIA GauGAN video (not consistent frame to frame) https://x.com/jonathanfly/status/1258127899401609217

I'm curious if this project can do a better job at the same idea. Maybe I can try this weekend.

[localfirst]

What is a technique/library that can take an image of a 3d environment/drawing of a room and detect a rough mesh highlighting ground, walls, barriers ?

[JonathanFly]

> What is a technique/library that can take an image of a 3d environment/drawing of a room and detect a rough mesh highlighting ground, walls, barriers ?

Well just in case it wasn't obvious, Toon3D, the project being discussed, is doing that. Part of the workflow is asking the user to indicate correspondences between geometry in different images, and each image is processed individually to create blocks of geometry you can toggle on or off visually.

Older projects:

https://github.com/sniklaus/3d-ken-burns

https://github.com/vt-vl-lab/3d-photo-inpainting

I believe there are some NeRF variants that do something like this from a single image as well, but I haven't personally tried any.