[atseajournal]

Animation has to be the most intriguing hobby I'm never planning on engaging with, so this kind of blog post is great for me.

I know hand-drawn 2D is its own beast, but what's your thought on using 3D datasets for handling the occlusion problem? There's so much motion-capture data out there -- obviously almost none of it has the punchiness and appeal of hand-drawn 2D, but feels like there could be something there. I haven't done any temporally-consistent image gen, just playing around with StableDiffusion for stills, but the ControlNets that make use of OpenPose are decent.

3D is on my mind here because the Spiderverse movies seemed like the first demonstration of how to really blend the two styles. I know they did some bespoke ML to help their animators out by adding those little crease-lines to a face as someone smiles... pretty sure they were generating 3d splines however, not raster data.

Anyway, I'm saving the RSS feed, hope to hear more about this in the future!

[yosefk]

The 2nd paper actually uses a 3D dataset, though it explicitly doesn't attempt to handle occlusion beyond detecting it.

I sort of hope you can handle occlusion based on learning 2D training data similarly to the video interpolation paper cited at the end. If 3D is necessary, it's Not Good for 2D animation...

AI for 3D animation is big in its own right; these puppets have 1 billion controllers and are not easy for humans to animate. I didn't look into it deeply because I like 2D more. (I learned 3D modeling and animation a bit, just to learn that I don't really like it...)

[wongarsu]

Maybe there is also value in 2d datasets that aren't hand drawn. A lot of TV shows are made in Toon Boom or Adobe Animate (formerly Macromedia Flash). Those also do automatic inbetweening, but with a process that's closer to CSS animations: everything you want to move independently if it's own vector that can be moved, rotated and squished, and the software just interpolates the frames in between with your desired easing algorithm. That's a lot of data that's available on those original project files that's nontrivial to infer from the final product

[yosefk]

I doubt you can learn much out of tweened flat cutouts beyond fitting polynomials to data points. The difficulty with full animation is rotation & deformation you can't do at all with cutouts. (Puppet warp/DUIK cutouts are much less rigid than Flash but the above still applies)