React Renderer for Three.js

[miklosme]

When AR/VR finally happens, UI developers will have to deal with complexity from a completely different paradigm. For me, React's biggest strength has always been its ability to organize complexity into a manageable order. Combine this with the large pool of developers and extensive ecosystem, I think React will be the go-to tool for AR/VR apps. For this reason, I'm super hyped for R3F.

[tuckerconnelly]

I really don't think React will be the go-to tool for VR; it's based on the DOM and trees of function calls, which are both hierarchical, which necessarily means you have the gorilla-banana problem.

If you have a coffee cup on a table in VR, is that coffee cup a child of the table? How do you move the coffee cup off the table and put it onto another table? Is it now a child of that other table? What about the coffee in the cup? Is that a child of the cup? How do you change properties of the coffee without necessarily accessing the table and the cup?

Developers working on 3D systems have developed much better paradigms than the DOM for dealing with this problem. An Entity-Component-System architecture with "constraints" is the current best solution. In that architecture, you would create a coffee cup "entity" with a mesh "component" with another "constraint" component, constraining that coffee cup to the table (or better yet, mass component acted on by a physics system). Then you can simply remove the constraint component when removing the cup from one table, and re-add the constraint component when adding it to the other table.

Overall, I think web developers are in for some intense learning and paradigm shifts if 3D becomes the norm.

[nobbis]

As a counterpoint, I've enjoyed building VR on the web using C++ (with WebAssembly/WebGL/WebXR) and not having to touch the DOM or JS, see: https://twitter.com/nobbis/status/1425266634982248451

Benefits include complete control at frame and pixel level, being cross-platform (same code runs on web, iOS, macOS, Linux), and having access to third-party C/C++ libraries for 3D graphics.

[webgfxdev]

React is one of the worst choices of doing something like that.

The underlying abstraction model of having a tree of components and re-rendering only the parts that have changed between renders doesn’t map to the hardware at all, meaning you’ll waste most of the HW performance just on maintaining the abstraction.

You’ll also get zero benefits from the third-party libraries - there’s nothing in them that can help you with stuff that matters, like minimizing amount of the GPU state transitions for example or minimizing amount of GPU/CPU syncs.

It will be scenegraphs all over again, and the graphics industry has ditched these long ago in favor of simpler models, for good reasons.

Long story short, the happy path in graphics programming is very narrow and fragile, and you typically want to structure your abstraction around it.

[rock_hard]

Totally agree!

We switched https://flux.ai from vanilla ThreeJS to R3F and it’s been a huge productivity gain for the whole team!

Less code, that is more capable and more reusable!

In case anyone is interested, we are hiring: https://coda.io/@flux-ai/flux-jobs

[hirako2000]

Except that for three.js, it's react introducing complexity rather than improving the organisation of the code. A simple component with defaults look neat, but start building a complex scene and jsx gets in the way.

three.js isn't dom elements updated in js. The state of each object is updated in the scene depending on more than whether they changed.

Where three.js lacks abstraction is a component system, in plain js, to organise application with decent patterns. Most three apps are a big blob or code.

[short_sells_poo]

I think they need to take a very serious and hard look at performance before it can go anywhere near VR (where rendering speed and stability are paramount). I'm sure it works for simple things and can handle GUIs fine, but the overhead seems huge currently.

[pjmlp]

BabylonJS and PlayCanvas do it just right without jumping into React fashion.

[tills13]

Is there an error in the examples? You have

    const mesh = useRef()

    ...

    <mesh ref={mesh} ...

You'll be rendering an undefined element (before the ref has a chance to attach).

Also, the TypeScript example makes me head hurt.

    const ref = useRef(null!)

Ah, yes. A non-null null literal.

[sswu]

The TypeScript example does feel a little weird. I've used this library successfully with React + TypeScript with the following pattern:

  const mesh = useRef<THREE.Mesh>(null); // mesh has type RefObject<THREE.Mesh | null>
  ...
  useFrame((): void => {
    if (mesh.current != null) {
      mesh.current.rotation.x += 0.01;
    }
  });
  ...
  return <mesh ref={mesh} ... />

As others have mentioned, the ref will typically attach by the time a `useEffect` hook is called, but for safety it's nice to have the null check.

[lxe]

I was scratching my head over this one too. Other answers in the thread don't seem right. I then realized this:

    const mesh = useRef(... 

assigns an instance of something (`React.mutableRefObject`) to a variable in scope, but

    <mesh ...

after compiling into normal JavaScript and becomes

    (0, _jsxRuntime.jsxs)("mesh"...

(see `console.log(__SANDBOX_DATA__.data.transpiledModules['/src/App.js:'].source.compiledCode) `)

So that's why you're not seeing the editor or runtime complain about this. `<mesh` (lowercase), as with other lowercased react elements get translated into `Something('mesh'`, and doesn't require a defined component variable.

Think this:

    const div = 'not a div -- i am a string'
    return <div>{div}</div>

Just one of the many gotchas of many layers of JavaScript/TypeScript/React/etc... programming environments.

[mlsarecmg]

this is how refs work. it's the same with a div. the ref will be filled in useEffect, not before.

null! is a common typescript thing, it is a semantic guarantee that the ref is static and hence will be available. this saves you the if (ref.current) { ... } check.

[onion2k]

The useRef hook doesn't return a ref. It returns a function for setting the ref value. The value is in ref.current. The ref prop calls the setter with a ref to the element on mount.

[afloatboat]

I believe the `null!` is some trickery to let the rest of the code know that the `ref.current` will never be null, but with how React works you can’t really provide any other value. It prevents having to check for `null` everytime you address the value.

I don’t understand your first issue. The component is assigned to the ref, not the other way around, so why would it render `undefined`? It’s possible that `mesh.current` is undefined at first render, but that shouldn’t matter for the component, only for the reference.

[CraftThatBlock]

I'm guessing they are overloading the ref's usage with the custom renderer. I believe that a renderer can accept non-function/class components in theory, which is why this works.

I don't really like this though, although interesting, it feels like a hack and might not integrate well with tooling (e.g. the Typescript example).

How I would have implemented it was exposition a Mesh component which wraps this, and a useMesh hook to wrap and type the ref.

[shallowthought]

I have to wonder: Have we gone too far? Is TypeScript becoming more tedious than helpful? Has React lost its way in its blind zeal for pure functions?

...no, no, definitely not. Must've gone crazy for a second there.

[gentleman11]

So react builds a tree of react elements that react dom turns into html. So this must replace react dom?

Reusable components are easy in a 3d engine like 3js. You can still program declaratively if you liked. It’s claim to outperform raw threejs is surely untrue. React is also bad for animations, and they recommend you sort of use reacts “back door” to do complex animations. 3d engines are all about animations. You could use redux easily without react if you liked. I bet you still have to learn a new API.

This doesn’t seem that useful. Am I mistaken? (Honestly curious)

That said, I bet it was interesting and pleasantly challenging to write

[Jasper_]

The creator of React-Three-Fiber has come out with some extremely bizarre tweets, like claiming updating 2,000 cubes at 60fps is an "impossible amount of load" [0].

The solution it uses for performance is... punting your calculations to the next frame if you didn't make it this frame. They call this "the scheduler". Turns out this makes no sense in any serious context if you want your 3D frame to be coherent.

I'm sure this is helpful for some people, but I can't rely on it when making serious applications. There's lots of things you can punt to the next frame, but ideally that should be decided per system, and not a global framework.

Keep in mind that rendering here is still happening every frame, so the only thing that's happening is 2,000 transform updates for cubes. Inexplicably, without the scheduler that punts updates to the next frame, it takes 700ms to do this [1]. For 2,000 matrix muls. That's 3ms per cube. What on earth is this doing?

[0] https://twitter.com/0xca0a/status/1199997552466288641 [1] https://twitter.com/0xca0a/status/1199997561358213120

[joshwcomeau]

I've built "vanilla" Three.js projects, and worked with react-three-fiber. It is incredibly useful.

Having a new set of "primitives" that map to three.js objects and being able to render them in a React application makes so many things easier. It's also just way less code to write compared to standard three.js.

RE: animation performance, I haven't had any issues with it. I presume either they're doing a bunch of optimizations, or the concerns are greatly exaggerated. You can check out an app I built using it if you want some proof: https://beatmapper.app/

[mlsarecmg]

react has little to do with html. it just calls functions, what you see in react-dom isn't html either, these are nested document.createElement(...) calls. this can be configured for any platform, web, native or otherwise, so <mesh/> in this case is just new THREE.Mesh(). in react terms it's called a custom renderer.

the other thing you say, that react is bad for animations — r3f operates outside of react, there is no overhead, it actually quite easily outperforms threejs. but there are many other benefits, it will usually save you lots of code, and it can be more memory efficient, see this thread: https://twitter.com/0xca0a/status/1426924274527477764

the biggest advantage is the component model, because it allows for a true eco system, something that threejs does not otherwise have due to the lack of a common ground. and interop. every react library can now act on meshes and materials.

[runawaybottle]

The interesting thing about switching renderers is that you are now free of the DOM. I just think web developers only know one api really well (the DOM and it’s offspring frameworks) that when you give us a blank canvas (no pun intended), we resort to the same data structure and api of what we’ve always known.

We’re truly free to make a <Modal /> however we like. Perhaps not even in those tags. We just don’t know it yet.

[franciscop]

We're using these at my job, and it's been a pretty amazing experience. We built a wrapper[1] so that every front-end dev can build components even easier (without needing to know Three.js), since in the end of the day they are just React components that can be easily composed together:

    <View3D>
      <Box position={[1, 0, 0]} color="green" />
      <Box position={[-1, 0, 0]} color="red" />
    </View3D>

[1] https://www.npmjs.com/package/@standard/view

[irevdev]

For what its worth I've been very impressed and happy with what I've been able to achieve with R3F [1]

Its an online IDE for a number of CodeCAD packages to help lower the barrier to this paradigm.[2]

Going from knowing nothing about 3d, just hacking up example code it's been easy to put something respectable together without much dedicated learning. Im super grateful to the pmndrs team

[1] https://cadhub.xyz/dev-ide/cadquery [2] https://news.ycombinator.com/item?id=27649270

[shadowgovt]

This is very cool. Sounds like there's room for performance improvements, but those can happen now that the framework exists.

[nojvek]

How are they making custom Tags with lowercase names. I assumed react would render that as native tags.

[mlsarecmg]

this is a custom renderer. "react" does not know what a "div" is, this comes from "react-dom", that is why they have split these two packages apart.

but they also allow you to make your own renderer (https://github.com/facebook/react/tree/main/packages/react-r...) which then defines its own elements. you can try a mini threejs renderer here: https://codesandbox.io/s/reurope-reconciler-hd16y

[throw_m239339]

VRML anyone?

[mlsarecmg]

you can't express a 3d scene with markup alone, that is why VRML didn't succeed. the JSX you see just masks function calls, it is not XML or HTML, but the true power is in components and hooks (useFrame, etc). a r3f component is self-contained, it will even subscribe to the render-loop. click into these two examples to see the difference: https://twitter.com/0xca0a/status/1426924274527477764

[moron4hire]

There's always one of you. It's not funny.

[beebeepka]

One of those things that make zero sense but use react, and that makes them cool enough to blog about.

[mlsarecmg]

It has similar benefits as react-dom has for the dom. Less code, faster performance, less memory consumption, real interop with a growing eco system.

[arcanon]

The age of WebGPU is almost upon us. Chrome 94 comes out this quarter. I doubt React will be the major player in this next epoch. It will be something new that can pull off an instant load shared world game.

[Jasper_]

In my experience ths issue with this has always been asset download speeds, not web frameworks. The lack of a storage solution means we can't ship game quality assets and have things be instant load.. I run a large WebGL application and the Chrome network cache is still my biggest issue ( e.g. this unsolved bug despite me having an 100% repro https://bugs.chromium.org/p/chromium/issues/detail?id=770694 )

[mlsarecmg]

Threejs already has a webgpu renderer.