[PBnFlash]

There are probably a lot of optimizations in the silicone and software to find. It's not necessarily obvious what corners can get cut or where, the tradeoffs of tapping out new chips is worth it. Yet Another Matrix Multiplication Chip is not going to set the world on fire. Nvidia has that market pretty well captured.

But perhaps it turns out that subnets can be trained independently or swapped with semantically equivalent but qualitatively different ones. The routing network would effectively "Standardize" and could in principle be well enough understood to "hand optimize" the routing network into hardware. Or maybe back propagation has some novel physical analogue that can be exploited in scales we can access. The real question is if Altman is capable of finding the right path in the notoriously dead end filled field of chip design. His backing of helion [1] didn't bode well in my view. But with enough R&D maybe he will flail into something useful trillions is enough for a lot of flailing.

[1] https://youtu.be/3vUPhsFoniw Edit: more derisive link