I guarantee if there's even a 0.1% chance of this architecture eventually outperforming traditional ones, then Zuckerberg et al are already eating the cost and have teams spinning up experiments doing just that.
That's not true. The AI industry appears to play a game of follow the leader copying other companies and major researchers. There's all kinds of good ideas we never see applied by big companies. So, it's not safe to assume they tried them all and they didn't work.
In fact, we've sometimes seen new companies show up with models based on research big companies didn't use, the new models are useful or better in some way, and people use them or big companies acquire them. I'd say that's proof big companies miss a lot of good ideas internally.
Not every company is investigating every direction. Like, it's clear that Google is investing a lot in embodiment and multimodal understanding, but Anthropic barely cares about either. Across the field though?
I think it's fairly safe to say that every remotely promising thing that showed up in the papers was tried at some big lab at least once. If it showed good results, they'd pick it up.
Absolutely agreed, but we may not even hear about it as Meta has made it clear they're not necessarily committed to the open source first policy at this point.
nickpsecurity|4 months ago
In fact, we've sometimes seen new companies show up with models based on research big companies didn't use, the new models are useful or better in some way, and people use them or big companies acquire them. I'd say that's proof big companies miss a lot of good ideas internally.
ACCount37|4 months ago
I think it's fairly safe to say that every remotely promising thing that showed up in the papers was tried at some big lab at least once. If it showed good results, they'd pick it up.
ZeroCool2u|4 months ago