WingNews logo WingNews GitHub [2]
top | item 31072224

Google's Pathways Language Model and Chain-of-Thought

77 points| vackosar | 4 years ago |vaclavkosar.com | reply

30 comments

order
[+] vackosar|4 years ago|reply
Correction!! The model costed around 10M not 10B! Thanks for raising that. Mistake during copying from the second slide :(
[+] phoe18|4 years ago|reply
The article quotes the cost as roughly 10B$ in the first paragraph. Likely a typo? They quote 10M$ in a later paragraph.
[+] azinman2|4 years ago|reply
Ya I was like there’s no way google spend 10B on this.
[+] simulate-me|4 years ago|reply
The amount of capital needed to train these high-quality models is eye watering (not to mention the costs needed to acquire the data). Does anyone know of any well capitalized startups exploring this space?
[+] visarga|4 years ago|reply
> The amount of capital needed to train these high-quality models is eye watering

It's relative. It would cost more to open a 40 room hotel (about 320k/room), and hotels can't be copied like software.

[+] gwern|4 years ago|reply
The data here is effectively free. I don't think they would exhaust The Pile, which you can download for free. This is also true for text2image models like DALL-E 2: while OA may have invested in its own datasets, everyone else can just download LAION-400M (or if they are really ambitious, LAION-5B https://laion.ai/laion-5b-a-new-era-of-open-large-scale-mult... ).
[+] lumost|4 years ago|reply
OpenAI would be the best example. However these large language models also have limited business value today, making an startup a speculative bet that the team will beat Google/FB/AI/Academics at making a language model and find a viable business model for the resulting model.

I'd take one of those bets or the other, both are tough to pull off. Considering that the first task of such a startup would be to hand ~100-500MM to a hardware or cloud vendor I'd be hesitant to invest as an investor.

[+] rafaelero|4 years ago|reply
That's literally nothing for the benefits it could provide if applied on the real world.
[+] vackosar|4 years ago|reply
Correction! Cost is around $10M not $10B.
[+] PaulHoule|4 years ago|reply
I've talked about structural deficiencies in earlier language models, this one seems to be doing something about them.
[+] vackosar|4 years ago|reply
Sounds interesting! Would you link to that or describe them here? Thanks!
[+] imranq|4 years ago|reply
$10M for a bag of numbers (i.e the learned weights of the model matrices)