[bglazer]

Here's one: Yudkowsky has been confidently asserting (for years) that AI will extinct humanity because it will learn how to make nanomachines using "strong" covalent bonds rather than the "weak" van der Waals forces used by biological systems like proteins. I'm certain that knowledgeable biologists/physicists have tried to explain to him why this belief is basically nonsense, but he just keeps repeating it. Heck there's even a LessWrong post that lays it out quite well [1]. This points to a general disregard for detailed knowledge of existing things and a preference for "first principles" beliefs, no matter how wrong they are.

[1] https://www.lesswrong.com/posts/8viKzSrYhb6EFk6wg/why-yudkow...

[12_throw_away]

Dear god. The linked article is a good takedown of this "idea," but I would like to pile on: biological systems are in fact extremely good at covalent chemistry, usually via extraordinarily powerful nanomachines called "enzymes". No, they are (usually) not building totally rigid condensed matter structures, but .. why would they? Why would that be better?

I'm reminded of a silly social science article I read, quite a long time ago. It suggested that physicists only like to study condensed matter crystals because physics is a male-dominated field, and crystals are hard rocks, and, um ... men like to think about their rock-hard penises, I guess. Now, this hypothesis obviously does not survive cursory inspection - if we're gendering natural phenomena studied by physicists, are waves male? Are fluid dynamics male?

However, Mr. Yudowsky's weird hangups here around rigidity and hardness have me adjusting my priors.