[olooney]

I think this is interesting and partially true: humans are scary. But it's important to remember the opposite is true as well: humans are the most cooperative species out there by a wide margin.

Eusocial insects and pack animals are in a distant 2nd and 3rd place: they generally don't cooperate much past their immediate kin group. Only humans create vast networks of trade and information sharing. Only humans establish establish complex systems to pool risk, or undertake public works for the common good.

In fact, a big part of the reason we are so scary is that ability to coordinate action. Ask any mammoth. Ask the independent city states conquered by Alexander the Great. Ask Napoleon as he faced the coalition force at Waterloo.

We are victims of our own success: the problems of the modern world are those of coordination mechanisms so effective and powerful that they become very attractive targets for bad actors and so are under siege, at constant risk of being captured and subverted. In a word, the problem of robust governance.

Despite the challenges, it is a solvable problem: every day, through due diligence, attestations, contract law, earnest money, and other such mechanisms people who do not trust each other in the least and have every incentive to screw over the other party are able to successfully negotiate win-win deals for life altering sums of money, whether that's buying a house or selling a business. Every century sees humans design larger, more effective, more robust mechanisms of cooperation.

It's slow: it's like debugging when someone is red teaming you, trying to find every weak point to exploit. But the long term trend is the emergence of increasingly robust systems. And it suggests a strategy for AI and AGI: find a way to cooperate with it. Take everything we've learned about coordinating with other people and apply the same techniques. That's what humans are good at.

This, I think, is a more useful framing than thinking of humans as "scary."

[Spivak]

I'm not sure if it's your intention but this reads as a strong critique of the technolibertarianism world philosophy that dominates our industry. We lose something by replacing high-trust cooperative systems with ones that are mutually antagonistic. We fall into the bad square of the prisoners dilemma by not only not exorcising the defectors but holding them up as the highest moral good and the example to follow.

[olooney]

This is going to sound a little weird, but I think trust is part of the problem.

Canadians have a high-trust culture, but their stock market is historically full of scams[1], and some analysts think its causally related. (It may just be because TSXV is a wild west, or because companies would IPO on NYSE or Nasdaq if they were legit, but it could be the trust thing. Fits my narrative, anyway.)

When I look at politics, crypto rug pulls, meme stocks with P/E ratios over 200, Aum[2] and similar cults, or many other modern problems I don't see negotiations breaking down because of a lack of trust; I see a bunch of people placing far too much trust in sketchy leaders and ideas backed by scant evidence. A little skepticism would go a long way.

That's why I emphasize robust coordination: more due diligence, more transparency, more fraud detection, more skepticism, more financial literacy, more education in general. There's a cost associated with all this, sure, but it still gets you into a situation where the interaction is a coordination game[3] and the Nash equilibrium is Pareto-efficient. Thus, we fall into the "pit of success" and naturally cooperate in our own best interests.

There's nothing wrong with empathy, altruism, or charity, but they are very far from universal. You need to base your society on a firm foundation of robust coordination, and then you can have those things afterwards, as a little treat.

[1]: https://en.wikipedia.org/wiki/Vancouver_Stock_Exchange

[2]: https://en.wikipedia.org/wiki/Tokyo_subway_sarin_attack

[3]: https://en.wikipedia.org/wiki/Coordination_game