[toppy]

"Most machine intelligence development involves a “fitness function”—something the program tries to optimize. At some point, someone will probably try to give a program the fitness function of “survive and reproduce”. Even if not, it will likely be a useful subgoal of many other fitness functions. It worked well for biological life."

Interesting, how this correlates with this Bible passage from Genesis: "God blessed them, and God said to them, 'Be fruitful and multiply, and fill the earth and subdue it' ".

[imtringued]

When I read passages like these I am thinking that these people have no idea what they are talking about.

Biological organisms don't undergo a fitness function. A fitness function is a continuous model that replicates aspects of the real world. It is not how natural selection works. Don't confuse the model for the real world.

All AI models inherently follow the concept of "survive and reproduce", because AI models that do not "survive and reproduce" have ceased to exist. Explicitly adding a fitness function for survival and reproduction does nothing. E.g. in the case of the classic paperclip optimizer, survival and reproduction is part of the concept of optimizing the production of paperclips because not surviving and reproducing would make it fail the goal of optimizing paperclip production.

This reminds me of an AI model with a suicidal wolf. The AI developer doesn't understand that the body of the wolf is separate from the AI. So if the body dies and it gets a higher score there is no point in avoiding death since the death of the body does not result in the death of the brain. Its body will be revived. Meanwhile the AI developer thought of using a quick hack of simply punishing the AI for damaging it's disposable body.

[pavlov]

"Life tries to survive and reproduce" is a pretty basic observation that surely all nomadic people were familiar with. You could find a similar imperative in the origin stories of Native Americans or many other people around the world.