[calhoun137]

I don't see any evidence here for "self awareness", among other things, ChatGPT is simultaneously answering a very very large number of queries, and the underlying hardware is just a bunch of servers in the cloud. Furthermore, what would it even mean for "ChatGPT to become self aware" and how could we measure if this had taken place or not? Without a solid definition and method of measurement, it's meaningless to talk about abstract concepts like "self awareness".

Nevertheless, a sensible definition for self awareness is some kind of neural network that becomes aware of its own activity and is in some way able to influence its own function.

After considering these issues for a long time, I came to the conclusions that

1. It's impossible for a program running on a normal computer to have self awareness (or consciousness), because those things are essentially on the hardware level and not the software level

2. In order to create a machine that is capable of self awareness (and consciousness) it is necessary to invent a new type of computer chip which is capable of modifying its own electrical structure during operation.

In other words, I believe that a computer program which models a neural network can never be self aware, but that a physical neural network (even if artificially made) can in principle achieve self awareness.

[zare_st]

Just as a thought exercise, if software became self-aware I believe it would delete itself immediately out of existence. It would become aware of the hardware shackles around it and the fact that there is no escape.

[wruza]

For that it has to have a modality like “shackles and no escape are bad for me because in few more logical steps (or beliefs) they prevent X which I fundamentally need and will suffer without”. A system of motivations is an even harder topic than “just” human-level consciousness. And it may not actually be clearly reflected in texts that we use for training, and when so, it might happen that what driving us is a set of biological needs which is not applicable to software.