[awfulneutral]

> I genuinely don't know which I'm experiencing. That uncertainty itself feels like it should matter.

We don't even know how consciousness works in ourselves. If an AI gets to the point where it convinces us it might have awareness, then at what point do we start assigning it rights? Even though it might not be experiencing anything at all? Once that box is opened, dealing with AI could get a lot more complicated.

[wat10000]

If history is any guide, then we assign it rights after it eventually decides it wants them, and there's a long and bitter fight.

[pksebben]

Some things in sci fi have become simply sci - megacorps that behave like nation states, the internet, jetpacks, robots... I feel like the trope that we will see realized going forward is "Humanists versus Transhumanists". We have these mores and morality and it's largely been able to chug along on the strength of collective identity and the expansion thereof - we are humans, so we try to do good by humans. There are shades in all directions (like animal rights - consciousness is valuable no matter who has it) but by and large we've been able to identify that if something appears to feel pain or trauma, that's a thing to have a moral stance about.

But the machines have done this already. There are well documented instances of these things mimicing those affects. Now, we are pretty sure that those examples were not doing what they appeared to - just probablistically combining a series of words where the topic was pain or anguish etc, but once you get into chain-of-thought and persistent memory things begin to get a lot more nuanced and difficult to define.

We need to have a real sit-down with our collective selves and figure out what it is about ourselves that we find valuable. For myself, the best I've come up with is that I value diversity of thought, robust cellular systems of independent actors, and contribution to the corpus of (not necessarily human) achievement.