[eggplantiny]

I broadly agree with the caution against naive "conscious AI" narratives, but I think this essay underestimates a key idea from Hofstadter’s GEB.

Hofstadter's claim was not that consciousness is "just computation" in a trivial sense, but that within the space of computable processes, certain self-referential, symbol-manipulating systems can give rise to something structurally indistinguishable from consciousness — a "strange loop."

In that framing, consciousness is not a magical biological add-on, nor merely subjective illusion, but an emergent property of sufficiently rich, recursive models that represent themselves and the world simultaneously.

Biology may be one implementation of such loops, but it’s not obvious that it's the only possible one. Dismissing computation wholesale risks conflating "today's AI architectures" with "the entire space of computable systems."

The real open question, I think, isn't whether computation can support consciousness-like phenomena, but what class of computations is required — and whether our current systems even approximate it.