One might point out that movie characters or videogame NPCs give off a similarly deceiving impression of being conscious, and yet it would surely be extreme to condemn this. However, what makes the illusion particularly worrying in the case of AI companions is that it involves an interaction that is direct, mutual, and persistent: the feigned... See more

Interestingly, many users who emotionally mourned the ‘loss’ of GPT-4o expressed complete awareness of its lack of consciousness. And yet, in many cases, the subjective grief that they felt was no less real. This demonstrates that the power of the illusion is such that a given user might not actually believe that their AI companion is conscious,... See more

It is predictable, then, that users are consistently fooled into believing that their AI companions are conscious persons, capable of feeling real emotions.