Oliver Burkeman sums up my views on AI and creativity pretty well:

To start with, picture this scenario. You’re at home, feeling lonely and sorry for yourself, when the phone rings out of the blue. It’s an old friend, checking in on you. For an hour, you have one of those rare, wonderful, uplifting conversations; by the time you hang up, you’re glowing inside. Ten minutes later, checking your email, you find a note from Meta revealing that it wasn’t your friend at all, but a brilliant voice-cloned simulation, automatically generated from videos your friend had uploaded to Facebook.

A question: does this change anything?

I didn’t say it was a hard question. Obviously, it changes everything! What made the conversation meaningful was that there was another thinking and emoting conscious awareness at the other end of the line — a person willing to use a portion of their finite time and attention to hold you in mind, and to connect, across the miles, with your thinking and emoting conscious awareness. Or so you believed. Now it turns out that ih fact there was no-one there.

My contention is that creative work is like this, too. Some people argue that an AI will never write a novel as well as Jane Austen because there’s something intrinsically special about human creativity; this infuriates AI enthusiasts, who see it as nothing more than an appeal to magic creativity fairy dust. I agree with the AI boosters that there’s no reason to assume AI won’t match the output of a human in any given field. But this misses the point. The point about a good novel produced by a human (or a song, or painting, or dare I say it, an email newsletter) isn’t that only a human could ever have produced it. It’s that a human did in fact produce it. There really was another thinking, emoting consciousness at the “other end of the line.” And when you consume the work, you enter into a kind of relationship with them.

Nowhere is this clearer that in psychotherapy. “AI therapy” is shaping up to be big business, with advocates arguing it’s already getting as good as the human equivalent. But if, like me, you believe that the core value of therapy lies in the connection between two conscious beings, one striving to hold the other in “unconditional positive regard” while focused on the his or her inner growth, all this talk of AI therapy is liable to make you feel like you’re losing your grip on sanity. If there’s no second consciousness present, there’s simply no therapy occurring, good or bad! Maybe simulated therapy has a role to play, for example as a supplement to therapy. Maybe, depressingly, it’s better than nothing in a culture where therapy is so hard to access. But at least by the definition above, it’s still not actually therapy – and anyone who thinks it is is deceiving themselves, or being deceived.

AI, Heidegger, and Evangelion