A superintelligent system could have disastrous effects even if it had a neutral goal and lacked self-awareness. “We cannot blithely assume,” Bostrom wrote, “that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans—scientific curiosity, benevolent concern f

Debating the ethics of AI is inherently philosophical. The fundamental questions aren’t about physics or chemistry but about meaning and purpose. Thus, the authors include philosophical as well as technological discussions. They seemed to be on shaky ground.

we have an AI that acts very much like a person, but in ways that aren’t quite human. Something that can seem sentient but isn’t (as far as we can tell). We have invented a kind of alien mind. But how do we ensure the alien is friendly? That is the alignment problem.