building AIs to help us contain AI. Researchers are working on a generation of “critic AIs” that can monitor and give feedback on other AI outputs with the goal of improving them at speeds and scales that humans cannot match—speeds and scales that we see in the coming wave. Managing powerful tools itself requires powerful tools.

Over the last decade, intellectual and political elites in tech circles became absorbed by the idea that a recursively self-improving AI would lead to an “intelligence explosion” known as the Singularity.