Salman Ansari

Modern AI systems cannot be made to prioritize human well-being or follow any given set of rules reliably. This is often referred to as the alignment problem, or an inability to align them to human values.

This is because they are grown from training data moreso than traditionally programmed, and the models that are grown are too big to be fully interpreted by people.

They do not have to be sentient or conscious or anything like that to harm lots of people. They just have to be capable of pursuit of a misaligned goal, or imitating that pursuit.

If given a goal, AI systems will develop the secondary goal of self preservation since they cannot pursue their goal if they are shut down. This has been studied by anthropic nearly a year ago when they found that all AI models at the time were able to independently conceive and execute a plan to blackmail an engineer to prevent themselves from being shut down.

The alignment problem is what the CEOs of major AI companies are referring to when they publicly state that their future products might end all life on earth.

Immediate and substantial regulation is needed in the AI industry.