Possible Minds: Twenty-Five Ways of Looking at AI

Though one can imagine hypothetical combinations of the most malevolent totalitarians with the most advanced technology, in the real world it’s the norms and laws we should be vigilant about, not the tech.

We are not stuff that abides, but patterns that perpetuate themselves.*

society with channels of feedback that maximize human flourishing will have mechanisms in place,

This problem requires a change in the definition of AI itself—from a field concerned with pure intelligence, independent of the objective, to a field concerned with systems that are provably beneficial for humans.

“[T]he fate of information in the typically American world is to become something which can be bought or sold”; most people, he observed, “cannot conceive of a piece of information without an owner.”

This is the value-alignment problem.

next step is to try to do the same thing but at scale, something I refer to as building a trust network for data. It can be thought of as a distributed system like the Internet, but with the ability to quantitatively measure and communicate the qualities of human society, in the same way that the U.S. census does a pretty good job of telling us abo

We can address the problem of building an accurate credit-assignment function in many different settings.

natural stupidity can wreak far more havoc than artificial intelligence;

The genetic system in every living cell is a stored-program computer. Brains aren’t.