Playing With Reality

The problem is not that a paper-clip-maximizing Al will arise in the future and turn the universe into paper clips. The maximizers are already here. Any consequences too subtle to measure—environmental costs, civic discord, troubled diplomatic relations—are simply omitted from the score

We increasingly live in our maps of reality instead of in reality itself. The territory has been replaced with towering layers of abstractions: sovereign nations, states, counties, personal properties, deeds, mortgages, mortgage-backed securities, mortgage-backed credit default swaps, and so on.

For truly effective social reform, we can use the tools of game theory to understand why we're in the equilibrium we're in and to map out realistic alternatives. We should be willing to iterate on entirely new game structures, not just tinker with the rules of games we're already in.

No model or mechanism, in the end, can correct the imperfections of reality. Games temporarily suspend the rules of daily life, allowing players to enter alternate moral universes where they're rewarded for swindling their sibling, lying to their partner, or shooting their friend.

Design is what happens when we uncover rules latent in the world and use these to define the logic of a new, separate system. The inventors of chess abstracted the different capabilities of army divisions to invent the rules of a game that's entertained us for centuries. We can't change reality, but we can tinker with designed systems to encourage

Games, a universal in human societies, might have emerged as a framework for building mental models of one another. They may have been preserved and elaborated across generations thanks to their benefits in teaching people how to reason about one another. Players must understand the beliefs and preferences of other players to secure the best outcom