Wal-Mart had to apologize when people who searched for Martin Luther King: I Have a Dream were told they might also appreciate a Planet of the Apes DVD collection. Amazon.com similarly offended some customers who searched for “abortion” and were asked “Did you mean adoption?” The adoption question was generated automatically simply because many

As the size of datasets balloons almost beyond the scope of our imagination, it becomes all the more important to continually audit them to check for the possibility of error.

This is a new kind of caveat emptor, where consumers are going to have to search more to make sure that the offered price is fair. Consumers are going to have to engage in a kind of number crunching of their own, creating and comparing datasets of (quality-adjusted) competitive prices.

The openness of the Internet is even transforming the culture of medicine. The results of regressions and randomized trials are out and available not just for doctors but for anyone who has time to Google a few keywords. Doctors are feeling pressured to read not just because their (younger) peers are telling them to, but because increasingly they

The timing is best explained by the digital breakthroughs that make it cheaper to capture, to merge, and to store huge electronic databases.

regression is a statistical procedure that takes raw historical data and estimates how various causal factors influence a single variable of interest.

“Taguchi Method,”

She found that the NIT didn’t reduce employment nearly as much as people feared, but there was a very unexpected spike in divorce. Poor families that were randomly selected to receive the NIT were more likely to split up.

In each of these cases, the companies not only know the generalized probability of some behavior, they can make incredibly accurate predictions about how individual customers are going to behave.