Quantifying the User Experience: Practical Statistics for User Research

rely on a subset of the population to use as a proxy for the population.

We can say there is a difference when one doesn’t really exist (called a Type I error), or we can conclude no difference exists when one in fact does exist (called a Type II error).

Using the Excel function =TDIST(2.3,11,2) we get 0.04, which is called the p-value. A p-value is just a percentile rank or point in the t-distribution. It’s the same concept as the percent of area under the normal curve used with z-scores. A p-value of 0.04 means that only 4% of differences would be greater than 15 seconds if there really was no di

The t-distribution is like the normal distribution (also called the z-distribution) except that it takes the sample size into account.

The well-known (and often misused and maligned) guideline that five participants are enough to discover 85% of problems in a user interface is true only when p equals 0.315.

It’s a two-sided test because you care about both sides of the confidence interval. Anytime you care about outcomes that can be either significantly higher than a criterion or might just as well be significantly lower (e.g., when testing one product against another without any preconception of which is better), you’d use a two-sided test.

Using the Empirical Rule and z-scores to find the percent of area only works when we know the population mean and standard deviation. We rarely do in applied research.

A narrated visualization of the standard deviation is available online at http://www.usablestats.com/tutorials/StandardDeviation.

Usability has an international standard definition in ISO 9241 pt. 11 (ISO, 1998), which defined usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use.