Quantifying the User Experience: Practical Statistics for User Research

Most tests contain some combination of completion rates, errors, task times, task-level satisfaction, test-level satisfaction, help access, and lists of usability problems (typically including frequency and severity).

Errors provide excellent diagnostic information on why users are failing tasks and, where possible, are mapped to UI problems. Errors can also be analyzed as binary measures: the user either encountered an error (1 = yes) or did not (0 = no).

Errors are any unintended action, slip, mistake, or omission a user makes while attempting a task. Error counts can go from 0 (no errors) to technically infinity

we’ll be right most of the time, but at a 95% level of confidence, in the long run we will incorrectly conclude 5 out of 100 times (1 out of 20) that a difference is statistically significant when there is really no difference.

For practical tips on collecting metrics in usability tests, see A Practical Guide to Measuring Usability (Sauro, 2010) and Measuring the User Experience (Tullis and Albert, 2008).

We are 95% confident in the method of generating confidence intervals and not in any given interval.

The more formative (diagnostic, focused on problem discovery) the focus of a usability test, the less it is like a traditional experiment. The more summative (focused on measurement) a usability test is, the more it should resemble the mechanics of a traditional experiment.

Table 7.1 shows the sample size requirements as a function of the selected values of p (problem occurrence probability) and P(x ≥ 1) (likelihood of detecting the problem at least once).

Chapters 6 and 7 contain a thorough discussion of power and computing sample sizes to control Type II errors.