Quantifying the User Experience: Practical Statistics for User Research

A similar strategy is to multiply the observed percentage frequency of occurrence by the impact score (Lewis, 2012). The range of priorities depends on the values assigned to each impact level.

From an analytical perspective, a useful way to organize UI problems is to associate them with the users who encountered them, as shown in Table 2.1.

Most tests contain some combination of completion rates, errors, task times, task-level satisfaction, test-level satisfaction, help access, and lists of usability problems (typically including frequency and severity).

In practice, this means if you intend to draw conclusions about different types of users (e.g., new versus experienced, older versus younger) you should plan on having all groups represented in your sample.

For practical tips on collecting metrics in usability tests, see A Practical Guide to Measuring Usability (Sauro, 2010) and Measuring the User Experience (Tullis and Albert, 2008).

Errors provide excellent diagnostic information on why users are failing tasks and, where possible, are mapped to UI problems. Errors can also be analyzed as binary measures: the user either encountered an error (1 = yes) or did not (0 = no).

If there is more variation in a population, each sample taken will fluctuate more and therefore create a wider confidence interval. The variability of the population is estimated using the standard deviation from the sample.

The confidence interval width and sample size have an inverse square root relationship. This means if you want to cut your margin of error in half, you need to quadruple your sample size.

In general, you should know that a confidence interval will tell you the most likely range of the unknown population mean or proportion.

Errors are any unintended action, slip, mistake, or omission a user makes while attempting a task. Error counts can go from 0 (no errors) to technically infinity