Measuring Bottom-Line Usability
Bottom-line data are useful when you want to verify that users are
able to perform a task within a certain amount of time,
or when you want to compare the user-performance between two designs.
- You should not gather bottom-line data during a thinking-aloud test,
because thinking aloud changes the way the user performs.
- You should analyze the data: calculate the mean, the standard
deviation and the standard error of the mean.
This shows how "valid" your conclusions are.
- You can also get data from questionnaires, but these numbers may
have little value.
- For comparing two designs you can do a between-groups
experiment, with two groups, or a within-groups test with
just one group. Both approaches have difficulties:
- In the between-groups approach the variability of both
groups needs to be taken into account to ensure that differences
between (mean) values are meaningful.
- In the within-groups approach you cannot use the same
task for both tests, because the users would take advantage of what
they learned during the first test in the second one. But now the
two tasks may not be comparable.