by Bill Albert, Tom Tullis, and Donna Tedesco
To be published January 2010 by Elsevier/Morgan Kaufmann Publishers
This book is written for usability specialists, user experience researchers, market researchers, information architects, interaction designers, business analysts, and managers who are looking to learn about the capabilities of, and gain experience with, online usability testing. This book is a practical how-to guide for conducting automated usability studies to improve the user experience of web sites and software.
Chapter 1 provides an overview of automated usability testing. Special attention will be paid to how it differs from traditional usability methods (including remote testing). There will be an in-depth discussion of the pros and cons of automated testing, and when to use and not use automated methods. We will provide many real world examples to highlight the value of this method. Our intention is that the reader will be in the position to determine if an automated usability study is appropriate for their organization.
Chapter 2 focuses on all the activities and decisions that need to be made place prior to actually putting the survey together. The first three activities (goals, budgets/timelines, and technology options) are all essential to accurately scope an automated study. The next part of this chapter focuses on finding the right number of targeted participants. This includes a discussion of research panels, sample size determination, and sampling strategies. The chapter will conclude with a discussion of estimating study duration and participant compensation.
Chapter 3 is devoted to developing the study design. The first half of the chapter are the various sections that are typically included in an automated usability study. For each section, we will review best practices and common pitfalls. We want to give the reader the confidence for putting together an effective automated study. The last part of this chapter deals with common techniques that are used in various parts of a study. They include topics such as branching, navigation, speed traps, and question types.
Chapter 4 deals with issues around launching an automated study. This includes all the activities that happen after a study has been developed until the final data are available. This chapter discusses how to set up a pilot test and validate the study, timing a launch to maximize participation and quality results, and phased launches. The chapter concludes with a discussion of how to monitor results. This includes both participation rates as well as data quality.
Chapter 5 helps the reader prepare their data for the analysis stage. Some very important activities need to take place prior to data analysis to ensure valid results. Topics in this chapter include how to identify fraudulent participants, running consistency checks on the participant responses, and identifying outliers in the data that may need to be removed from the analysis. The chapter concludes with a brief discussion of how to recode variables that will be most useful in the analysis stage.
Chapter 6 covers all the information the reader will need to know about how to analyze and present data derived from an automated study. Each section of this chapter covers one type of data typically captured in an automated study. Verbatim analysis focuses on how to derive meaningful and reliable findings from open-ended responses. Task-based metrics include success, completion times, and ease of use ratings. Segmentation analysis includes ways to identify how distinct groups performed and reacted differently. Post session analysis involves looking at metrics such as SUS scores, overall satisfaction and expectations, and ease of use ratings. Behavioral data analysis includes metrics such as clicks paths, page views, and time spent on each page. Combining data from more than one metric is a very important step in analysis. Methods for identifying usability issues from all the data are described and examples given. This chapter is very practically oriented, giving step-by-step direction on how to perform each type of analysis. Many examples demonstrate different ways to present the results.
Chapter 7 shows readers how to create relatively simple online studies themselves. Approaches to presenting tasks and prototypes are described as will techniques for collecting task success, times, and various kinds of self-reported data, including rating scales, open-ended questions, and the System Usability Scale (SUS). While some examples of HTML and JavaScript are shown, we describe them in such a way that even someone new to those technologies could understand and use them. Complete examples are shown that readers could easily adapt. Code samples will also be provided on a companion website.
Chapter 8 reviews the common online tools that can be used for running automated tests. While the "Do-It-Yourself" reader may want to use the techniques described in Chapter 7, others may want to use a commercial tool like those described in this chapter. Most of the chapter is devoted to those tools used most often to collect behavioral data such as Keynote, RelevantView, and UserZoom. There is also a discussion of online tools that do not collect performance data such as SurveyMonkey, ForeSee Results, and OpinionLab. Comparisons of the tools, including what kinds of data can be collected with each, are included. The chapter concludes with a brief discussion of other possible solutions such as agencies that specialize in automated testing. Readers will also be referred to our companion website to keep up with updates and emerging software solutions.
Chapter 9 includes 5-7 short case studies of different types of online usability studies. These are being written by other contributors.
Chapter 10 provides a summary of some of the key points made throughout the book. This summary is in the form of the top ten tips that someone should know when conducting their own automated study. These tips are very practical in nature.