Workshop: Online Usability Testing: Moving Beyond the Lab

Instructors:

Tom Tullis and Donna Tedesco

Where:

Hyatt Regency San Francisco
San Francisco, California, USA

When:

Monday, September 27, 2010
9:00 AM to 4:30 PM

Cost:

$375 HFES Members
$425 Non-members
$175 Students

Abstract:

Online (unmoderated) usability testing is a cost-effective, efficient, and reliable method for collecting qualitative and quantitative data from hundreds or thousands of users simultaneously. It is an increasingly popular method for measuring the holistic user experience with a product—especially web sites and applications. This workshop will give attendees the skills they need to conduct an online study from start to finish - including planning, design, implementation, analysis, and presentation of results. This includes topics such as determining when to use and not use online testing, tips for scheduling and recruiting for studies, task and question construction, choosing a vendor tool or utilizing discount approaches to implementation, piloting and launching the study, data preparation and analysis, and presentation tips. While the most common application of online studies is to the evaluation of websites, the presenters will also discuss and explore the applications of the method to other types of systems and products. The intended audience for this workshop is professionals within the human factors and usability field who have a basic understanding of typical in-lab usability testing techniques.

Detailed Summary:

As technology and product design permeates society, measuring usability and the holistic user experience plays a more critical role in the success of products. Users have little patience for poor designs; the competitive marketplace and ability for product reviews to be shared quickly and widely can make or break the success of a product. Online usability testing offers the ability to collect the behavior and perceptions of hundreds or thousands of users simultaneously. It brings together both quantitative and qualitative data to make informed decisions about designs. It can be used for many goals and purposes, from something as specific as comparing the way a button is worded, to benchmarking a product-wide user experience. It can be used as a stand-alone usability method or in conjunction with other methods such as in-lab testing, heuristic reviews, and focus groups.

Many user experience professionals recognize the potential value of this method but lack the information necessary to design a study and choose the appropriate method to implement and analyze a study with confidence. The small amount of information available today is directly from vendor services. The few vendor services in this domain have varying selling points and cost structures which makes it overwhelming to choose a service, and difficult to obtain unbiased information about the method. Having conducted hundreds of online studies, the presenters have the knowledge and experience to offer unbiased information about the many services available, as well as tips and tricks for implementing the study that is not offered elsewhere.

The workshop will be primarily lecture-based with hands-on exercises throughout the session. The exercises will focus on a couple of case studies, allowing attendees to apply their learnings about the various phases of online testing throughout the day.

Although no prior experience or knowledge of online usability testing is necessary, attendees should have a basic understanding of the principles of traditional lab usability testing. Online studies build off a foundation of these principles. For example, when discussing task construction, the presenters plan to talk about the tips and issues unique to creating tasks for online studies, rather than delving into an extensive background on task construction principles for any usability study.

Agenda:

9:00 - 9:30: Introduction to online testing and overview of case studies for class exercise

• learning mode: lecture and demo
• skill taught: fundamental background
  

9:30 - 10:30: Planning the study

• learning mode: lecture and demo; class exercise
  
• skills taught: Identifying target users, Type of study, Budget and timeline, Metrics, Recruiting, Sampling, Incentives
  

10:30 - 12:00: Designing the study

• learning mode: lecture and demo; class exercise
  
• skills taught: Introduction to study, Screening and Starter Questions, Constructing Tasks, Post-task and post-session questions, special topics for study design
  

12pm lunch break

1:00 - 2:00: Vendor services and Discount implementation methods

• learning mode: lecture and demo 1:00-1:30; class exercise 1:30 - 2:00 pm
  
• skills taught: overview of vendor tools including Loop11, RelevantView, UserZoom, WebEffective and other related tools; Building online studies using a discount (or even free) approach
  

2:00 - 2:30: Piloting and Launching the Study

• learning mode: lecture and demo
  
• skills taught: Technical checks, usability checks, full pilot checks, Timing the launch, Monitoring results
  

2:30 - 4:00: Data Preparation and Analysis

• learning mode: lecture and demo from 2:30 - 3:30, class exercise from 3:30 - 4:00
  
• skills taught: cleaning up data; recoding data; analysis for task success, time, efficiency; calculating confidence intervals; analysis of clickstream data; segmentation analysis; and identification of usability issues
  

4:00 - 4:30: Presentation tips and other keys to success

• learning mode: lecture and demo
  
• skills taught: Setting the stage, organizing data, displaying and explaining information, other tips

Online Registration Form (select Workshop "WK11- Online Usability Testing")

General information about the HFES Conference

Note: It is not necessary for workshop attendees to register for the entire conference. You can register for just the workshop if desired.