Sep 19 - Nielsen, Weekly User Testing: TiVo Did It, You Can, Too (Alertbox)

Weekly User Testing: TiVo Did It, You Can, Too

Summary: TiVo ran 12 user tests in 12 weeks while designing its new website. As TiVo's experience shows, frequent and regular testing keeps the design usability focused.


I've always recommended fast and cheap user testing, with as many iterations as you can squeeze into your schedule. My true preference is weekly testing; make, say, every Wednesday your User Day and schedule 4 customers to test.
Unfortunately, I know of few projects that have sustained this pace, and many of those were my own. Recently, I came across another project that employed weekly testing: the redesign of tivo.com.

12 Weeks, 12 Tests

The project team had a short timeline for the comprehensive site redesign. To make the best use of that time, team members devised a research plan to conduct 12 consecutive weeks of user testing leading up to the site's relaunch. "We had a certain timeline for the project," says Deborah Torres, a TiVo user researcher. "We were working backwards from a launch date."
For 8 of the 12 sessions, Torres and her team employed remote testing methods, engaging users with collaborative online tools such as Go-To Meeting or WebEx. "We tested users across the country," she says. "Midwest [and] East Coast tested designs and content remotely — we did remote usability testing where we shared the mouse."
For the last sessions, the team brought users into the company's usability lab and tested 1-on-1. "We got stakeholders in the room," says Torres, adding that it was good for the team to "see frustrations in people's faces" rather than simply listen over the phone.

5 Reasons to Take This Approach

a. Costs the company less.
TiVo's testing approach included many remote user sessions, so the company investment was mostly in staff time, with a limited amount devoted to incentives. And the ROI was great. As Schmidt put it, "one to two thousand dollars is not that much money for what you learn."

b. Offers motivation.
Rapid testing cycles give the design team reassurance that they're on the right path. "Test after test, you see that you're going in the right direction," says Schmidt. "It can be very motivating to the team."

c. Helps drive business decisions.
Because the feedback is weekly and available in real time, the team can react in real time. Schmidt used the feedback cycles to push back on business owners and ask, "Do we really have to keep pushing forward, or fix the issues? The users are telling you how they will react in the field."

d. Creates a testing culture.
Admittedly, TiVo's culture is already user-focused on the product side; until this exercise, however, the company wasn't particularly user-focused on the Web side.
After seeing the testing cycle results, the company committed to user experience across the board. "Now, user experience is key," says Schmidt. "It has changed how we structure all Web projects moving forward. Each new [Web] project will have the same level of UE testing as products."

e. Builds internal knowledge.
"Build internal knowledge," says Schmidt. "Use internal resources, internal research groups that can move quickly." "Agency timelines can be long and it takes a while for new people to get up to speed on issues," she says. "We present a new design to a researcher who can learn what's [right and wrong] about it in the context of other projects. Designers build a design gut," she says. "Every time you listen — you become a better designer."

8 Keys to Successful Outcomes

1. Just do it.
"Do it and listen to it," says Schmidt. "You have to take the ego out of the design. The design is great when the users say it is."

2. Get support from stakeholders and find a project champion.
"It is only valuable if you have stakeholders who are willing to listen and act on the feedback," says Torres.

3. Encourage participation — from across the organization.
"Invite everyone who has an interest in it," says Torres. "Every usability test session, the observation room was packed with designers, product managers, and content people. Even remote sessions, there were managers watching and logging in." That kind of participation ensures widespread buy-in across the organization.

4. Determine priorities so you get clear results.
Ask pointed questions and set clear research objectives for each screen/task so you can turn them into actionable results immediately after the test sessions.

5. Circulate the results — quickly.
The morning after each test session, Torres released a top-line summary of teaser bullets telling people the test results from the night before. People across the organization appreciated these summaries.

6. Match the research method to what you're trying to uncover.
Different test methods suit different goals. The TiVo team used a variety of methods, from focus groups and concept testing to remote user testing and onsite 1-on-1 task-based testing. By matching the method to the testing goal, the team obtained a range of valuable feedback.

7. Ask for buy-in on the results.
According to Schmidt, the most effective way to get buy-in — especially in a rapid testing situation — is to present the findings along with a solution. "When you share the raw research without proposing a solution," she says, "they worry about it. You don't need 20 people debating how to address the findings."

8. Keep a sense of humor and a good attitude.
Have a flexible staff that's willing to roll with the pace and unknown nature of the work. "You don't know what you're going to test next," says Torres. "It was really taxing on researchers. At times, I didn't know what we were testing until the night before, and I had to build a protocol and be ready to moderate the next day."


source:
Jakob Nielsen's Alertbox, July 28, 2008:
Weekly User Testing: TiVo Did It, You Can, Too
http://www.useit.com/alertbox/weekly-usability-tests.html
Weekly User Testing: TiVo Did It, You Can, Too (Jakob Nielsen's Alertbox)