Sep 19 - Nielsen, Location is Irrelevant for Usability Studies (Alertbox)

Location is Irrelevant for Usability Studies

Summary: You get the same insights regardless of where you conduct user testing, so there's no reason to test in multiple cities. When a city is dominated by your own industry, however, you should definitely test elsewhere.


As long as you're testing within a single country, there's no reason to expend resources traveling to multiple cities and conducting the same usability study again and again. You'll simply observe the same behaviors repeatedly, and learn nothing new.
Because traditional wisdom recommends conducting research in multiple locations, we've done so for many projects over the years. But, except for the few special cases discussed below, we've always identified the same usability findings, no matter where we tested.
By now, we can clearly conclude that it's a waste of money to do user testing in more than one city within a country.

Behavior vs. Opinion

Why does usability differ from market research when it comes to the number of required study locations? Because with usability, we test behavior, not opinion. Further, we test that behavior with a defined artifact (i.e., a specific user interface).
But when it comes to reacting to a set of interaction design options, people usually interpret the screen elements the same, no matter where they live. What's easy in one city is just as easy in another city.

Why One Location is Enough

In most cases, differences are due to diversity in the users' circumstances, not to geographical variation. It's therefore best to recruit a diverse set of users: some experienced users, some novices; some young, some old; some doctors, some nurses; some loyal customers, some who swear by your competition. Usually, you can cover the spectrum of user profiles in one location. What matters is the differences between people and their behaviors, not the differences between cities.
Another reason to limit your tests to your preferred city? It's usually where you're based. When you test in your own city, it's much easier for other members of the design and development team to observe the test.

When to Test Elsewhere

As often in usability, there are exceptions to the general rule. In a few cases, test location does matter.

Sometimes, a single industry dominates an area to the extent that it's considered a company town. Such dominance might be due to a single company or to several similar companies. Example company towns include:
* Detroit, if you're testing a car site
* Washington, DC, if you're testing a government site
* Hollywood, if you're testing a movie site
* Downtown Manhattan, if you're testing an investment site
* Brussels, if you're testing a European Union site
* Silicon Valley, if you're testing a technology site

In recruiting our test users, we followed usual procedures and screened out anybody working in usability, Internet marketing, interface design, graphic design, or programming. Unless they're your target customers, you don't want such people in a test because they can't stay within their role as users. They always have to be critics as well, and comment on how your design compares with their pet theories on what makes a good website.
The entire idea of usability testing is to observe how customers use your interface, not to hear them speculate on how other people might use it.

Another case where test location matters is for international use, which can obviously be studied only by testing in multiple countries.

Finally, some products are simply not used (or are used very differently) in some locations.
For example, you shouldn't test a website to sell a particularly powerful home heating system in Florida, where most people don't have experience with high-end heaters.
Similarly, you shouldn't test a tourist site in the city it's promoting, because it's targeted at travelers coming from elsewhere.

Conclusion:
The general rule remains clear: Usability findings are typically the same, no matter where you test. So, save your travel budget and conduct your studies in a single city.

Source:
Jakob Nielsen's Alertbox, April 30, 2007:
Location is Irrelevant for Usability Studies
http://www.useit.com/alertbox/user-test-locations.html
Location is Irrelevant for Usability Studies (Jakob Nielsen's Alertbox)