Usability Testing: You Get What You Pay For: “Also, if you have multiple distinct user categories, then you must test 6-8 of each type.”
She illustrates the point with a story of someone using developers to test a site. Now, I think most people know they shouldn’t use developers to test a consumer site, but the idea that your sample has to be part of your target audience when doing usability testing is wrong. It all boils down to how you’ve defined your target audience. Usually this is defined as the audience you want to reach with the website (the people that will make you money), which is not the same as the audience that you can do usability testing with, that’s usually a lot larger. I’m not saying you can test with anyone. But depending on what exactly you’re testing, you can do cheap and cheerful testing of generic interface properties with most people.
“i.e., have the same set of key characteristics, such as job experience level, computer experience level, age, gender, expected frequency of use, etc.) ” I mean, come on. How much difference will those characteristics make in most cases? That’s just wrong.
Defining your target audience is done for a purpose. Usually for marketing (who do we sell to and how?). The better developer does it again for design (marketing profiles don’t contain a lot of useful info when designing a solution). But 99% (a Nielsen-esque exaggeration) of usability problems can be uncovered by testing with almost anyone.
I guess it depends on the type of projects you work on. One of the examples shows “doctors, nurses, technicians and receptionists” – of course those are very specialized audiences and you should test with them. Again a bad example.
I just don’t like the tone of the article, the false examples, the “I’m an expert therefore I only can test” tone.
I think the flaw of the article is that the author seems to assume a “one time fix it all” type of testing, ignoring iterative design practices where you test things, say, three times a week. I’m all for in depth testing but it doesn’t mean quick and dirty testing is useless.