The aim of discount usability approaches is to identify the usability problems of a web site, using low budgets. These approaches are
Open Issues
characterised by a lower attention on details, while trying to capture the major usability issues.
6.2.1 Discount Usability Testing
This approach, described by Nielsen in the famous article "Why you only need to test with 5 users", is a simple method for evaluating interfaces using a few (3-5) experts, based on the heuristic evaluation of usability problems (see chapter 4.1).
The idea is that only a few experts are needed to evaluate interfaces, considering that they are able to pick up the majority of problems.
Thus, the organizations does not need to invest resources to test hundreds or even tenths of users, because "5 users is enough".
Nielsen (1993) states that two or more usability specialist are able to identify a majority of the usability problems in a web, with the problem-identification percentage increasing as you add evaluators but only to a certain point. If the number of evaluators is increased over 5, the percentage of discovered problems does not increase proportionally with the increase of costs.
While two evaluators can identify over 50% of the usability problems and three can identify about 60%, it would take 15 evaluators to identify 90% of usability problems. In particular according to a research held by Nielsen and Landauer (1996), the number of usability problems found out with n users is N(1-(1-L)n), where N is the total number of usability problems and L is the proportion of usability problems discovered while testing a single user. According to Nielsen studies, L has a typical value of 31%.
A single test user lets the researchers learn "almost a third of all there is to know about the usability of the design".
Open Issues
Figure 3 - Problem-finding curve
The idea exposed by Nielsen is that "after the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new".
The efficacy of an usability test can be defined as the number of usability problems found with respect to the number of test users, considering that each test user has a cost. As shown in Figure 3, the curve flattens after the sixth user. In fact, the number of users needed to perform an usability test is still under debate, as we will see in chapter 5.3.
Nielsen approach is often guided by the economical aspect of usability testing. In a famous paper (Nielsen, 2003), he asserts that usability tests for a seven pages site can be done with a b udget of 200 dollars.
The test would be divided in the following parts:
• one hour to question customers about their information needs;
• one hour to review an early version of the design;
• one hour for a quick, in-store test of the design;
• one hour to enhance the site's search engine visibility.
Open Issues
Nielsen suggests also to adopt this approach in an iterative way: that is testing repeatedly a web site or a product, during its design phase, each time with a small number of test users. Iterative design combined with multiple tests of 5 users would be a good choice in most cases.
The effectiveness of this approach is still being debated: one weakness of the approach is the assumption made that it is easy to predict how many experts are needed in order to identify usability problems.
Another criticism to this approach is that often discount usability tests do not answer to the question "how usable is this web site" but simply find out usability problems and spots. In fact the discount approach does not use any usability metrics to evaluate the usability of the web site under investigation.
However, this approach is quite simple to apply and it is cost effective, so it is widely diffused and accepted, more in industrial than in academic researches. It is especially adopted when time and resources are short, since it does not require much time for the recruiting of the participants and for the evaluation of their answers.
It is commonly acknowledged that discount usability testing can be very effective as the first phase of a two phase usability testing, since it allows easily gathering most common usability findings and then focusing only on them with a laboratory based or interview based test.
6.2.2 Common Sense Approach
The common sense approach is a very practical approach based on the idea that in most cases usability analysis should be a matter of "common sense" and that everyone can learn how to do that.
It is very similar to the discount approach, since it suggests that it is not necessary to test many users, because only 5 users are enough.
Open Issues
The main difference between this approach and Nielsen one is that common sense approach suggests that expert or representative users are not needed for an usability test, since most "expert" users are in fact inexpert, and if an inexpert can succeed, everyone can.
This kind of usability test evaluates if the web sites are auto evident, that is that an user does not have to think while surfing the site, or at least auto explicative.
All these characteristics are tested and measured simply by common sense.
It is evident that this approach lacks in formality and methodology, since it does not provide any good help for conducting an usability test and for measuring the usability of a web site.
Moreover it is very hard to understand and to define common sense, as it is a very personal thinking and feeling, and often what is common sense is decided ex-post. This approach is generally based on opinions, and it is really hard to decide which is the best opinion.
Another pitfall of both common sense and discount usability approaches, is that they are not suitable for accessibility tests and, as we addressed in the previous pages, accessibility is a subset of an usability problem.