This paper is an adapted excerpt from chapter 3, “A Basic Framework”, by Mayhew and Tremaine, in Bias and Mayhew's second edition of Cost-Justifying Usability (2005.) It presents a framework and example for cost-justifying usability engineering efforts during web development projects by describing how to calculate the costs and estimate the benefits of the Usability Engineering Lifecycle tasks (Mayhew, 1999) that can potentially be applied during the project.

This is a general framework that is relevant to cost-justifying a usability effort when developing any kind of software application from commodity trading to a Web e-commerce site.

A free Excel cost-justification tool downloadable from our Downloads page is a companion to this chapter excerpt. Table and page numbers referred to in this excerpt correspond to Table and page numbers in the book and referred to in the worksheets in the Excel spreadsheet tool.

The Usability Engineering Lifecycle

The first step in cost-justifying usability engineering on a particular web development project is to lay out a usability engineering plan for that project.

The Usability Engineering Lifecycle (Mayhew, 1999) documents a structured and systematic approach to addressing usability within the development process. The analysis framework presented herein is premised on the Usability Engineering Lifecycle approach.

General Approach

To cost-justify a proposed usability engineering plan, you simply adapt a very generic and widely used cost-benefit analysis technique. Having laid out a detailed usability project plan based on Usability Engineering Lifecycle tasks (see Mayhew, 1999), it is a fairly straightforward matter to calculate the costs of that plan. Then you need to calculate the benefits. This is a little trickier, and it is where the adaptation of the generic analysis comes into play (see the book chapter for details.) Then, you simply compare costs to benefits to find out if and to what extent the benefits outweigh the costs. If they do to a satisfactory extent, then you have cost-justified the planned usability effort.

More specifically, first a usability engineering plan is laid out. The plan specifies particular techniques to employ for each Usability Engineering Lifecycle task, breaks the techniques down into steps, and specifies the personnel hours and equipment costs for each step. The cost of each task is then calculated by multiplying the total number of hours for each type of personnel by their effective hourly wage (fully loaded, that is, including salary, benefits, office space, equipment, utilities and other facilities), and adding up personnel costs across types. (Sometimes it is hard to get data on fully loaded wages for an organization. In this case, I use a rule-of-thumb I have heard informally to simply double the before-tax annual salary, and then divide by the typical number of hours a full time worker is paid for in a year, usually about 2000.) Even if my audience is unwilling or unable to give me actual figures for fully loaded wages, they can contest – or not – my ballpark figure based on this rule of thumb.) Then the costs from all tasks are summed to arrive at a total estimated cost for the plan.

Next, the overall benefits of the specific usability engineering plan are predicted by selecting relevant benefit categories, calculating expected benefits by plugging project-specific parameters and assumptions in to benefit formulas, and summing benefits across categories.

In the case of web sites and web-enabled applications, the potential benefit categories relevant to a particular cost-benefit analysis will depend on the basic business model for the web site being developed. Benefit categories potentially relevant to different types of sites are summarized in Table 3.1 on page 58 of the book, and in the Benefits Categories worksheet of the Excel spreadsheet tool.

Note that the relevant benefit categories for different types of web sites and web-enabled applications vary somewhat. In a cost-benefit analysis, one wants to focus attention on the potential benefits that are of most relevance to the bottom line business goals for the site, either short term or long term or both

Note also that these benefits represent just a sample of those that might be relevant in these types of sites, and does not address other possible benefits of usability in other types of sites. Others might be included as appropriate, given the business goals of the site sponsors and the primary concerns of the audience, and could be calculated in a similar fashion as those described below.

Finally, overall benefits are compared to overall costs to see if, and to what extent, the overall usability engineering plan is justified.

When usability practitioners are invited to participate in web development projects already in progress, which is often the case, they have less chance of including all usability engineering Lifecycle tasks, and of influencing overall schedules and budgets. They are more likely to have to live within already-committed-to schedules, platforms and system architectures, use short cut techniques for usability engineering Lifecycle tasks, and minimally impact budgets. Nevertheless, it is almost always possible to create a usability engineering plan that will make a significant contribution to a web development project, even when one comes in relatively late. And, you can use the cost-benefit analysis technique to prepare and support even usability engineering plans that involve only parts of the overall usability engineering Lifecycle, and only short cut techniques for tasks within it.

Let us consider a hypothetical usability engineering plan, and see how its costs can be estimated. Then we will incorporate this plan into a scenario involving the development of one type of web site, and see how you would conduct a cost-benefit analysis of that plan for that project. The scenario we will use involves development of a product information web site.

A Product Information Web Site

This example is based on a hypothetical scenario given in a Forrester report from June 2001 called “Get ROI from Design” (Souza, 2001.) It involves an automobile manufacturing company, who has put up a web site that allows customers to get information about the features of the different models of cars they offer, and options available on those cars. It allows users to configure a base model with options of their choice and get sticker price information. Users cannot purchase a car on-line through this web site – it is meant to generate leads, and it points users to dealerships and salespeople in their area.

First, let's look at the overall results of the cost-benefit analysis for this scenario. We will assume the usability engineering plan with its associated cost that is shown in Table 3.2 on page 61 of the book, and presented in the TOTAL Costs worksheet of the Excel spreadsheet tool. The project usability engineer estimated that in the case of this project, the usability engineering plan will produce a new web site design with the expected benefits every month summarized in Table 3.31 on page 93 of the book, and in the ProductInfo worksheet in the Excel spreadsheet tool.

Comparing these benefits and costs, the project usability engineer argued that the proposed usability engineering plan will pay for itself in the first six months after launch, as shown in Table 3.32 on page 93 of the book and in the ProductInfo worksheet in the Excel spreadsheet tool , and that after that period benefits will continue to accrue at a rate of $37,500 per month. As the new site is expected to have a lifetime of something much longer than 6 months, the project usability engineer expected the usability engineering plan to be approved based on this cost justification.

Note that in this case the project usability engineer predicted benefits on a monthly basis, and then divided the total cost of the usability engineering plan by the total predicted monthly benefits to determine a “payoff period”, that is, the point at which the benefits accrue to equal the cost. After that point, net benefits are predicted to accrue on a monthly basis in the amount of the total monthly benefit each month.

1. Start with the Usability Engineering Plan

In this scenario, we start with the assumed plan represented in Table 3.2 on page 61 of the book and in the TOTAL Costs worksheet of the Excel spreadsheet tool.

2. Establish Analysis Parameters

Analysis parameters for this example are presented in Table 3.33 on page 94 of the book and in the TOTAL Costs worksheet of the Excel spreadsheet tool. In this scenario, we assume there is an existing site with known traffic statistics, and the project involves a redesign.

3. Calculate the Cost of each Usability Engineering Lifecycle Task in the Usability Engineering Plan

The project usability engineer used the cost calculations as shown in Tables 3.6 to 3.18 on pages 66 - 72 in the book and in the UserProfile through DUID worksheets of the Excel spreadsheet tool. These show the derivation of the numbers summarized in Table 3.2 on page 61 of the book and in the TOTAL Costs worksheet of the Excel spreadsheet tool.

4 . Select Relevant Benefit Categories

Since this is a product information site , only certain benefit categories are of relevance to the business goals of this redesign project (see Table 3.1 on page 58 in the book and the Benefits Categories worksheet in the Excel spreadsheet tool.) The project usability engineer decided to include just a single benefit category in the analysis: increased lead generation .

The project usability engineer selected this benefit category because s/he knew it would be of most relevance to the audience for the analysis, the business sponsors of the web site. There may be other very real potential benefits of the usability engineering plan (such as decreased late design changes), but s/he chose just this one for simplicity and to make a conservative prediction of benefits.

As compared to the existing web site design, the project usability engineer anticipated that in the course of redesign, the usability engineering effort would increase leads by insuring that visitors can find basic information, and successfully configure models with options. Achieving this will depend upon conducting the usability requirements analysis and usability testing activities in the proposed plan, as well as on applying general user interface design expertise.

5. Predict Benefits

Next the project usability engineer predicted the magnitude of the benefit that would be realized - relative to the current web site, which is being redesigned - if the usability engineering plan (with its associated costs) was implemented. Thus in this case, s/he predicted how much higher the lead generation rate would be on the site if it were re-engineered for usability as compared to the existing site. Table 3.34 on page 94 in the book and the ProductInfo worksheet in the Excel spreadsheet tool present the benefit assumption made in the analysis, and Table 3.35 on page 94 in the book and the ProductInfo worksheet in the Excel spreadsheet tool present the benefit calculations, which incorporate both the benefit assumption and the analysis parameters.

The project usability engineer defended the very conservative assumption that a better interface will result in 1% more leads each month based on traffic statistics from the current web site that show a much higher rate of abandoned configuration attempts than this, and by pointing out existing design flaws that might account for this and that could be rectified in the redesign process.

6. Compare Costs to Benefits

Next the usability engineer compared benefits and costs to determine the pay off period. This is shown in Table 3.32 on page 93 in the book and in the ProductInfo worksheet in the Excel spreadsheet tool.

The project usability engineer's initial usability engineering plan appears to be well justified. It was an aggressive plan, in that it included all lifecycle tasks, and the more reliable and thorough techniques for each task. In addition, the benefit assumptions were conservative. Given the short estimated pay off period, s/he would be wise to stick with this aggressive plan and submit it to project management for approval and funding.

Summary

The particularly critical value of usability to the return on investment (ROI) of web sites and web-enabled applications is illustrated by the following case study.

A contract development team was building a web site for a client organization. The web site was to include up-to-date drug information and was intended to be used by physicians as a substitute for the standard desk references they currently use to look up such drug information as side effects, interactions, appropriate uses, and data from clinical trials. The business model for the site was an advertising model. Physicians were expected to visit the web site regularly because more current information would be available, and that information would be more readily findable on the web site, relative to published desk references (such as The Physician's Desk Reference, or PDR.) The marketing plan was to have pharmaceutical companies buy advertising for their drug products on the web site because the visitors to the web site (physicians) represented their target market. Regular and increasing traffic due to repeat visitors, and new visitors joining based on word-of-mouth amongst physicians, would drive up the value of advertising, generating a profit – an ROI - for the client.

The development team generated a prototype design that the client would use to pursue venture capital to support the full-blown development and initial launch and maintenance of the web site. The client paid for this prototype development.

Once the prototype was ready, a usability engineering staff member was brought in to design and conduct a usability test. Eight physicians were paid to fly to the development center and participate in usability testing. Several basic search tasks were designed for the physicians to perform. They were pointed to the prototype's Home page, and left on their own to try to successfully find the drug information that was requested in the first task.

Within 45 seconds of starting their first search task, seven out of the eight physicians gave up, and announced, unsolicited, that the web site was unusable and that if it were a real web site, they would abandon it at that point and never return.

Clearly, if the web site had launched as it was designed prior to this test, not only would an optimal return on investment (ROI) not have been realized, but in fact, the web site would have failed all together and a complete loss of the clients' investment would have resulted. If 7/8ths of all visitors never returned, enough traffic would not have been generated to have motivated advertisers to buy advertising. The entire investment would have been lost.

Instead, the test users were asked to continue with the entire test protocol, and the data generated revealed insights into the problem that was a show-stopper on the first task, as well as other problems uncovered in other test tasks. The web site was redesigned to eliminate the identified problems. Clearly, the usability test, which had an associated cost, was worth the investment in this case.

This true anecdote illustrates something that distinguishes web sites from commercial software products. In a commercial software product, the buyers discover the usability problems only after they have paid for the product. Often they cannot return it once they have opened the shrink wrapped package and installed the product. Even if it has a money-back guarantee, returning it takes some effort and they are not likely to return it, and anyway, perhaps there are not many alternative products on the market with noticeably greater usability.

On a web site, on the other hand, it costs the visitor nothing to make an initial visit. On a web site based on an advertising model, such as the one described, the web site sponsor makes nothing at all unless there is sufficient ongoing traffic to attract and keep advertisers. On an e-commerce web site, the sponsor makes nothing at all unless the visitors actually find and successfully purchase products, and unless usage of the web channel increases sales or reduces the costs of other sales channels.

A web site is not a product and the user does not have to buy it to use it. The web site is just a channel, like a TV show, magazine, or catalog, and if users do not find and repeatedly and successfully use the channel, the investor gets no ROI for having developed the channel. Thus, usability can absolutely make or break the ROI for a web site even more so than for traditional software products.

It's also true that success competing in the marketplace is even more dependent on relative usability on web sites than is the case with traditional software products or sales channels. Someone wishing to buy a book may be inclined to buy from a particular bricks and mortar bookstore that is easy to get to, even if it's not the best bookstore around. On the other hand, if a customer cannot easily find the desired book through, for example, the barnesandnoble.com web site, they need not even get out of their chair to shop at a competitor's site instead – for example, amazon.com. It's not enough to simply have a web site that supports direct sales; your web site must be more usable than the competition's web site (as well as have equivalent or superior content), or business will be lost based on the relative usability of the selling channel alone. For example, 60% of a sample of consumers shopping for travel online stated that if they cannot find what they are looking for quickly and easily on one travel web site, they will simply leave and try a competitor's web site (Harteveldt, 2000.)

In addition, if you are a catalog order company such as LL Bean or Land's End, and your product is good but your e-commerce web site is bad, customers will not use your web site and will resort to traditional sales channels (fax, phone) instead. This will result in a poor ROI for the web site that was intended to be justified by relatively low operating costs compared with more traditional catalog sales methods.

Web site usability is the equivalent of good - or great - customer service. Consider a company's current level of investment in traditional customer service channels. They need to make an equivalent investment in the usability of a web site meant to replace or augment traditional channels of customer service. Web site usability – like good traditional customer service – insures that customers can find what they want to buy – an obvious prerequisite to sales. Just as salespeople in stores help you find the book you want, so a good user interface on a bookseller's web site must make browsing and searching easy and successful. Usability - like good customer service - also reduces errors in business transactions, and the corresponding costs to fix those errors. For example, just as a good catalog offers accurate pictures of clothing and size charts to minimize the costs of processing returns, a clothing web site must also insure that customers don't order clothing in the wrong size or color to minimize returns. This might entail providing multiple views of the garment, and sizing charts. In addition, usability - like good customer service - motivates customers to choose to use a web site over traditional methods of doing business, insuring an ROI in the web site itself. Usability helps insure customers will return repeatedly to a web site, just as good salespeople help insure customers will return to bricks and mortar stores – again contributing to ROI. And, usability done right in initial development is always cheaper than fixing usability problems identified after launch – or going out of business.

By contrast, lack of usability on an e-commerce web site is the equivalent of poor customer service. Imagine having the following conversation with a human telephone customer service representative (CSR):

•  CSR: Hello. Thank you for calling XYZ Shopping, how can I help you?

•  Shopper: Hello, I would like to place an order.

•  CSR: Do you know the name and extension number of the order taker?

•  Shopper: Of course not! I just want to order something! Can't you take my order?

•  CSR: Heck no, you need to know who to ask for!

Much later, after finally finding out how to reach the order taker, and then being put on hold repeatedly, and then finally completing giving the required order info:

•  Shopper: Oh, I just realized I need to make a change - can you do that?

•  CSR: If you don't know the name and extension of the order changer, you will just have to wait until your order arrives, then send it back and reorder . . .

Would you shop on-line with this company again after such an experience? Unfortunately, this hypothetical interaction is very analogous to many on-line shopping experiences. E-commerce web sites too often make it very difficult to figure out how to place an order and make changes in mid-stream, and there are often long periods of being “on hold”, waiting for graphics-heavy pages to download.

Thus, there are real risks associated with not investing in web usability. When customers are unsatisfied with the quality of customer service on web sites, customer loyalty erodes. Even companies with well established brand loyalty, may find this loyalty beginning to decline with the launch of an unusable web sites meant to replace traditional customer service channels. Customer dissatisfaction may result from an unacceptable learning curve to accomplish desired goals (as in the example of the physician's drug reference web site cited earlier), from unacceptable task times to accomplish desired goals (e.g., too many clicks, download times that are too long), or from an unacceptable rate of errors and confusion during task completion (e.g., abandoned shopping carts). Potential sales may be lost because customers can't find what they want to buy or get the information they need to make buy decisions. Customers may make errors in business transactions that cost time and money to rectify, and create customer dissatisfaction. For example, a vendor recently agreed to pay my return shipping costs because the web site erroneously misled me into ordering an item twice. In another case, a shopper was charged California sales tax on a product shipped to New Jersey because the “out of state” button was placed in a non-obvious place on the page. We shoppers will think twice about using these web sites again, because of the extra time lost in repairing the needless errors. Customers may remain loyal but refuse to use a web site and return to traditional methods of doing business, reducing the ROI on the web site investment. Customers may also defect to competitor companies whose web sites are more usable. And, costly rework of a web site to fix problems discovered after initial launch will eat into the ROI.

The cost-benefit analysis example offered in this chapter is based on a simple subset of all actual costs and potential benefits, and very simple and basic assumptions regarding the value of money over time. More complex and sophisticated analyses can be calculated (see Karat, chapter 4 in Bias and Mayhew, 2005.) However, usually a simple and straightforward analysis of the type offered in the example above is sufficient for the purpose of winning funding for usability engineering investments during web development in general, or planning appropriate usability engineering programs for specific web development projects.

The sample cost justification analysis offered here suggests that it is usually fairly easy to justify a significant investment of time and money in usability engineering during the development of web sites and web applications. The framework and example presented in this chapter, along with the free Excel tool available from our Downloads page, should help you demonstrate that this is the case for your web development project.

References:

Bias, R. G., and D. J. Mayhew. 2005. Cost Justifying Usability – An Update for the Internet Age . San Francisco : Morgan Kaufmann Publishers.

Bias, R. G., and D. J. Mayhew. 1994. Cost Justifying Usability . Chestnut Hill , MA : Academic Press.

Bias, R. G., D. J. Mayhew, and D. Upmanyu. 2003. Cost Justification. In The Handbook of Human-Computer Interaction, edited by J. Jacko and A. Sears. Mahwah , NJ : Lawrence Erlbaum Associates.

GVU. 1999. http://www.gvu.gatech.edu/user_surveys . Atlanta , GA : Georgia Institute of Technology.

Harteveldt, H. H. 2000. Travel Data Overview. Cambridge , MA : Forrester Research.

Karat, C.M. 1989. Iterative Usability Testing of a Security Application. Proceedings of the Human Factors Society 33rd Annual Meeting pp. 273-277, HFES.

Mantei, M. M. and T. T. J. Teorey. (1988). Cost/benefit for incorporating human factors in the software lifecycle. ACM Communications 31 (4), 428-439.

Mayhew, D. J. 1999. The Usability Engineering Lifecycle . San Francisco , CA : Morgan Kaufmann Publishers.

Mayhew, D.J., and M. M. Tremaine. 1994. A Basic Framework for Cost-Justifying Usability Engineering. In Cost justifying Usability , edited by R. G. Bias and D.J. Mayhew. Chestnut Hill , MA : Academic Press.

Mayhew, D. J., and R. G. Bias. 2003. Cost-Justifying Web Usability. In Human Factors and Web Development 2 nd Edition , edited by J. Ratner. Mahwah , NJ : Lawrence Erlbaum Associates.

Schmitt, E. 1999. Measuring Web Success. Cambridge , MA : Forrester Research.

Sonderegger, P. 1998. The Age of Net Pragmatism. Cambridge , MA : Forrester Research .

Sonderegger, P. 2000. Scenario Design. Cambridge , MA : Forrester Research .

Souza, R. K. 2000. The Best of Retail Site Design. Cambridge , MA : Forrester Research.

Souza, R. K. 2001. Get ROI From Design. Cambridge , MA : Forrester Research.

Vaananen-Vainio-Mattila, K. & Ruuska, S. 2000. Designing mobile phones and communicators for consumers' needs at Nokia. In Bergman, E., Ed. Information Appliances and Beyond . San Francisco: Morgan Kaufmann Publishers.