Testing methodology
Creative Good conducted more than 50 consumer tests to generate the data for this report. The tests were conducted at an independent testing facility in a suburb outside New York City. We conducted the first round of tests in August 2000.
We tested eight major e-commerce sites in four categories important to holiday shoppers:
- Apparel: Gap.com, LandsEnd.com (not mentioned in our checkout discussion)
- Books and Music: Amazon.com, bn.com (Barnes&Noble)
- Electronics: BestBuy.com,Buy.com
- Toys: eToys.com,KBKids.com
The consumers recruited for testing were representative of customers who will shop online this holiday season. The consumers we tested met the following criteria:
- All had used the Internet for at least 6 months.
- All had successfully made an online purchase in the past.
- All had either bought gifts online last holiday season or were seriously considering buying gifts online for this year's holidays.
- None were Web developers, Web designers or Internet professionals.
- None worked for any of the companies whose sites we tested.
To simulate the actual shopping experience, we conducted three-quarters of the tests on dial-up connections and one-quarter of the tests on an ISDN line. We tested on PCs running Microsoft Windows and the Internet Explorer Web browser.
About 'listening labs'
The Holiday 2000 tests employed Creative Good's "listening lab" methodology. Though based primarily on traditional usability tests, listening labs are less task- oriented and more open-ended than usability tests. In particular, listening labs are one-on-one sessions in which the customer -- not the moderator -- sets the context. The labs are non-directed in an attempt to recreate the environment at home or work, where customers actually use the site -- and where there are no pre-defined tasks or scripted moderators sitting beside them.
Listening labs overcome some key shortcomings of usability and focus groups:
- Listening labs are non-task-oriented, since pre-defined tasks tend to neglect what each individual customer wants to do on the site, and often miss larger strategic findings. In our Holiday 2000 listening labs, we merely asked customers to use the site to shop for a holiday gift for someone on their gift-buying list.
- Listening labs rely less on quantitative measurement than some usability tests. For example, we did not use "eye-tracking" or any other detailed measurement during the tests. We did, however, quantify the most important metrics: how often, and where, customers succeeded and failed on the sites.
- Focus groups, in which a scripted moderator asks questions to a group of customers sitting in the testing room, measure what customers say they like. Listening labs observe what customers do, not what they say they do.
- Traditional usability and focus groups are controlled by a moderator, following a script and specifying the user's task. Listening labs are controlled by the customer, following whatever path he or she wants through the site.
For more information on the listening lab methodology, refer to Creative Good's Dotcom Survival Guide (free download).
Timing of screenshots
The screenshots featured in this report were taken in August and September 2000. Because sites are constantly changing, some of the sites may have redesigned since we took the screenshots. In fact, in most cases we hope that the site has redesigned by the time you read the report, so that the site improves the experience for its customers. The screenshots in the report depict what customers saw in our tests.
