The answer is simple, scientists say: Amazon and most other e-tailers have yet to perfect a practice known as "data mining," the use of statistical analysis to uncover hidden patterns in otherwise random information.
Experts predict data mining will be one of the most revolutionary developments of the next decade, key to delivering a "personal Web," tailored to an individual's preferences, by identifying a useful structure in collected information and analysing it in real time. The influential MIT Technology Review recently hailed data mining as one of the 10 emerging technologies that will "change the world."
But some academics warn that mainstream mining merely "dumbs down" the sophisticated craft--and may result in screwy conclusions. Already, analysts are cautioning potential investors that the volatile segment may be unduly hyped.
"A lot of people think, 'I'm just going to put this in the hands of the marketer and we'll get the secret sauce,'" said Bob Moran, a managing vice president at the Aberdeen Group. "But there's no such thing as 'secret sauce.' Data mining is all about pushing back the gray zone. It's never entirely uncovering the black and white."
But marketers who recognise its vast commercial potential see data mining as more than black and white. They also see green in the science's potential to create higher margins and inflate revenue.
Sophisticated or not, various forms of data-mining development are being undertaken by companies looking to make sense of the raw data that has been mounting relentlessly in recent years. A recent article in the Engineering News-Record noted that e-commerce has empowered companies to collect vast amounts of data on customers--everything from the number of Web surfers in a home to the value of the cars in their garage.
"Over the past few years, while (database) construction has gradually taken up digital information tools in pursuit of efficiency and profit, a by-product--mountains of recorded data--has been gathering," Tom Sawyer wrote in a November edition of the industry trade publication. "Now, the realisation is spreading that the mountains are filled with gold."
About a dozen small data-mining companies are jockeying to gain market share, and database and software companies such as Oracle and IBM are edging into the field. Others are creating more automated data-mining applications for nonstatisticians, making the science more tangible to marketers and other algorithm-ignorant users.
Through data mining, marketers can target customers with personalized stock quotes, news updates, special promotions and other information they are most likely to use, dramatically reducing advertising budgets and boosting revenue. It is also entirely automated, reacting instantly to changes in a customer's behaviour, unlike the vast majority of personalised services on the Web today that require people to fill out questionnaires.
Perhaps the biggest challenge for data mining is one that many experts say cannot be solved--and one that may justify scepticism about the entire niche. Data mining is a good predictor of consumer behavior based on past behavior--what people are likely to purchase based on previous transactions, demographic information and other data points. But, critics say, it will never be able to predict what people really want to buy.
For example, data mining can determine that a 34-year-old, home-owning woman with two children is likely to purchase a detached microwave every three years for the next decade. Yet it cannot determine that this particular consumer would rather purchase a more expensive integrated microwave-convection oven combination if it came vaguely into her price range.
Kyle Johnstone, director of business intelligence for Emerald Solutions, said figuring out what people would rather purchase, as opposed to what they merely settle for, is the key to inflating profit margins--the ultimate goal of marketers. The only way to do that is to ask people what they really want, as opposed to relying on previous spending habits.
"People will tell you they like steak, but when they have parties for the Fourth of July, they buy hamburger. There's a disconnect between what you buy and what you desire," Johnstone said. "You can figure out the behavior of performance metrics, but what you're missing--the biggest piece of the puzzle--is what it is that people really want...It's mathematically impossible to determine that."
