Applying data mining to e-commerce
For decades, utility companies have been using data mining to predict with some accuracy when generators are likely to fail. The technique started making more inroads into the corporate world in the 1990s, catching on as a means to detect fraud in the insurance, health care and credit card industries. By finding patterns and predicting likely behaviour, companies can catch people who lie on applications or are likely to engage in dangerous or illegal activities.
Department stores, supermarkets and other brick-and-mortar retailers have used data mining to guess customer buying habits for years, but relatively few general consumer e-tailers and content producers have fully exploited the research technique. That's partly because the practice--involving algorithms, samplings and parallelisms--is complicated and poorly understood. But it's starting to find its way into the mainstream.
"E-commerce is the newest and hottest use," said Michael Gilman, president and chief executive of Data Mining Technologies. "Anywhere you have historical data, you can use it to get patterns that you can't see with the human eye."
One of the oldest and largest data-mining companies is the 25-year-old SAS Institute, which says it had already been working with 98 percent of Fortune 500 companies and is now targeting e-commerce. Retailers that sell products via catalogs and Web sites routinely increase their return on investment by more than 1,000 percent by using data mining, according to SAS statisticians.
"A lot of catalog companies were doing a fine business before, thank you very much," said Anne Milley, analytical strategist for SAS. "Then we came in and they were amazed. You look at who they're targeting, what they're sending and how often, and the frequency of repeat purchasers. You look at marketing mix--who is buying through catalogs, who is buying online--and figure out what is the optimal way to contact customers."
Data mining is likely to penetrate society further as the technology becomes easier to use.
Epiphany is one of several Web-based customer relationship-management companies that is deeply involved in data mining and is well known for its relatively easy-to-use tools.
George John, who has a doctorate in statistics from Stanford University and is the self-declared "data mining guru" of Epiphany, said the company's controversial simplification of data mining was intentional. He considers it one of Epiphany's biggest attributes when vying for business against other data-mining companies--which feature software that may be more sophisticated but is usually vastly more elusive to the average business.
"In the first generation of data mining at Epiphany, we tried to step back and see what business users would use it for--we knew they'd be asking lighter questions, where you wouldn't need 10 Ph.Ds forecasting profitability down to the penny," said John, an IBM veteran who began the data-mining program at Epiphany. "Every time we tried to make the (user interface) cleaner, we thought, 'Now the marketers will use it.' It was just paying attention to what people wanted to do."
Though it seems logical, the practice of simplifying data-mining results has its detractors. Fayyad and other experts warn that excessive simplification can skew results and lead executives to make pricing or inventory decisions based on faulty reasoning.
A more fundamental controversy is also brewing as data mining moves out of academia and into the corporate world: Academic statisticians take pride in their complex analyses, and many snub fellow Ph.Ds who enter corporate environments, calling them turncoats pandering to marketers.
John, the Epiphany guru, says he must constantly correct people who use the term "dumbing down" to refer to the company's color charts and other simple statistical diagrams. He prefers to call it "deeper penetration" of data mining into the ranks of marketers and other nonstatisticians.
"We profile a set of customers with nice charting, drawing pictures of what customers are like," John said, almost apologetically. "The key was admitting that was OK. It was OK if the technology behind it wouldn't get you a Nobel Prize."
