Skirting privacy concerns
Most data-mining companies get customer information from the corporate clients that hire them to build and host their databases for fees that usually start at about US$10,000 per month. The data miners skirt privacy concerns by keeping the information in-house.
They then crunch the data and send it back to the client in the form of spreadsheets, graphics, bar charts and other visual documents. Some data-mining companies also act as consultants, recommending to clients how to tweak Web pages for maximum effectiveness.
Few data-mining companies are willing to discuss real-world examples of how the craft has boosted sales or customers. But Usama Fayyad, a former Microsoft executive, who left the company to create DigiMine, said he used data mining to help revamp Microsoft's MSNBC.com Web site and boost readership.
Fayyad found that a 22 percent slice of MSNBC readers had nearly identical online behavior, clicking on exactly the same reports. But these users didn't fit into any of the company's five reader categories, which included political news-hounds, sports junkies and weather buffs.
Fayyad, who holds a doctorate from the University of Michigan, said his company determined that the glue holding this mysterious group together was vaguely scandalous stories similar to those in gossip tabloids. MSNBC changed its format significantly to appeal to this large group, and now the home page is required to have at least one such feature per day. The research helped turn MSNBC's Living section into the site's most popular destination, Fayyad said.
"The lesson is that before data mining, they didn't know what was happening to a quarter of their database," Fayyad said. "If three or four shelves fall over in a brick-and-mortar store, the customers won't walk around them and the clerks will fix them. The equivalent is happening on the Web, but no one knows how to fix the bottlenecks."
