Google search bias: the eternal PR battle

Google's domination of the search market could enable it to engage in anti-competitive behaviour, and Ben Edelman believes he has evidence they're doing just that. Google strongly denies the claim, and points out Edelman's links to arch competitor Microsoft.

Edelman, an assistant professor at Harvard Business School in Boston, visited Australia this week to meet with business leaders and regulators in the continuation of a long-running public relations campaign against Google that last reached a peak in late 2010.

He documented instances where he believed Google had hard-coded parts of its search algorithm so that links to its own services appeared ahead of those to competitors.

The top Google search result for "mail" wasn't Microsoft's Hotmail, the most popular web mail service, nor long-standing provider Yahoo Mail, but Google's own Gmail.

"Now we know from comScore that fewer people click on the Gmail link than click on the Hotmail and Yahoo Mail links, and yet there it is front and centre. Google puts its own stuff at the top," Edelman told ZDNet Australia this week.

Edelman also showed how search results for a stock ticker such as "CSCO" for Cisco-listed Google Finance first, and those for a health-related term like "acne" listed the now-discontinued Google Health service first — even though those properties were not the most popular sites.

Similar searches with a comma added at the end, ie "acne,", yielded what Edelman considered to be more natural results.

"While exact searches for the listed terms yield prominent links to Google Health, any tiny variant causes the Google Health results to disappear completely. Compare results for 'a sore throat' (no Google Health results) to 'sore throat' (prominent Google Health results), and compare 'my acne' and 'stop acne' (no Google Health results) to 'acne' (prominent Google Health results)," Edelman wrote.

Edelman found that this was the case for all 2642 terms then listed on Google's Health Topics index page.

"It's not that the Google Health result would fall to position two or three or 10 or 20. It would disappear completely, never to be seen. And so what do we conclude from this? It looks like Google put its own stuff first," Edelman told ZDNet Australia.

Edelman now accuses Google of de-emphasising search results for companies that don't participate on Google's terms or resist takeover offers, including Yelp and Groupon.

"[Yelp] didn't want to be acquired by Google at the price Google unilaterally set. Google stopped linking to Yelp nearly as often, so Yelp links became much less prominent because Google started putting its own links front and centre," he said.

"That was the punishment for declining Google's acquisition. If you won't sell to Google at the price Google wants, then Google will bury you."

Edelman acknowledges this could be a coincidence, but said the same sort of behaviour can be seen "in category after category".

After Groupon refused Google's takeover offer, its emails started going to Gmail users' spam folders at a much greater rate, he said.

It's natural for any business to favour its partners. A doctor will refer patients to his friend the surgeon, for example. But Edelman sees two key problems with Google: its market dominance and its lack of transparency.

"If there were 10 small search engines each with a market share of 10 per cent, competition would cure most problems. If there was one search engine that was biasing its results too much towards advertisers or too much towards its house-brand services, consumers would notice that and would be able to go elsewhere," he said.

"The problem is when there's only one company it's a little bit hard to detect bias. What are we going to compare it to? How do we know whether Google has given us the best results, or just the results that it found most profitable and most convenient?"

"In Google's context, I believe it's misleading for them to bias the results in the way that they have to suggest that all of their results come from 'the algorithm', which Google has said, when the fact of the matter is that Google's staff can make manual adjustments and over-rides, and even the algorithm isn't as magical or infallible as they might have you believe."

Edelman has suggested a number of remedies for this perceived search bias. His favourite harks back 30 years to the days when airline reservation systems would heavily bias search results to their own company's flights.

"Even if you were flying from London to Paris, where there are non-stops on BA and Air France, Lufthansa would tell you you should change planes in Frankfurt. It's nonsense ... The regulators, of course, stepped in and said, hey, this is not fair game. A reservation system needs to provide fair and unbiased information. It's permissible to sort it this way, but not permissible to sort it that way," Edelman said.

Google has responded to Edelman's claims by reinforcing its message that it built Google to "delight" users, not necessarily website owners, aiming to provide "relevant answers as quickly as possible".

"Given that not every website can be at the top of the results, or even appear on the first page of our results, it's not surprising that some less relevant, lower-quality websites will be unhappy with their rankings," wrote Google Fellow Amit Singhal in 2010.

"We might show you a Google Map when you search for an address. But our users expect that, and we make a point of including competing map services in our search results (go ahead, search for 'maps' in Google). And sometimes users just want quick answers. If you type '100 US dollars in British pounds', for example, you probably want to know that it's '£63.9p' — not just see links to currency conversion websites," he wrote.

Google said its search algorithm is "one of the world's worst-kept secrets", with its core PageRank formula documented in academic journals and on Wikipedia.

"We provide more information about our ranking signals than any other search engine," Singhal wrote.

"We may be the world's most popular search engine, but at the end of the day our competition is literally just one click away ... not just to other search engines like Bing, but to specialised sites like Amazon, eBay or Zillow."

Edelman's Australian trip was organised by ICOMP, the Initiative for a Competitive Online Marketplace, which describes itself as global initiative funded by member contributions as well as sponsorship from Microsoft.

ICOMP makes no secret of the Microsoft sponsorship, noting the fact at the foot of every media release as well as on its website. However, Google considers that characterisation to be inadequate.

In March 2010 The Wall Street Journal reported: "One ICOMP signatory is a Spanish artisanal cheese maker called Artequeso, whose owner, Alfonso Álvarez, says he doesn't use computers. Mr Á said someone — he couldn't recall who — had asked him to join the group, and he signed because he opposes monopolies generally."

UK newspaper The Telegraph reported that Microsoft, as ICOMP's trustee, both selects the organisation's directors and guarantees its debts, and refers to Edelman as "a Microsoft consultant".

"ICOMP is an organisation whose sole purpose appears to be to attack Google," they wrote.