Securing Microsoft 2: hackers invited to Redmond

Editors' note: This is part 2 in a series examining how Microsoft's security strategy has evolved over the past decade. Read part one here.

A limo speeds away from Seattle's Pioneer Square carrying an unlikely party on an unusual quest.

A group of security researchers and a member of Microsoft's security response team have bonded, in search of -- a haircut.

The expedition, held in September, was part of the Limo Races, a citywide scavenger hunt serving as the informal end to Blue Hat, the internal Microsoft security conference that started two years ago. The conference has become a twice-yearly event bringing some of the world's top hackers inside Microsoft's walls for two days of presentations before the software maker's executives and engineering ranks.

In the end, the team that included Microsoft's Andrew Cushman and IOActive's Dan Kaminsky failed in its mission. They found several tattoo parlours open for business, but no all-night barbers. None of them was really up for a buzz cut, anyway.

But while Cushman may have failed to win the Limo Races competition, he and his colleagues met a larger goal. Microsoft had succeeded in its twin aims for Blue Hat -- becoming a more accepted part of the security community and ensuring that the people writing Microsoft's code are acutely aware of the threats facing its products.

The company realised that security issues are about more than preventing buffer overruns and keeping up to date with the latest fuzzing tools.

"It is a really human problem," said George Stathakopoulos, the head of Microsoft's security response efforts. "The human element plays a massive role."

These days, Microsoft's security strategy is one that focuses on both people and technology. While Microsoft spends a fortune on automated testing and creating institutional processes to avoid bugs, it also spends money reaching out to its front-line engineers as well as to the security community that find the bugs that Microsoft misses.

That attitude represents a sea change from where the company was a decade ago. At that time, Microsoft took a hands-off stance toward the security research community. In its earliest days of security issues, the company didn't even disclose the extent of vulnerabilities.

"We had almost a cold-shoulder approach," Stathakopoulos said. The idea of talking more to the outside world was controversial, prompting meetings with "many raised voices," he said.

Stathakopoulos admits his was one of the voices arguing against such transparency.

"People already think our products are bad and if we start talking about those issues more and more, people will think we are horrible," Stathakopoulos said he argued at the time. But his boss, Mike Nash, persisted, arguing that the move would pay big dividends over time.

Ultimately, Stathakopoulos was at the forefront when it came time to take the next step toward being more open. He was among those who pushed Microsoft to start working more closely with the community of security researchers known for poking holes in the company's products.

"This is where we did most of our soul-searching," he said. "Are those good guys? Are they bad guys? Are they helping or hurting?"

In the end, Microsoft concluded that while they might have different approaches, both the company and bug hunters shared the goal of a more secure computing world. Plus, Microsoft needed them as allies.

That notion is still somewhat controversial and not universally accepted. There are some people who believe that once a bug is found it must be publicised so those at risk can protect their systems on their own, even if a patch from the vendor is not available. There are also frequent debates over what constitutes a reasonable amount of time.

And, there is still something of a culture clash at Blue Hat. Inside Microsoft's conference centre earlier this year, engineers sat quietly and tapped on their laptops as the invited researchers made their presentations. In the speakers' lounge, located in an adjoining building, the researchers snacked on munchies and watched their colleagues on a TV monitor.

Shane Macaulay spoke about ways of using Microsoft's new graphics engine to help visualise the way code interacts with data. At one point, he made a reference to "hookers" in the data. Although the term refers not to prostitutes, but to the practice of intercepting an application's call to the operating system, the word choice nonetheless prompted a chorus of snickers from the speakers' lounge.

Stathakopoulos wasn't among those cracking jokes, though. He was taking mental notes.

"Underestimating him would be a mistake," he whispered as Macaulay gave his talk.

Beginnings go back to Black Hat
Even though they were in separate rooms for the show, the fact the two groups are interacting at all is a sign of significant change. The seeds of today's collaboration can be traced back to 1997, when Microsoft first sent people to the Black Hat security show in Las Vegas. It took things a step further at Black Hat in 2003, renting out the Palms Hotel's ultra-hip Ghost Bar and inviting the attendees to have a drink at Microsoft's expense.

At first, Stathakopoulos said, it was like a high school dance, with the hackers on one side and the Microsoft people on the other.

But gradually, folks began talking, and conversations continued beyond Las Vegas. Cushman said that by engaging with the security community, they were able to see that inside Microsoft there were people as passionate about security and as smart as those in the outside community. Without such interaction, it was far easier for outsiders to assume that Microsoft just didn't care and for those inside Microsoft to assume that the security community just wanted to find holes in Microsoft products that others could exploit.

A better understanding helped fuel mutual respect, but Cushman was quick to warn executives and others at the company that there would be disagreements over the best way to protect customers. It's up to Microsoft, he said, to prove through actions that its approach worked.

Two years after its Black Hat party, Microsoft brought some of that crowd inside for the first Blue Hat, an event that plays off the Black Hat show and the blue badges worn by Microsoft employees.

Not everyone at the software maker was so enthusiastic about the idea, including Jim Allchin, head of Windows development at the time.

"When we started talking to the security community, he was not very happy about it," Stathakopoulos said. But Allchin warmed to the idea. "As a matter of fact, he was the first person sitting in the executive session," Stathakopoulos said.

Allchin ultimately became one of Blue Hat's biggest proponents, citing the event's value in goodwill and developer education.

The latter piece is extremely important. Microsoft has a large security team -- and smaller teams of people focused on security within its various product units. However, it relies heavily on the developer ranks to use secure processes when writing the millions of lines of code that go into flagship products like Windows and Office, as well as things like the Xbox and Windows Mobile.

Blue Hat has also led to closer relationships with some of the hackers. Kaminsky, who attended the first Blue Hat, now does extensive work with the company. A well-respected outsider, Kaminsky has quite a bit of sway with Microsoft's programming ranks.

Microsoft's success with Blue Hat has not gone unnoticed. Earlier this year, eBay launched its own "Red Team eBay" security conference, with plans for another next year.

Redmond's security outreach goes beyond Blue Hat, though. For one thing, Microsoft has hired a huge chunk of the security research community at one time or another. In particular, for Windows Vista, it sewed up much of the world's penetration testing expertise to find security holes in the operating system as it was under development.

"Microsoft has invested more money in securing Vista than anybody has ever invested into securing any piece of software before," said Halvar Flake, who authored a tool that analyses what parts of code a patch has changed. He is chief executive of security firm Zynamics, formerly known as Sabre Security.

Flake estimates that Microsoft has spent US$1 billion on its security push during the past few years. "One of the reasons Microsoft could afford doing this is because they are a monopoly. No company that is exposed to real market forces could afford spending $1 billion on securing software products," he said.

Stathakopoulos declined to confirm how much Microsoft spent hiring outsiders to do penetration testing but said whatever it amounted to was "well worth it."

"You get some pretty amazing perspectives," he said, noting Microsoft will likely do the same thing with the next version of Windows.

Imposing some limits
There are limits to Microsoft's connection to the hacker world. Microsoft's relationships are largely with the finders -- the people that discover security bug -- not with those who take the exploits and use them for malicious purposes. Microsoft relies on law enforcement to deal with the latter group.

There are also limits imposed by the other side. Not everyone who comes to speak at Blue Hat wants to get all snuggly with the company. Flake has spoken twice at Blue Hat, including the most recent one, but talks only infrequently with folks in Redmond.

"They sometimes tell me when I've caused them ludicrous quantities of work," Flake said. "There's not really an ongoing dialogue. I'm more focused on building the tools than doing unpaid quality assurance for Microsoft."

Still, whether it was influenced by Microsoft or not, Flake did shift his attention somewhat, using a variant of his tool to help analyse and classify malicious software, the topic he presented at Blue Hat this year. "It's hopefully a bigger market than the malicious market, which is quite small unfortunately," he said.

Although Microsoft has made significant inroads in the security community, it has also ruffled feathers at times, in particular with its entry into the security software market. That put Microsoft in competition with companies like Symantec and McAfee.

In that arena, the company struggled to create all of the human connections it needed to be successful. When it launched its OneCare product last year, it didn't have the relationships it needed with other antivirus software makers to get their virus definitions. As a result, the company scored poorly on some tests.

"If you don't get Kaspersky's collection, and that collection is in the test, and you're not having any detection for it, the numbers aren't going to be there," said Vinny Gullotto, a longtime McAfee executive who joined Microsoft last year as part of its effort to deepen ties to the antivirus community.

And, when it comes to giving back to the security community, some say Microsoft should do more.

One of those people is former Microsoft employee Window Snyder, who points to the work done by her current employer, Mozilla. Mozilla announced a program at this year's Black Hat to share some of its tools, starting with a JavaScript fuzzer. The company gave browser makers a heads up, and once they were OK with it, Mozilla released the tool publicly. Within days of having access to the tool, Opera found several bugs. The browser maker eventually contributed code back to the effort.

"I know all the major vendors have tools they develop," Snyder said. "They could share some of these with each other, making the Internet more secure for everyone. That's something I'd really like to see from Microsoft."

Stathakopoulos counters that, in many cases, Microsoft can be more successful by building its know-how into programs like its Visual Studio development tools, where they can be part of the overall software design process.

"I don't think people will adopt them if we start giving one-off tools," he said. "What you end up doing is you give a lot more opportunity to the evildoers. The minute you bunch them up and you build them into the process...then you will see a lot bigger adoption."

But all in all, Microsoft now gets fairly high marks for its social skills in the security community. Dragos Ruiu, who organises the CanSecWest security trade show, said Microsoft has gained much by developing relationships with the security community. Other companies should be taking note, he said. "I think they are going to reap the benefits for years."