After a round-the-clock weekend watch for any infection of the so-called SQL Slammer worm--also know as Sapphire and SQL Hell--that hammered other companies' networks, the software maker apparently had escaped with only minor incidents in its international offices.
Until Tuesday. More than three days after the worm started spreading on the evening of January 24, Slammer somehow got into Siebel's internal network and sent traffic skyrocketing.
"It tied up our network," said Mark Sunday, chief information officer for the California, manufacturer of e-business applications. "The quantity of network traffic generated was an order of magnitude greater than anything we had seen before."
The disturbing lesson: Regardless of what protective measures have been taken, no network can be considered secure. Companies deemed bastions of security--such giants as Bank of America, American Express and Microsoft, under its year-old Trustworthy Computing initiative--found their internal networks deluged with data from the Slammer attack.
Although the worm caused roughly US$1 billion in damage by some estimates, its most significant casualty may be the perception that companies can remain secure by keeping up with software patches and other protective updates. Instead, security experts say, companies need to begin treating such attacks as inevitable and focus on limiting their damage, rather than expending every effort trying to create an ironclad perimeter.
"We have recognised over the last few years that you cannot prevent a virus," said Joe Hartmann, director of North American antivirus research for security software firm Trend Micro. "There will always be an entry point."
Many of last week's victims found that their internal networks were more vulnerable than they should have been. And with a worm like Slammer, a small crack in the security surrounding a company can mean days, if not weeks, of cleaning the infection from inside systems.
In Siebel's case, the worm wreaked havoc even though the company had moved the lion's share of its network infrastructure to two data centres in Utah. Slammer was still able to swamp the company's internal network at its San Mateo headquarters, limiting use of email and other resources for more than 24 hours while security teams hunted down infected servers.
"This ranked right up there with Nimda and Code Red," said Peter Allor, director of operations for the Information Technology Information Sharing and Analysis Centre, one of many federal agencies that watch for major threats to critical networks.
The SQL Slammer worm, at 376 bytes of computer code, is much smaller than either Code Red's estimated 4KB (4,096 bytes) or Nimda's 60KB (61,440 bytes). Exploiting a hole that had been announced and patched by Microsoft exactly six months earlier to the day, the worm inundated other computers on the Internet with a copy of itself. The worm's small size meant that it could send itself out in a single data package, or packet, that automatically infected the victim by loading Slammer into memory.
That efficiency made Slammer the fastest-spreading worm to date, infecting 90 percent of all vulnerable servers in its first 10 minutes, according to a report by a coalition of researchers from University of California San Diego, Lawrence Berkeley National Labs, and Silicon Defense, a security consultancy.
Code Red, by contrast, had to first search for vulnerable servers and then send a copy of itself afterward. While the number of servers infected by Code Red doubled every 37 minutes, Slammer-infected servers doubled every 8.5 seconds. Code Red infected nearly 400,000 computers in July 2001, while Nimda proliferated tenaciously through corporate networks two months later.
Yet Slammer was far easier to clean up than Nimda: Restart a server, and it was gone. And the worm infected 200,000 Microsoft SQL servers, about half the number of machines hit by Code Red, according to data from Incidents.org. Nevertheless, the effects of the SQL Slammer were more visible than those of its predecessors.
"This worm did something that we have not seen before in that the customer is seeing it," Allor said. In the past, a worm or virus would disrupt the internal operations of its victims but, most times, would not be noticed by anyone outside the companies.
"In this case, the customer was affected," Allor added. "People weren't getting dial tones; airplanes couldn't fly; ATMs weren't giving cash."
Those problems weren't caused by any particular feature of the self-replicating program; they were caused by the fact that the worm attacked a weak point in many companies' systems--the database.
"You can have millions of users all over the world that need to get information from that database," said Phyllis Schneck, chairwoman of the national executive board of the FBI InfraGard, the security world's equivalent of the Rotary Club.
"Those people might not be vulnerable, but because they have to interact with the database they are affected."
That's exactly what happened to Bank of America, whose automated teller machines suddenly stopped dispensing cash early Jan. 25. The reason: The sheer volume of data produced by servers infected with Slammer smothered databases in Bank of America's internal network.
"When a person uses an ATM, (the ATM) communicates with databases on our internal networks," said Lisa Gagnon, a spokeswoman for the bank. "That communication couldn't happen because our network was so congested." Indeed, a single infected server churning out copies of the worm could theoretically congest the bandwidth of a 100Mbps Ethernet, according to analyses of the program.
Later that evening, most of the company's ATMs were back in service. The company located the entry point but, for security reasons, would give no details of how the worm got in.
"Either the patch failed or we missed some servers when we applied the patch," Gagnon said. "Going forward we will analyse what we can do to make sure this doesn't happen again."
Easier said then done. Even technology powerhouse Microsoft wasn't fully prepared for the worm, despite a year-long focus on improving corporate and software security.
A string of emails circulated within Microsoft's internal information-technology groups, subsequently viewed by CNET News.com, portrays a chaotic scene as the company scrambled to fight the SQL Slammer's onslaught.
"All apps and services are potentially affected and performance is sporadic at best," Mike Carlson, director of data centre operations for Microsoft's Information Technology Group, stated in an email sent at 8:04 a.m. PST on Jan. 25. "The network is essentially flooded with traffic, making it difficult to gather details concerning the impact."
Rick Devenuti, Microsoft's chief information officer, said the software giant learned some hard lessons.
While the company had hardened its networks, it hadn't cut connections between buildings on all ports--the software addresses on which an application listens for data from the network. The worm found its way to a connected port and, because the buildings weren't isolated, was able to spread throughout the campus.
"It just takes one machine to get going," Devenuti said in an interview last week. "At any given point in time, it is hard to be 100 percent patched with any machine. We are working hard to make patch management easier, but 100 percent is a high bar, and in this case we are not there."
Many corporate network administrators insist that they can win the race to close the holes through which worms squirm, but a rising number of security veterans are questioning whether companies should continue to make "totally patched" the rallying cry of their defence strategies.
"Slammer showed us that it's hard for everyone to keep up with patches, no matter who you are," said Mary-Ann Davidson, chief security officer for database leader Oracle.
Like Microsoft, Oracle has struggled to prioritise the proliferation of patches so that system administrators can get the most critical ones in place first.
Such a ranking is necessary to ease the burden on overworked administrators and to help companies gauge whether to spend time testing the patches to ensure that they don't break critical functions of the system they are supposed to protect.
"The last thing you want is your order management system to go down in the last quarter because you didn't get a chance to test a patch before applying it," Davidson said.
Microsoft knows that well. A month before Code Red, the company twice had to release a patch for its Exchange servers to fix problems that the software upgrade introduced into the applications. Then the software giant pulled a December patch for Windows NT after reports that repaired systems were crashing.
The problem is simply one of time, said Steve Solomon, CEO of security software maker Citadel Security Software. His company develops systems that automate the installation of patches--assistance that is welcomed by many taxed network managers.
"Right now," Solomon said of these overburdened administrators, "the task is so tedious and so time-consuming they can't keep up with the manual process."
There's no excuse. This vulnerability was easily averted by anyone who simply is alert to new service packs and bug fixes.
After all, that's simply "part of the job" of being an I.T. _professional_, rather than a _hobbyist_.