The buffer overflow vulnerability affects versions of America Online's popular ICQ instant messaging software prior to version 2001b, which was released October. Only versions for Microsoft's Windows operating system are vulnerable.
AOL posted a page urging people who haven't already downloaded the latest version of ICQ software to do so.
"We are encouraging people to upgrade," AOL representative Andrew Weinstein said. "And we are taking additional server-side precautions. But we do not believe this vulnerability has ever been exploited."
AOL learned of the vulnerability, which lies in the application's Voice Video & Games feature, after an alert was posted to the Bugtraq security mailing list.
The company said it worked with discoverer Daniel Tan, a sophomore at the University of Pennsylvania majoring in computer science and business, to address the problem. AOL has weathered criticism in the past for its accessibility to and treatment of bug hunters.
It is the second buffer overflow vulnerability to surface in AOL's instant messaging software since the beginning of the year.
The first, in AOL Instant Messenger (AIM), affected Microsoft Windows-compatible versions 4.7 and 4.8 beta.
The holes have surfaced as security analysts are giving IM applications new scrutiny. Although virus and worm authors have thus far concentrated on e-mail as a means of propagation, the rising popularity of instant messaging has made the technology an increasingly attractive target.
Buffer overflows are among the most common computer security glitches. They crop up when an application crashes after being flooded with more code than it can accommodate. In a buffer overflow attack, maliciously written excess code can wind up being executed on the target computer.
"Worse case scenario is that if someone sent you a message, and you click on it, it would be possible to execute arbitrary code," Tan said in an interview. "They could pretty much do anything they wanted."
Among the problems associated with buffer overflow vulnerabilities are self-propagating worms of the type seen in the destructive Melissa, I Love You, Code Red and Nimda infestations.
