When administrators and security professionals hear the word rootkit, most think first of a UNIX-based system. Unfortunately, this only leads to a false sense of security for Windows-based systems. The fact is that Windows rootkits do exist, and you need to be able to detect them.

What is a rootkit?
To clarify, a rootkit is not an exploit -- it's the code or program an attacker leaves behind after a successful exploit. The rootkit then allows the hacker to hide his or her activity on a computer, and it permits access to the computer in the future. To accomplish its goal, a rootkit will modify the execution flow of the operating system or manipulate the data set that the operating system relies on.

Windows operating systems support programs or processes running in two different modes: user mode and kernel mode. Traditional Windows rootkits such as SubSeven and NetBus operate in user mode.

Also known as backdoors or Trojans, user-mode rootkits run as a separate application or within an existing application. They have the same level of system privileges as any other application running on the compromised machine. Since these rootkits operate in user mode, applications such as antivirus scanners can detect the rootkit's existence if they have a signature file.

On the other hand, a kernel-mode rootkit is remarkably different -- and much more powerful and elusive. kernel-mode rootkits have total control over the operating system and can corrupt the entire system.

By design, kernel-mode rootkits control the operating system's Application Program Interface (API). The rootkit sits between the operating system and the user programs, choosing what those programs can see and do.

In addition, it uses this position to hide itself from detection. If an application such as an antivirus scanner tries to list the contents of a directory containing the rootkit's files, the rootkit will suppress the filename from the list. It can also hide or control any process on the rooted system.

Rootkit detection
Methods to detect rootkits fall into two categories: Signature-based and heuristic/behaviour-based detection.

• Signature-based detection: As its name implies, this method scans the file system for a sequence of bytes that comprise a "fingerprint" that's unique to a particular rootkit. However, the rootkit's tendency to hide files by interrupting the execution path of the detection software can limit the success of signature-based detection.
• Heuristic/behavioural-based detection: This method works by identifying deviations in normal operating system patterns or behaviours. For example, this method could detect a rootkit by determining that a system with a 200GB hard drive reports 160GB of files and only 15GB of free space available.

Rootkits are hard to detect. But there are programs -- some free and from reputable companies such as F-Secure and Sysinternals -- to help you detect their presence on your systems. Microsoft has even stepped up to the plate with its Malicious Software Removal Tool, designed to detect and remove Windows rootkits.

Final thoughts
If you discover someone has compromised your machine, it's vital that you take the necessary steps to find out if the attacker has installed a rootkit -- and then eliminate the threat. Applying vulnerability patches after someone has installed a rootkit on your machine won't close the security holes that already exist on your network.

Mike Mullins has served as an assistant network administrator and a network security administrator for the US Secret Service and the Defense Information Systems Agency. He is currently the director of operations for the Southern Theater Network Operations and Security Center.