Catching malicious hackers isn't impossible. With the right tools, you can gather important information to help stop hack attacks. How can you use common UNIX tools to hunt down network attackers?

You have determined that your network has been breached. There are two standard approaches on what to do next:

• Close the holes as fast as possible and put in safeguards from future attacks.
• Identify the perpetrator and prepare for prosecution.

• A list of open ports and services
• Aberrant packet behavior
• Accurate dates, timestamps, and images of evidence
• Suspicious IP addresses
• Geographic locations of suspect IP addresses

Various software tools and UNIX commands can help you gather this information. Some of the UNIX tools that come bundled with most UNIX operating systems or that are freely available on the Internet and are worth familiarising yourself with include:

• netstat
  This tool prints a listing of network connections, routing tables, interface statistics, masquerade connections, and multicast memberships.
• tcpdump
  This tool prints out the headers of packets on a network interface that match a Boolean expression.
• dig
  This tool sends domain name query packets to name servers.
• traceroute
  This tool prints the route that packets take to network hosts.
• find
  This tool searches for files in a directory hierarchy.
• dd
  This tool converts and copies a file.
• grep, egrep, and awk
  These tools are used to process text.

Use netstat to determine open ports and services

Netstat is the tool to use to determine what ports and services are currently open. When you execute the command netstat -an, you'll see a listing of all the connections along with their listening ports and the network addresses associated with these ports. The output will look something like this:
TCP 128.88.41.2:1025  140.216.41.2:80  CLOSE_WAIT
TCP 128.88.41.2:2180  140.216.41.2:80  CLOSE_WAIT
TCP 128.88.41.2:1188  140.216.41.2:80  CLOSE_WAIT

Look for patterns such as similar source ports used to connect to different sockets. (A socket is an IP address and port together e.g., 206.208.163.15:80.) In the above example, three connections (now closed) were used to connect to the Web server port, all from different source ports. If you discover a server on a particular port that is not normally in use, it's possible that a hacker (with root level access) installed it for malicious purposes.

Use tcpdump to look for aberrant packet behaviour

Use the tcpdump tool to trace packets and print out packet header information. When you execute the command tcpdump > outputfile on a shared network that is not switched and dumping to an output file, you'll see a timestamp, a source socket, a destination socket, a TCP flag, a sequence number and offset, and a maximum segment size. The output will look something like Listing A.

One thing that should tip you off to a problem in this example is that 295.128.16.0 is not a valid Internet address, because 295 is not in the range of acceptable octets. Another indicator that something could be awry is that port 4950 is a known hacker port for ICQ exploits or attempts to log into port 23 (telnet) from clients that shouldn’t need to use telnet. The TCP flag indicated is S, which, if there were a lot of these flags in sequence, could indicate a SYN flood attack. You'll want to look at the packet sizes and see if they look typical for the traffic on your network. In short, you should look for aberrant behavior.

Keep in mind that your tcpdump output is valid only for the time during which you executed the command, and any aberrant activity that has already ended will not be captured. For that reason, tcpdump is particularly useful when you have observed suspicious behavior on a live system.

Use dig to uncover suspicious IP addresses

The dig utility, a replacement for the older nslookup, is a good tool to use to look up suspicious IP addresses discovered through netstat, tcpdump, or other commands. To use this command, insert the IP address or hostname after the dig command, like so: dig 140.216.41.2.

The dig tool takes the IP address and returns the hostname, or, if you put the hostname after the dig command, it returns the IP address. If you need to map an IP address to find the fully qualified domain name, you can use the dig command to find out who owns it. You can then go to the American Registry for Internet Numbers (ARIN) or the WHOIS server and find contact, registration, and ownership information regarding the domain in question.

Use traceroute to find geographic physical locations

The traceroute tool can help you figure out the route a packet follows to get from one place to another. Most administrators use traceroute to find out the physical geographic location of a system.

If you decide to involve law enforcement, you’ll likely need to know the actual location where crime originated. If the hacker isn't using relaying, false identities, or anonymisers, you can use the traceroute command in conjunction with the information garnered from the WHOIS database to find his or her physical location.

You run traceroute by listing the hostname, preferably the fully qualified domain name, after the command, like this: traceroute company.com.

The output will list the name and IP address of the destination, and will list all hops along the way, similar to the output shown in Listing B. The output is in the form hostname |IP address | probe1 | probe2 | probe3. Each probe is a timed attempt to get the proper response. Each probe also returns the time it took to get that response.

The traceroute tool is most useful if you execute it while suspicious activity is in progress, or shortly thereafter. The goal is to find the route through which the traffic between your compromised host and the suspect host traveled. The route determined from your traceroute could be different from the route an intruder used, depending on Internet traffic conditions, but usually it will be accurate. If you can determine the ISP of the suspected host, the ISP, with proper encouragement from law enforcement, can help identify the owner.

Note, however, that if there are routers or firewalls along the route that block time to live (TTL) packets, or ICMP message types, traceroute will not be able to report any findings from these nodes.

Find

There are many ways in which you can use find in incident-response forensics. For example, if you suspect that an intruder broke in by exploiting SUID or SGID files, you can use find to locate all these files (as shown in Listing C) by looking for any log file that references the SUID (2000) or SGID (4000) permission bits.

An even better use is to find all these files in advance and dump the output into a log file. You can then check this list by running new find commands from time to time and using the diff command to compare the two log files. By doing this, you’ll see if anyone, possibly an intruder, installed new SUID or SGID files. You would use diff to examine two log files by issuing this command: diff suid_gid_logfile1 suid_gidlogfile2.

Typically, the SUID and SGID file list shouldn’t change. If you see new additions, examine any new files from the SUID and SGID list and determine whether or not they should be there. You can also use find to look for world-writeable files, because these files are also often exploited. To look for all the world-writeable files, run find as shown in Listing D.

This list shouldn't change much after your system is in production, so I recommend saving this log file and using diff to determine whether any changes occur. Running diff on the log file outputs (e.g., diff worldwrite_logfile1 worldwrite_logfile2) will tell you if new, world-writeable log files have been installed after the system has gone into production.

You can also build these commands into shell scripts and set up cron to run them on a nightly basis to check to see if new SUID, SGID, or world-writeable files have mysteriously appeared.

Use dd to preserve the evidence

The dd utility, a UNIX command for dumping data, has been a part of almost every UNIX-based operating system for as long as I can remember. Use the dd utility when you want to preserve evidence and create a forensic disk image, without changing timestamps or anything else. First, figure out where you're going to dump the image; a writeable CD or another disk are the most likely places.

Various syntaxes for dd exist, but a typical command-line version of it looks like the example in Listing E.

The above command would tell your system to copy crime scene partition /dev/hd1 to a disk image called evidence.img. In the command, the bs switch specifies the block size of the files being transferred, and the count tells the system how many blocks to transfer. If you're transferring the evidence image to a CD (CDs can hold 660 MB), set the count to something less than 660, to be sure you don't accidentally overwrite information to the CD. You can then take the image and burn it on the CD. CDs might be preferable over hard drives, since they’re easier to send in the mail and not susceptible to magnetic interference.

If you have a 6-GB drive, you'll need to run through a series of dd commands that chops the evidence into a series of images that will fit on a set of CDs. If you're counting by 600-MB blocks, tell dd to skip the 600 block you already copied, and copy the next bunch of 600 blocks. To do that, use the skip switch and increment it by 600 each time, as shown in Listing F.

Grep, egrep, awk find 0 UID and GID accounts other than root

You'll want to look for root accounts that might not look like root accounts. Often, intruders set up new accounts with root access that aren't associated with the root name. To find these accounts, use the command shown in Listing G.

From the output of the command in Listing F, look for non-root accounts that have 0 UID privileges. These accounts will give you clues about where some of the damages might have been inflicted. Once you locate these 0 UID and 0 GID accounts, you can poke around in their directories for hacker rootkits and other malicious tools.

What next?

Figuring out who cracked your network is only part of the battle. After you have gathered all the information you can find, you must report the activity to the right institution, such as the FedCIRC, a government-run organisation dedicated to computer security-related issues affecting the civilian agencies and departments of the federal government. Other groups include the Department of Energy’s Computer Incident Advisory Capability (CIAC) and the Federal Bureau Of Investigation National Computer Crime Squad.

When you find you have been cracked and you’re able to gather the necessary information, report the criminal. The more of these cracks that are reported, the better the chances that something can be done to stop them.

Editorial disclaimer: The authors and editors have taken care in preparation of the content contained herein but make no expressed or implied warranty of any kind and assume no responsibility for errors or omissions. No liability is assumed for any damages. Always have a verified backup before making any changes.

TechRepublic is the online community and information resource for all IT professionals, from support staff to executives. We offer in-depth technical articles written for IT professionals by IT professionals. In addition to articles on everything from Windows to e-mail to fire walls, we offer IT industry analysis, downloads, management tips, discussion forums, and e-newsletters.

©2001 TechRepublic, Inc.

Copyright © 2009 CBS Interactive, a CBS Company. All Rights Reserved.
ZDNET is a registered service mark of CBS Interactive. ZDNET Logo is a service mark of CBS Interactive.