When was the last time you sat in your computer room and just looked around? Have you ever found yourself staring at all the pretty blinking green and yellow lights?
Did you ever try to hum along with the buzzing of cooling fans or the Liebert compressor? While these questions may seem frivolous, spending part of your day just observing things working breeds a familiarity you can use for troubleshooting. While network monitoring tools are important, using your eyes and ears is sometimes the best way to uncover problems. When you cultivate an innate understanding of your systems, it is much simpler to spot trouble.
A well-tuned network is a wonderful thing. All the packets are flowing freely and none of the users are complaining about performance. Investigating the behavior of everything in your data center when things are quiet is the best time to make mental (and written!) notes of how normal performance "looks and feels". The various status lights all tell a story. There should be some pattern that signifies all is well. Being able to recognize when the pattern is off can often help determine what the problem is and get you one step closer to fixing it.
You should gain a familiarity with all your hubs, routers and communications equipment. Status lights may often indicate collisions or other error conditions. Always be alert to the obvious signs that indicate trouble. Traffic indicators that glow steadily on one port while others only blink occasionally. Ports with cables that show no activity at all. Knowing where the warning lights are can make at a glance scans for problems fast.
High-end file servers are sometimes equipped with small LED readouts that indicate internal temperature, memory errors or other warning messages. Make sure you know how to manipulate the menus to get the vital information you need before communicating with technical support. Be sure to check around the back of the rack too. Network cards, power supplies and other components usually have lights that report on their status. Drive arrays can report when a disk has failed and needs to be replaced.
Listen carefully for changes in the ambient noise in your data center. In a loud computer room it can be difficult to hear. One of my managers used to keep a stethoscope in his toolkit to check hard drives in massive drive cabinets. By holding the diaphragm of the instrument on each disk, he could verify if a drive was indeed spinning. Though I never tried this technique myself, my ears have served me well over the years--sound is one of the best clues for diagnosing analog modem connection problems. Several years ago when modem pools were all the rage I used to be able to tell the negotiated baud rate just from the tones. More recently I identified a cooling fan that was whining and barely moving air.
I just love low-tech tools. One of the best is simply looking around. Now you have another excuse to escape to the computer room for a break!
Neil Plotnick is the author of the "IT Professional's Guide to Managing Systems, Vendors and End-Users." He welcomes your questions and comments at Neil@NeilPlotnick.com.
