Running IPSec to secure your network's communication traffic provides a very strong layer of defence to your network. However, it's important that you test these policies before deploying them and verify that they're running properly. Here are some troubleshooting tips for when you run into trouble.

Securing your organisation's LAN and WAN traffic from prying eyes is an ongoing struggle. In the past, I've written about securing that traffic using IPSec policies. If you followed my recommendations, then good for you!

But what if you've been experiencing problems with your IPSec implementation? We can usually trace most IPSec problems to difficulties during the Internet Key Exchange (IKE) phase of authentication.

Computers go through a process in which they authenticate each other's identity and form a security association. This identity authentication occurs via a preshared key, a digital certificate, or Kerberos (the default for Windows Server 2003).

However, before you begin troubleshooting the authentication process, let's start at the beginning. First, make sure IPSec is running.

The easiest way to determine whether IPSec is running on a computer is to fire up Network Monitor, capture a few packets, and see which protocols are running across your Ethernet interface. If the machine has IPSec configured, you should see only Encapsulating Security Payload (ESP) and Internet Control Message Protocol (ICMP) protocols in your capture.

Remember that Windows tends to be very chatty, with a lot of Lightweight Directory Access Protocol (LDAP) and Server Message Block (SMB) traffic. If you see these two protocols listed in the capture, IPSec probably isn't running.

To restart IPSec, you could reboot the computer. But if you've recently made some significant policy changes and can't afford to reboot the machine, you can stop and restart IPSec via the command line. Simply issue the following commands at the command prompt:

net stop policyagent 

net start policyagent

Your IPSec policy should be working, but if you continue to experience problems, you need to keep troubleshooting. Your next step is to look at the authentication method and the policies themselves.

Begin by verifying which policy is operating on the machine, and determine whether it has a compatible method for authentication -- one policy can't use Kerberos if the other uses a preshared key. You need to check which policy is active and find out which authentication method the policy is using.

To do so, run the Microsoft Management Console (MMC) by going to Start | Run, entering mmc, and clicking OK. Add the IP Security Monitor snap-in by going to File | Add/Remove Snap-in, clicking Add, and selecting its name. This will show you which policy is active as well as the authentication method.

Depending on what you find, you might need to just apply the policy or modify the authentication method. Let's look at some of the possibilities:

• If your policies use a preshared key, make sure the keys are the same. Type the key in Notepad, and cut and paste it into the policy.
• If your policies use digital certificates, verify that you've installed the certificate and it's still valid. IPSec policies expire every two years, and they do not automatically renew.
• If your policies use Kerberos, chances are good that you're actually having problems with Active Directory (AD), which you need to troubleshoot first. Head on over to the Windows Server 2003 Active Directory Technology Center, and fix your AD problems.

At this point, IPSec should be working. If it isn't, you need to disable IPSec, take your implementation back to the lab, and start from scratch.

On of the most common mistakes people make during IPSec policy implementation is setting all of the policies to Client (Respond Only), the default setting on the IPSec policy template. If you've set all of your machines to Client (Respond Only), no machine will ever request or require IPSec -- and all of your network traffic will remain unencrypted. Change one of the policies to Request or Require, and run Group Policy Update to activate the policy change.

Final thoughts
Running IPSec to secure your network's communication traffic provides a very strong layer of defence to your network. Test your policies before you deploy them to the production servers, and verify that they're running properly.

Mike Mullins has served as an assistant network administrator and a network security administrator for the US Secret Service and the Defense Information Systems Agency. He is currently the director of operations for the Southern Theater Network Operations and Security Center.

TechRepublic is the online community and information resource for all IT professionals, from support staff to executives. We offer in-depth technical articles written for IT professionals by IT professionals. In addition to articles on everything from Windows to e-mail to firewalls, we offer IT industry analysis, downloads, management tips, discussion forums, and e-newsletters.