In this column, I want to talk about metrics some more, because the sad fact is that using a wrong or misleading metric can be worse than not having any metrics at all. I'll provide some examples of how IT managers can end up measuring the wrong thing, and also give some tips on how to avoid this and other mistakes.
Another look at the art and science of measurement
This isn't the first time I've talked about metrics. In a previous column, I pointed out that when managers decide to measure something (it doesn't matter what), that very decision changes employee performance. In the old Soviet Union, I noted, nail factory managers used to be rated on the weight of the nails produced each year. The result was that factories made a lot of very big nails and few small ones. No one could hang a picture on a wall, because there was no incentive to produce small nails. Then the powers-that-be changed the metric to the number of nails factories produced annually. The result was that factories produced way too many tiny nails, and not enough big ones.
Let's look at another example. Suppose you're running the corporate help desk. To help with the workload of your support techs, you install an automated ticketing system, which includes elaborate FAQs on how to solve common problems before opening a trouble ticket.
You're feeling pretty good about your new system, until your boss calls and quizzes you on why the total number of tickets cleared by your group has declined. More importantly in your boss's mind, the average ticket resolution time just went through the roof. To your boss, it looks like your people are taking longer to do less work.
This points out three possible problems with inappropriate metrics:
- When your organisation changes, so should your metrics
In our example, it makes perfect sense for the new system to reduce the total number of tickets served. After all, the extensive FAQ files are helping the end users to solve many of their own problems without having to open a ticket for a support tech. At that same time, since your technicians aren't having to clear tickets for obvious problems (-Next time, make sure the PC's power strip is turned to the ON setting before calling us..."), a higher percentage of the tickets they encounter are for legitimate support problems. Therefore, it's entirely reasonable to expect that their average time per ticket will go up. This isn't because they are slacking off, but rather because they are dealing with more true problems and less entrants for the TechRepublic -Stupid User Awards." -
People outside your department often misinterpret metrics
The misinterpretation of metrics can cause problems, particularly in situations where non-technical managers supervise or oversee a technology department. In this case, a brief conversation with your boss should clear up the problem, but not all situations are so easy to explain. Indeed, you run the risk that your boss will misinterpret your explanation as an attempt to gloss over poor performance.Even worse, you might have to do your explaining to more than just your supervisor. This is the third potential problem with inappropriate metrics.
- Metrics take on a life of their own
You'd be surprised at how often statistics and measurements designed for use within a specific departments end up being distributed to the entire organisation. Of course, it can be difficult to interpret even meaningful metrics. Imagine the difficulty of having to educate people in the organization who don't understand what your group does. This difficulty is multiplied when they are looking at an inappropriate or outdated metric.
Good metrics, not no metrics Please don't misunderstand me. I'm not arguing against the use of metrics. Nor am I saying that you shouldn't publish those metrics when appropriateâ€"for example, to your supervisor and your clients (internal and external, as the case may be).
Instead, I'm asking that you do three things:
- Make sure when you set up a metric that it accurately reflects the performance of what it measures, and that it can be easily understood.
- Periodically review your existing metrics to make sure they are still accurate and relevant.
- Keep a handle on who sees each metric, so that you explain any changes to the numbers, and reduce confusion.
Happy measuring!
TechRepublic is the online community and information resource for all IT professionals, from support staff to executives. We offer in-depth technical articles written for IT professionals by IT professionals. In addition to articles on everything from Windows to e-mail to fire walls, we offer IT industry analysis, downloads, management tips, discussion forums, and e-newsletters.
©2001 TechRepublic, Inc.
