E-mail (and its cousin, instant messaging) so radically reshaped the business world, we do not even think about it anymore. Well, the users do not think about it much anymore.
E-mail remains one of the most important services an IT staff offers, so ensuring its constant availability and security remains a very high priority for most organisations. However, e-mail’s ubiquity makes it difficult to measure. What do our users do with it? How do they measure server reliability and stability? Do our users want us to improve the service, or leave it alone so they can get their jobs done?
While working on a project gathering measurements and metrics data for a client, he posed these questions and a handful more about his e-mail administration staff. His was not an idle interest; I took the hint and started to work immediately on defining the inquiry’s scope and building measurements that would, I hoped, open up the black box.
At first blush, the problem seemed simple. E-mail systems were, after all, the bread and butter of my original consulting practice. Check for uptime, check for bounces, and check problem reports from inside the organisation.
However, when I diagrammed the system on the board, I noticed something. The metrics I usually used told me a great deal about service availability, but nothing at all about utilisation, functionality, or impact. If I gathered the usual data, I would be able to tell my client within a hair’s breath how well the system delivered mail … and not a single thing about what the mail did for the company.
One of the IT articles of faith states that e-mail is important. When it goes down, our user communities haul out the torches and pitchforks. To what extent, though, do they really use it? How many of the features do they use, for what, and to what degree?
These questions forced me to face something I knew, but never really wanted to articulate. E-mail services long ago left the realm of strictly technical measures. My beloved SMTP headers and availability charts barely touched the surface of what we really needed to know.
E-mail was no longer just a tool. Somewhere it metamorphosed into a service, with customers who used that service in ways we could only dimly imagine. In order to measure that, we needed to apply customer service style measurements.
Measurements and metrics
Determining those measurements, and the metric we could apply to them, was an even larger leap for me than the initial conception. Every time I tried to think about it, I kept coming back to charts of uptime and bad header reports. Eventually, though, I sat down in a room with a white board. On one, I listed every report I wanted to write. On the other, I listed the various areas that I knew I would have to measure for a service rather than a tool providing organisation. I kept erasing the first board and filling it, brainstorming until I had reports and metrics by all of the measurement areas in the other one. Eventually I came to the following compromises: availability, budget, and functionality.
In the beginning, the availability metrics looked a lot like my old reports. However, I realised that uptime and header troubles did not tell the whole story. So I built a survey instrument to determine not only what the "technical" uptime was, but also what people thought about it. Did they feel that the system enhanced or delayed their communications? How frequently did they blame the e-mail system for delays? How often did the system, as a whole, fail to provide them with timely service? That last proved particularly interesting: A few months after we implemented the instrument, we discovered, through the questions, a delay in the document management system that our technical reports missed.
Budget is always a concern in every organisation, but I was at a bit of a loss as to how to relate it to e-mail systems. After all, the basic infrastructure budget covered all of the e-mail system’s expenses. So, rather than look for a positive, I broke one of the unspoken rules of leadership metrics: I fixed the budget metric to a negative. If the system required unbudgeted funds for additional expenses, thereby impacting other aspects of the infrastructure budget, it lost points from a baseline. It could make up these points by returning to baseline for a period of time.
Functionality gave me similar fits. What does it mean to measure the functionality of an e-mail system? Eventually I decided that what it really meant was that we needed to be involved with our users' use of the system. We needed to find a way to put our knowledge of how these tools worked at their disposal.
So, I designed a set of metrics that showed how much the e-mail administration team worked with the community. How many usage questions did we answer? How often did the team reach out to contact users to determine what they needed? How many of those contacts resulted in training for the user or in the implementation of a new technical solution?
At first the team strongly disliked the last measurement. It forced them out of their shells, made them make contacts in the user community, and even learn a bit about their own products. It also forced my client to be much more involved with his team’s creative process.
However, after two months they began to realise that their lives were a lot easier when they had friends in the user community. Problems appeared, as they always did. Rumours flew fast and furious, as they did every day. But the e-mail team, in its role as the communication experts, was a part of the process rather than its target.
TechRepublic is the online community and information resource for all IT professionals, from support staff to executives. We offer in-depth technical articles written for IT professionals by IT professionals. In addition to articles on everything from Windows to e-mail to firewalls, we offer IT industry analysis, downloads, management tips, discussion forums, and e-newsletters.
©2004 TechRepublic, Inc.
