The Four Fallacies of IT Metrics 223
snydeq writes "Advice Line's Bob Lewis discusses an all-too-familiar IT mistake: the use of incidents resolved per analyst per week as a metric for assessing help-desk performance. 'If you managed the help desk in question or worked on it as an analyst, would you resist the temptation to ask every friend you had in the business to call in on a regular basis with easy-to-fix problems? Maybe you would. I'm guessing that if you resisted the temptation, not only would you be the exception, but you'd be the exception most likely to be included in the next round of layoffs,' Lewis writes. 'The fact of the matter is it's a lot easier to get metrics wrong than right, and the damage done from getting them wrong usually exceeds the potential benefit from getting them right.' In other words, when it comes to IT metrics, you get what you measure — that's the risk you take."
Business planning (Score:5, Insightful)
Comment removed (Score:5, Insightful)
Re: (Score:2)
Re:Business planning (Score:5, Insightful)
Support doesn't just mean 'fixing bugs'. It also means 'helping you set things up right', 'helping you optimize your configuration', 'helping you figure out what tool you need for the job at hand', and so on.
Selling support does not require that the underlying product be broken.
Re:Business planning (Score:5, Insightful)
Any company that is relying on support to keep their production up and running deserves to be down and losing money. Just because some analyst uses big words in a conference room doesn't mean you don't need good people on staff that know their shit!
I hate IT and I am so sick of it. Here's an ask slashdot! What the hell can I do now? I hate this place! All I know is *NIX and enterprise storage. Load balancing and JBOSS. Virtualization and a decent amount of PERL and Assembly. Maybe flipping burgers ain't so bad after all...
Re:Business planning (Score:5, Insightful)
Flipping burgers ain't bad if you own the restaurant.
Re: (Score:3)
Restaurant managers are notable as one of few groups of people who work longer hours than IT.
Re:Business planning (Score:5, Interesting)
The major fallacy many big companies fall into is that some of these systems have been running flawlessly for years, because they hired a competent IT staff. They look at the price of those paychecks and shiver. Why are we paying so many high priced engineers when we've never had a problem, they think.
So they reduce staff and start to rely on support contracts instead of on-site gurus. The gurus are still there to solve any oh-shit moments. But that back investment in good engineers has produced a stable infrastructure that runs with few problems for years. So they reduce staff more, pay for more support contracts, and eventually the system critical mass is greater than the engineers who can support it. It's no problem until it's a problem.
Eventually something minor goes wrong, but nobody notices or if they do it's not really their field of expertise so they don't understand it's minor now but could escalate. When it does, something else goes wrong, and a cascade effect takes out more and more systems. With a full staff, you have enough guys that when the critical mass is reached, they can start defensive measures and get things back in working order in no time. With support staff only, things are going wrong faster than they can deal with it.
"Call on our support contracts," shout the bosses! So now your on-site staff are all on hold instead of troubleshooting. When they get through to someone, they have to spend the first hour or two describing their infrastructure to the technician on the other end, who starts making random suggestions that maybe help, but probably don't.
My anecdote on this front is a company I used to work for. It's a long read, but demonstrates the failures at several levels which is the direct result of this kind of thinking. The Oracle transaction log disk was getting full. Some warnings came in, but disks running low on space was an every day occurrence, we'll send an email to the person on record as being responsible for those servers, and troubleshoot why the "Executive Dashboard" is responding a bit slow today (it's for the execs, it's automatically high priority). Except that person is currently aboard an airplane on his way to help reduce staff in east Asia, he'll be incommunicado for the next 19 hours or more.
It seems like an innocent enough problem, it's just a log disk, the worst thing that could happen is we lose some logs, right? Whoops, transaction logs are pretty important for Oracle. The fact that the disk is filling up at all is itself an indicator that something bigger is wrong; this shouldn't happen. But critically once the disk does fill up, Oracle will enter read-only mode. Or it should. This time it doesn't, it shuts down. BOOM, offline. So down goes SAP. With SAP down, our entire business is offline. We can't take orders, we can't ship orders, we can't pay bills, we can't pay paychecks, the hourly workers whose shift is starting can't even clock in. Some buildings with tighter badge access can't even be entered unless someone inside opens an emergency door to let someone in.
Once the transaction log disk was full, Oracle will no longer start up, it needs some space on the log disk to log startup-related transactions. Two hours on hold with Oracle Gold Pressed Latinum level support they finally get an engineer. Wow, this is something he's never seen before, Oracle should have gone into read-only mode before this happened! The only solution anyone can seem to think of is to get some bigger disks for the transaction logs, clone the data over to these new disks and give the startup another go. We have hot spares on a shelf, but nobody knows this. Finding disks requires a different support contract, they can have disks out to us tomorrow. Yeah, that's not going to cut it. Someone literally drives out to a distribution warehouse. Two more hours down (they actually send two different guys in different cars with instructions to take different routes in case one runs into traffic or gets in an acciden
Re:Business planning (Score:4, Insightful)
As someone that works in support it would be nice if customers understood how to use their support system.
All the screaming and yelling that it's broke will not tell me what 'it' is or what is 'broken' about it.
Re:Business planning (Score:5, Funny)
- "How can you tell? Is there an axe sticking out of it?"
and
- "How can you tell? Is it on fire?"
One day I had this young kid came up to me saying, "My computer is broken." so of course I respond, "How can you tell? Is it on fire?"
He looked a bit embarrassed and said, "Well it was smoking and made a buzzing sound but it has stopped now."
His one day old computer's power supply had burned up in a spectacular fashion.
(Still waiting to see an axe...)
Re:Business planning (Score:4, Insightful)
-what do you see
-what did you do
-what do you expect to see
Re:Business planning (Score:5, Insightful)
-what did you do
Of course, that one has a universal answer, "nothing, it just stopped working!"
Re: (Score:2)
My best (worst?) tech support call was an "Our printer's broken." How can you tell? "There's a bullet hole in it."
Ahhh, right, we'll get that replaced then. And I will never work at a bank branch.
Re:Business planning (Score:5, Insightful)
1) Did it ever work?
2) When did it quit?
3) What have you done to it lately?
Pretty much the foundation of my IT Career right there.
Re:Business planning (Score:5, Interesting)
Support... ... also means 'helping you set things up right', 'helping you optimize your configuration', 'helping you figure out what tool you need for the job at hand', and so on.
Worked at a support center... I was a "talk to them until they understand" guy, playing the long game... I figured while it might not take every time, if I got people to understand, they could get back to work and not break things for just a little bit longer. You know, it costs two people money if they have to talk to me while I help them.
One of my coworkers got huge amounts of management praise for processing lots and lots of cases... My management was too dumb to run numbers on how many callbacks he had, that the rest of us were fixing...
Yeah, sure I was spending too much time with each person, but half of my time was fixing this jerk's mistakes. There's probably some of that at every support center. It takes 10 minutes to fix a problem, but 5 minutes to get them to go away. You can look very busy by making them go away, if management isn't clever enough.
I'm rather happy with my new position... I get to review other people. And I do it fairly.
Re: (Score:2)
Don't you normally include the phrase "nobody can deny"?
So insurance companies make money on the promise that your house will burn down, and that includes saying that houses are flammable?
Re: (Score:2)
So insurance companies make money on the promise that your house will burn down, and that includes saying that houses are flammable?
Well yes - insurance companies tell you all the time that houses are flammable. They even list everything that you will lose if your "so flammable, it could be considered burning already" house do actually burn in their advertisement. They also tell you that you will lose your phone, have your car stolen, crashed and destroyed. They also tell you have a good chance to die unexpectedly and leave your family in misery if you don't get their life insurance policy.
Any metric can be gamed (Score:5, Insightful)
Losers realize this simple fact, instantly think of several ways to game the metric, then don't do it figuring that "obviously" the decisionmakers realize the metric is horribly broken. Then they get laid off. Winners spend hours, days, or weeks coming up with one way to game the metric, pat themselves on the back for being so clever, and do it. Then they get promoted, eventually to a position where they come up with metrics of their own.
Re:Any metric can be gamed (Score:5, Insightful)
Unfortunately, this is true. Evil will always triumph because good is dumb.
Re: (Score:3, Insightful)
Unfortunately, this is true. Evil will always triumph because good is dumb.
Evil will never truly triumph over Good because if it does it will have nothing left to eat next season.
Re:Any metric can be gamed (Score:4, Insightful)
Unfortunately, this is true. Evil will always triumph because good is dumb.
Evil will never truly triumph over Good because if it does it will have nothing left to eat next season.
No, Least Evil becomes next season's Good, and then the spiral continues.
Re:Any metric can be gamed (Score:4, Insightful)
Evil will never truly triumph over Good because if it does it will have nothing left to eat next season.
Unfortunately, this isn't very true. Even if you know the job is a ripoff where someone else is taking 99% of the profits, in the choice between a job and no job people will work under sweatshop conditions to put food on the table. That is something that's happening today. To take an even more extreme historic example, the slave trade lasted for centuries. You are a slave, your children will be slaves, your grandchildren and great-grandchildren will be slaves. If anyone did the starving, it wasn't the slave owners. Yes, the circle of slavery was eventually broken, but you'd need a really, really long perspective to say that Good will triumph some day. I don't see that the future holds any guarantees either, there's plenty sci-fi futures where you have robotic or unmanned weapons system that are utterly loyal. They will never desert, never refuse to obey orders, never let their feelings get in the way of killing unarmed civilians and imposing a reign of terror. Technology is no guarantee you will have more freedom, even though some technologies have obviously helped.
Almost. (Score:5, Insightful)
Winners understand that tech support is a stepping stone and treat it as such. Which means that they move up as soon as possible.
Tech support managers are under pressure to keep their costs down. So unless you're okay with working for less money than the others there (but still solving as many problems / answering as many calls) you will be replaced with a new, cheaper person as soon as they can find one.
The metrics are just there to justify replacing you.
Re:Almost. (Score:5, Insightful)
It really depends on what you are supporting and your skill level. I'm willing to pay more for someone who is a great problem solver; someone who can connect to a client's environment and do whatever it takes to solve a problem - take risks, explore, and find solutions. I'm certainly not going to replace that person with someone cheaper who can't resolve the problems we face.
Re:Any metric can be gamed (Score:5, Insightful)
Yeah, this is pretty much the problem. Performance evaluation should really be done by crazy, high-tech methods such as you and your peers and manager sitting down and discussing what you've achieved, but that kind of thing is way too hard to stick into an Excel macro, after all...
Another classic example: call centres which measure 'performance' mainly by the average call time metric. Which gives tech support workers all the incentive in the world to give out any piece of bogus advice that'll get the customer to hang up as fast as possible. Or just hang up on them, if the phone system isn't sophisticated enough to detect it.
Re:Any metric can be gamed (Score:5, Insightful)
Any idiot can see that the misguided sweatshop-style metrics cause the following problems:
Cherry-picking - Techs choosing and even stashing away (!) the returns with the easiest and quickest problems to fix. It matters not that your expensive gadget has been sitting there for a month, there are numbers to be made and we'll get to yours when we want to regardless of the order they came in.
Racing - When there are no "easy" ones to be cherry-picked, then the techs will race to fix your item. They will ignore problems and cut corners on others. Stripped screw hole? Super-glue the screw in. Low output? Game the settings so the tests will pass. Part shortage? Cannibalize and rob Peter to pay Paul in a hardware-sort of Ponzi-scheme.
Status Quo and mediocrity - The top performers will become accustomed to the attaboys and will continue to produce slipshod repairs, even if there is a slowdown in work when they can do their job right. Meanwhile, the low performers will become used to it and feel no need to better their work.
My idiot boss in the company I'm in now considered it and was shot down by every tech. In this company, due to the variety of products, one person could make tens of thousands of dollars with 1-2 days work while another tech working on a different product will have to spend more labor and overhead juggling external vendors and all the headaches it involves only to make a couple thousand dollars. Yeah.
Fortunately, the consultants we brought in are smart. They listed generic milestones and a cheeky "100%" as the goal with the smiling disclaimer that it will probably never happen.
Comment removed (Score:5, Interesting)
You get what you reward (Score:5, Insightful)
Typo: it does *not* matter if management says ... (Score:2)
Re:You get what you reward (Score:5, Insightful)
Talented or not, those people are not ethical.
On the third hand, you've got those of us who put our effort into work instead, bill accurately, and don't dick about with meaningless excel spreadsheet metrics. We do our work, ignoring the rest, and end up getting a lot more done than ticket munching assholes.
Then we quit for a substantially higher paid job when the idiots start thinking you're doing nothing. Maybe we steal our previous boss's clients (because they like our work). Maybe we start our own company.
Metrics are what kill small IT companies. IT (and programming) are some of the least trackable/accountable things you can think of - it's how people get by doing absolutely nothing for years and years while still making close to $100k a year: their lack of work and fuck ups fall in someone else's lap, while they spend their time attempting to look important and knowledgeable (while they are neither).
The problem is that the people doing these things - metrics - don't know better. That's what they were taught in management school, and that's what so-called industry knowledge says you should do. (It works in India, right? So it's gotta work here. Except, it doesn't really work in India, and that's why their product is shit.)
Ultimately, it will be their end. No company can thrive without actually paying attention to what the smallest unit within their organization is actually producing and accomplishing, to some level. Metrics just attempt to abstract actual work into a number discernible by those who don't understand anything but numbers. It doesn't end well.
Re:Any metric can be gamed (Score:5, Insightful)
Losers realize this simple fact, instantly think of several ways to game the metric, then don't do it figuring that "obviously" the decisionmakers realize the metric is horribly broken. Then they get laid off. Winners spend hours, days, or weeks coming up with one way to game the metric, pat themselves on the back for being so clever, and do it. Then they get promoted, eventually to a position where they come up with metrics of their own.
It's not just IT. Our entire society has converted over to metrics. An easy example comes to mind: the stock market versus a company's quarterly performance. Another set of particularly nasty examples is found in our justice system: police officers evaluated by their number of citations, prosecutors by their number of convictions, prisons by their dollars per inmate per day.
I get the financial impetus to switch to metrics. Where it used to be one skilled manager overseeing per 5-7 employees, it can now be one schmuck manager with an Excel spreadsheet overseeing 30 employees.
I even get the psychological impetus. Numbers give us that all-important feeling of certainty, and at low cost too... while the traditional alternative requires legwork, mindwork, judgment, contemplation, and mistakes.
But it's wrecking our society.
Re: (Score:3)
Re: (Score:3)
All told, I think that's a pretty fair metric to hold me up to.
Re: (Score:3)
It's wrecking more than our society.
Where it used to be one skilled manager overseeing per 5-7 employees, it can now be one schmuck manager with an Excel spreadsheet overseeing 30 employees.
This is short-sighted thinking. Sure, they make a lot in 6 months, a year, maybe two years. But then something goes horribly wrong (in their eyes), and shit starts to fall apart.
I suppose it's great if you plan to pump and dump, as the inclination is in this culture... a bit of a catch 22, I suppose.
God save the children, because they're not going to inherit much of anything.
Re: (Score:2)
Re: (Score:2)
Read what you quoted.
Maybe you have not converted to metrics. Your society has, and it's judging you. Take a look at the writing on the wall...
And this is the writing that was written, MENE, MENE, TEKEL, UPHARSIN.
This is the interpretation of the thing:
Why do ACs insist on remaining anonymous while coming up with gems like this? This is good writing, funny and insightful. But anonymous.
Who does this? (Score:3)
So fuck em and dump em.
Anyone worth their salt will look at downtime, stability, and resolutions before they look at resolution time.
Re:Who does this? (Score:5, Insightful)
Probably the same people who consider number lines of code written per hour as a good metric to evaluate their employees productivity.
Re: (Score:2)
That's almost asking for a shitload of dead code to be pasted into a routine that just adds two numbers.
The use of metrics to measure peoples performance is usually implemented incorrectly. You want to measure my performance? Okay, I did 1 thing all day. It was rebuilding a $50,000 test stand. Yesterday, I built 3 custom test fixtures for the production line, and worked on maintaining some of my equipment.
The number of events being measured is correct, but what about the value to the company of each event?
Bill Atkinson got there before you... (Score:3)
At least give credit where it belongs for the idea that that is a bad metric:
-2000 lines of code
http://www.folklore.org/StoryView.py?story=Negative_2000_Lines_Of_Code.txt [folklore.org]
-- Terry
Re: (Score:2)
*gasp* What're you talking about? Programmers with a high LOCPH (lines of code per hour) are rock stars. MOTHERFUCKING ROCK STARS, you hear? It doesn't matter if it's good code, it doesn't even matter if it's maintainable. They're putting out the features and making the big dollars.
Oh, I see what you did there.
Re:Who does this? (Score:4, Interesting)
Anyone worth their salt will look at downtime, stability, and resolutions before they look at resolution time.
Ahh so number of resolutions are better than resolution time are they?
My recent experience with our IT call centre at work (Company is top 10 in the Fortune 500) I needed access to a shared drive. We've been asked to email the call center with specific detailed requests if we know what the exact problem is to save the phone support for problem identification sessions.
Anyway my email went along the lines of: "I have recently for some unknown reason lost access to network share with no explanation. Access to the drive is necessary for "
I got not 1 but 3 replies from the service centre:
Email 1 (autogenerated): Your Incident INC xxxxxxx1 has been raised for access to a network share.
Email 2 (typed): Dear User, Requests for access to network shares need to go through .
Email 3 (autogenerated): Your Incident INC xxxxxxx1 has been closed with successful resolution.
Errr no it hasn't. They generated a case number and closed it successfully but I still have no access to the network folder. Anyway off I go to the other system and request access through it. I get 3 emails again:
Email 1 (autogenerated): Your request has been received and has been forwarded to the service centre.
Wait for real? The same schmucks who I just requested this through and been sent away get the request?
Email 2 (autogenerated): Incident INC xxxxxxx2 has been raised for access to a network share.
Email 3 (autogenerated): Incident INC xxxxxxx2 has been closed with successful resolution, please wait up to 1 hour for permissions to propagate.
Well there you go two incidents were raised and closed, and in neither case was the end user asked if it actually worked. I wonder what happens if they gave me read only access instead of read-write as per my original email, given that the system we were supposed to use didn't specify. I guess that would raised a 3rd case.
Re: (Score:3)
In most medium+ organizations the hel dek woul have nothing to do with stability or downtime. Resolutions would not be any better than resolution time. First call resolution are tracked a lot. They are also subject to eBay manipulation. There are not any help desk metrics not subject to what is discussed in the article.
The view is cynical. Unfortunately, it is also accurate.
Dilbert Minivan (Score:5, Funny)
This problem was aptly portrayed in the classic dilbert comic strip in 1995.
I'm going to code myself a minivan. [dilbert.com]
Re: (Score:2)
You have an amazing memory. You've managed to remember every single Dilbert comic strip since 1995 (or beyond). You then picked the one most relevant to this article. You should be the one getting the mini-van!
Re: (Score:3)
I'd pity him instead. Being able to remember something like that means it's probably applicable to where he works. I'll bet it's pinned to his wall.
ain't pretty. (Score:5, Insightful)
Such metrisc also disincentivize people taking proactive steps to reduce the number of incoming tickets (i.e. making the system/environment more robust or your users more educated), and disincentivizes managers for so doing by reducing the number of people needed to service incoming tickets (thus reducing the size of the empire and the pay grade of the manager).
I've seen both "disincentives" in action. It ain't pretty.
Re: (Score:3)
Such metrisc also disincentivize people taking proactive steps to reduce the number of incoming tickets (i.e. making the system/environment more robust or your users more educated)
Tell me about it. The job I'm in is support/maintenance/administration/installation etc. for companies too small to have their own IT depts. I'm constantly saying "we should do this because it will prevent xxxxx in 6-12 months time". I'm sometimes listened to. "Has the customer logged a problem? No? Well we're flat out, there are more important things that need to be done". Usually true, but if we always did the things I recommend we'd have far fewer sudden customer crises demanding immediate attention. Of
The problem with metrics is (Score:2)
Dunningâ"Kruger effect (Score:5, Insightful)
That's an example of the Dunning-Kruger effect.
http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect [wikipedia.org]
Anyone can design a metric that they themselves cannot figure out how to game.
Re: (Score:3)
Anyone can invent a security system that he himself cannot break.
Re: (Score:2)
This makes me sad (Score:5, Interesting)
Re:This makes me sad (Score:5, Informative)
Sounds like academia, actually. It's all about impact factor, citation count, and grant dollars these days...
Re:This makes me sad (Score:5, Interesting)
Re:This makes me sad (Score:4, Interesting)
I have no large-scale study, but my anecdotal information on those kinds of schemes is that they can have the opposite problem in large companies: rather than promoting uncooperative individual greed, they instead promote a sort of feeling that, "well, as one peon I can't possibly move the needle on this gigantic corporation". People then get demotivated when their bonus depends on things totally divorced from what they actually do, e.g. you do a great job in IT this year, but your bonus goes down because you work for an oil company and the price of oil went down, something you don't have any control over even remotely.
Re:This makes me sad (Score:4, Interesting)
People then get demotivated when their bonus depends on things totally divorced from what they actually do, e.g. you do a great job in IT this year, but your bonus goes down because you work for an oil company and the price of oil went down, something you don't have any control over even remotely.
Not to mention that the slacker next to you got exactly the same bonus. I don't mind it as part of my bonus though, because usually they wouldn't risk increasing your fixed pay that much. It's a lot easier to announce a small or no bonus than it is to announce pay cuts. I know a company that in desperation cut pay by 15%, it saved their bacon then and there but the moment the market lightened up just a little then boom, mass resignations of the good people and it didn't matter what offers they'd make at that point - nobody would risk another downturn employed there. Bonus is bonus, sure if you usually get a nice bonus you'll miss it but it doesn't have nearly the same destructive effect.
Re: (Score:2)
I once worked at a company that used exactly one metric for determining employees' bonuses -- company profit.
I'm pretty sure I work for a company that does that. Guess what? "The company isn't profitable this quarter, no bonuses" all while the owner pockets a million dollars in pay for the year.
Re:This makes me sad (Score:5, Funny)
Of course it wasn't. If it had been a hedge fund in 2006, they'd have all still gotten bonuses.
How would getting your friends to call in help? (Score:2)
Re: (Score:2)
Re: (Score:2)
Only if the boss isn't looking at individual metrics to determine who to fire. If one person's metrics are substantially lower than everybody else's, then that person can and usually will be let go.
If all your coworkers are doing this, and you are on a team of more than 3 people, then you will actually be benefitting yourself the most if you *DON'T* do this, because since the tickets are usually assigned randomly, if you don't try to game the system, then you lower the chance for everybody else getting
Re: (Score:2)
If it's not your friend, you hang up, wait a minute, then call back.
I'm currently reading TFA... (Score:5, Informative)
Re:I'm currently reading TFA... (Score:5, Insightful)
Yeah, I used to work tech support for what was then the largest Mac products reseller in the US, and that's the kind of metric they used (just calls per hour, not even resolved issues).
There was one tech so bad that people would just hang up and call back. When asked about my long call times, I showed them a dozen calls from the logs where they talked to 'Hank' for 5 minutes, got back in the queue, and then talked to me for 20-50 minutes (the source phone # was in the logs with destinations and timestamps). I never left a customer with an unresolved problem, but that's not what was being measured.
They did understand that the real waste of money was the guy who had 'great' call times, but they also had no way to measure our actual performance, so they used the reports they did have as proxies.
Re:I'm currently reading TFA... (Score:5, Informative)
I can tell you that I can't think of a single IT support org that uses this as a metric
Some years ago, I had a help desk in my organisation that did use this metric as part of how its analysts kept tabs on their performance. It was one metric in an overall package, and the whole team (all the analysts) reviewed the package every week. As I recall, other metrics in the package included Customer Satisfaction, Average Call Length, Number of Calls Back to Users per Agent, Incidents Resovled on First Contact, Incidents Escalated to Second Level, and others.
The help desk team very successfully used the overall metrics package as part analyst self motivation and peer motivation (as well as management oversight). Bob Lewis's piece is provocative journalism: devoid of concrete detail and full of high level innuendo. It doesn't contain sufficent detail (say, by way of actual detailed examples) to allow a typical reader to apply the thoughts he has expressed.
Re: (Score:2)
Re: (Score:2)
Oh? I've got friends who are judged by that very metric. It incentivises their coworkers to cut corners and do things poorly, if not at all: close the ticket and get credit. Someone calls in with the same issue "again", it's another ticket. Nobody protests, because that's another resolution for them, too... (The other big one is "number of calls ber hour, but that's basically the same thing, isn't it? You're supposed to be an automatron, getting off the call faster than a telemarketer gets hung up on.)
Phras
best buy and the other office supplies stores tech (Score:2)
have sales metrics and even the tech side has a big pull to sell stuff even when people don't need it and the extended warranty and accidental damage warranty are even lied about what they cover to push more sales.
Re: (Score:2)
Metrics are only fair for homogeneous work (Score:5, Insightful)
Metrics work if you are comparing two workers on an assembly line doing the exact same work - you can compare their widgets built-per-hour rate (offset by any QA problems).
But when you're dealing with a helpdesk team, the work is no longer homogeneous. The more senior helpdesk person usually gets the hard problems... and he spends more time mentoring his peers (at least he'll do that in a well run team). But tell him that his time-to-resolve metric will determine his bonus and suddenly he'll focus on solving tickets as quickly as possible and instead of volunteering to track down that intermittent printing problem reported by the finance team, he'll leave that for his cohorts and instead will jump on the fast easy tickets.
Re: (Score:3)
The more senior helpdesk person usually gets the hard problems...
But they probably get paid more, too! So if I lay him off, my costs go down and my average number of calls finished per hour go up (because people get tired of trying to talk with idiots) and I get promoted and I win!!!
That's what they call management, my friends!
Re: (Score:2)
You do realize that widget building assembly line work is what most managers think IT is, right? Same for programming.
If it were any other way, they'd not instill assembly line logic (x per y) as a guide for requirements, and the West wouldn't have to contend with Indian accents anytime we actually need to get something done.
Re: (Score:2)
So if you use this metric for anything important to the employee the end of the day becomes either "chatting with Doris" or fast ugly 'fixes', to get the numbers to match the 'right' value.
Next ...
"That which gets measured gets fudged." (Score:5, Informative)
The quote above is from Jerry Weinberg, and it is true.
There's an entire brilliant, short book about this problem: Measuring and Managing Performance in Organizations [amazon.com] by Robert Austin (1996). It's actually a fairly rigorous, somewhat philosophical work, but it is pretty unrelenting to documenting that, indeed, trying to manage by metrics almost always introduces distortions, which in turn are almost always counter-productive. The problem isn't just with IT, it's with any type of effort that seeks to reward or punish based on metrics.
The only metrics that I've found actually useful in IT are those that are predictive -- for example, aiding to estimate the actual delivery date of a project under development. The metrics that seek to somehow measure "accomplishments to date" solely for the purpose of reward or punishment are always gamed and are almost always useless. ..bruce..
Re: (Score:2, Interesting)
Actually, I'd recommend something outside the business field. Someone's going to cry BASTARD LIBERAL ARTS MAJOR on this, but what about Foucault's Surveiller et punir: he traces the development of the modern concept of uniform regimentation -- and the assessment process that accompanies it -- in the prisons and schools that shape or re-shape populations since the nineteenth century. I'm not sure he gets it quite right, since he focuses on France and tries hard to pretend that Prussia doesn't exist, and th
Re: (Score:2)
even non tech call centers suffer from this (Score:3)
even non tech call centers suffer from this and then that's way so many times you get people who don't care are fast to get you off of the phone.
unions in other jobs poor metrics can do worse (Score:3)
unions in other jobs poor metrics can do have done worse and unions are big help to fix bad metrics.
I saw a old post hear or it was linked I think it was about about a glass factory where one one shift was doing more units then the other but the quality was slipping and management was pushing the other shits to do more. So the union pushed the one shift to slow down. In the end they did slow down and quality when up. There is more to it but I don't remember the full post.
Also I think there was this fork lift job where people where by pass safety locks to hit metrics.
Now maybe a IT union can you know step in to talk back about poor metrics to they can do a better job and not just do whats needed to hit numbers.
Re: (Score:2)
>unions in other jobs poor metrics can do have done worse and unions are big help to fix bad metrics.
Really?
Re: (Score:2)
>unions in other jobs poor metrics can do have done worse and unions are big help to fix bad metrics.
Really?
Just wait for his newsletter. All will be explained. You'll find it fascinating.
Re: (Score:2)
The problem is, fish are slippery, and positioning them so as not to chop off half a head or several inches of tasty fillet is tricky. So the technique was to hold them in place near the neck with one hand and use the elbo
Fix the symptom and not the problem (Score:2)
I used to work at a cell phone call centre and many times I got calls from people who have had the same billing issue month after month. The previous call takers had credited the bill for the amount charged but never fixed the reason for the charge. They fixed the symptom rather than the problem. Sometimes it took half an hour to fix the issue instead of the five minutes to issue the credit. My call times went up where their call times stayed low. When they moved me up to Tier two I also got many issues tra
QA isn't much different (Score:2)
QA at our company would be rated better or worse by the number of bugs we filed that got fixed. So it made more sense to focus on small issues (e.g. typo in some text) than bigger issues (e.g. data corruption in a stress environment).
I could file 5-10 easy bugs that get fixed in a day, while someone else suffers trying to explain how to reproduce the data corruption issue.
This wasn't the absolute deciding factor keeping your job, but it had quite some weight to it.
I also wrote the report that the managers u
Stupid measures and Schrodinger management (Score:4, Interesting)
Some years ago I worked for few months at a steelworks, and it was managed as badly as the most rabid Libertarians imagine that the worst of government is run. Management never saw the operation or the city it was in. They just saw numbers, and the most important number graphed on noticeboards everywhere around the plant was "tonnes of steel per man hour".
Now the hours were only counted for permanant employees, so contractors were shuffled in and out by the thousands to skew that number. Since they were not employees and were theoretically off the books there was no training for those that came in from outside of the industry, which of course given the number of people involved and the nature of the workplace resulted in some very serious accidents with multiple deaths. Quality suffered from untrained staff and a desire to increase the tonnage above all else. Revenue went down becuase a lot of material had to be sold as a lower grade of steel, in addition to increased amounts of scrap which still counted in that magical number even though it had to be remelted. Nobody on site had the authority to make any major changes and any reports beyond the interesting numbers were ignored.
It went from being a profitable operation to almost completely shut down within two years - of over 16,000 employees only 300 remained to operate a small rod rolling mill that could get steel shipped in from elsewhere. The losses exposed the company to a takeover bid and they are now owned by Swiss Bankers that have some odd remote control management quirks of their own (which has created a billionaire that picked up one of their discared operations for just about nothing).
Performance metrics are just a simple model and you have to make sure that model actually fits the situation. Trying to change reality to fit an inappropriate model can result in the opposite to what is intended.
The Human Element (Score:5, Interesting)
Stats by themselves will only ever be an indicator what is happening. You really need managers on the ground that are trust worthy to give you feed back on how things are actually going.
Taking humans out of the loop when rating other humans is always a mistake
"cards closed per week" (Score:3)
I worked as an engineer at a company the professed to be "agile" (the quotes are because really, not so much). They started judging performance by "cards closed per week".
You'd be amazed at the number of cards that will be created and closed under those conditions. Our productivity *soared* (according the graph that showed productivity as a measurement of cards closed per week ... ).
this reminds me of a blog post (Score:4, Insightful)
This will not happen in public-sector IT (Score:2, Interesting)
because there are no metrics used.
There is no such thing as bad statistics (Score:5, Interesting)
Be warned: my example is way off topic, but a pet statistic I keep track of.
There is no such things as bad statistics, only bad layman statisticians who don't understand what the numbers actually measure.
Take lines of code, for example. Some people hate it because you can bloat the numbers by adding comments, neglecting to consider how useful those comments are for future maintenance, and thereby a useful application of a developer's time. If you use a consistent formatting style for two projects, you can get a fair grasp of their complexity from the line count, though that will gloss over details about how the code actually works.
The most interesting pattern I've notice in line counts over the years is that the use of templates and other code abstraction facilities really hasn't decreased the size of code much at all, though it's improved readability, maintainability, and programmer API usability substantially. So line counts only give you an approximation of complexity with a language like Java, but do nothing to measure the quality of the code.
One other thing I've found is that complex code looks fat and heavy from it's sheer size, but often compiles to very reasonable executable size and runs rings around supposedly "tight" code that makes heavy use of dynamic techniques like introspection. As only one image of an executable is loaded by a reasonably competent OS, a fat binary does not mean a fat application at runtime.
Big code is only scary if it's not following recognizable patterns and is instead a mishmash of different developer's pet syntax, algorithms, style conventions, naming conventions, and even preferred APIs. If you manufacture it predictably, fat source code becomes a joy to maintain, enhance, and use.
But back to the core topic: help desk performance.
The only help desk stat I care about is a low number on customer complaint reports about the quality of information and assistance provided by the tech team. If it's my company and my budget, I'd rather hire more technicians to handle the load and produce happy customers in the end than I would saving money by overworking and burning them out by even thinking about useless numbers like "calls handled per week."
In the end, if you care about your business, the only thing that truly matters are happy customers who want more services or products in the future, and who will gladly tell others about their good experiences in dealing with you.
There is no substitute for a good word-of-mouth reputation and repeat business. No one ever got fired for buying IBM not because they're perfect, but because their people will go the extra mile to make things work.
Effect and Efficiency. (Score:4, Interesting)
It should be remembered that efficency and effectiveness generally are unrelated.
Efficiency is something that can be measured: responces to calls, forms processed, etc, the sort of thing you can count. It's pretty easy to do this sort of thing, and often the PHBs will take some metric and use it as a measure of activity. Because of this, one often sees things like proformance indicators, and the process and often salary, becomes connected to the indicator. The industry stops being what it is and starts producing 'red beans' for the bean counters. The indicator changes, and one produces blue beans.
Effect is something that is about getting the right job done, both for the customer and for the system. It's not even about what the customer wants, since this supposes that it is the role of the customer to diagnose the problem and the solution, and simply ask for the solution to happen. One needs to think of what happened with the system that responded to cyclone Katrina in New Orleans, which the responce was based on customer wants, rather than pre-assessment by those who should have done this. A call for help is an indicator to a problem, not a proposed solution.
Of course, even though an indicator might be proportional to effect in the wild, when it is proportional to money, the indicator becomes more important to the effect. A doctor, who might have an indicator on consultations, will split several illnesses to several consultations. On a help desk, one is more intent on creating calls, then on providing effect. A call that seeks three problems would be terminated at the first, and new calls needed for the second and third. Also, the process might be extended to several calls to create extra indicator traffic.
In the main, help desk traffic is not a really good indicator of effect, since there are things that effect this. Response time, time to fix, etc, all serve to alter traffic, in some cases, it might be better served by the section guru rather than the help desk. The effectiveness of the guru's solutions may well impede the help desk's overall issues, since it might make matters worse.
One should also note that recording the help calls is also an impediment. It serves no effect, and in many cases, might take as much to make happen as the call does in nature. One might answer say, 90% of the calls first up, yet spend more than 50% of the times making the necessary beans for the counter. A good deal of issues can be condensed into a few batch files (yes, i did this: system configuration is a good candidate for script files), so that while the call is terminated relatively fast, the actual recording might be tedious.
My experience of help desk is that particularly Microsoft rograms (eg Word, Access, Windows), use common names, which makes them very hard to grep for in the system. This reduces the effectiveness of any sort of 'search the job tables' for help. To this end, i used Wart, Abcess, Windoze, much to the annoyances of the PHBs.
I worked in a helpdesk like this once... it sucked (Score:5, Interesting)
Our boss at the time, started giving a $100 incentive to the most number of closed calls. One of the guys in there consistently got the prize. One day, while looking up a call I fat fingered a digit and found myself looking at one of his tickets... it was a ticket, opened and closed about receiving a phone call from X. $ticketnum +1 was the actual ticket for X.
In a nutshell with some sorting/filtering I saw that the guy was not only gaming the system, but hiding the fact that he was grossly incompetent. I wrote everything up and showed it to our boss. Needless to say, he was less than happy not only with this guy, but with me. He was being pushed on from his boss to generate metrics and basically was complicate.
Long story short, I went to his bosses boss i.e. the CIO and voiced my frustration. I pointed out that fallacy of this metric that me imaging a laptop (which back then took hours) vs. Answering the phone both being basically equal to the same measure of productivity made the metric useless. Not to mention the fact that it provided zero incentive to provide better support, just incentive to close tickets.
Obviously, this caused some huge changes. Not the least of which was a much more comprehensive analysis of what people were actually doing. This made quite a few people unhappy because it exposed them for being the incompetent hacks they were. Not the least of which were my boss at the time and that employee.
Re:My metrics are superior. (Score:5, Interesting)
Re:My metrics are superior. (Score:5, Insightful)
Good. glad to see that some VP did the smart thing for once and cut the middle managers instead of the people who actually get the work done.
It is deliciously ironic that you would take a swipe at "middle managers" in this conversation about metrics.
The only way to eliminate middle management, is for upper management to utilize metrics in order to evaluate lower management. There is no time for hands-on management and evaluation with a keen eye in one of these vaunted "flat organizations" with no middle management. And so lower management quickly realizes that their jobs and bonuses depend on the metric, rather than on quality or long-ranged action.
After that, the company is humped... but by then, the "aggressive VP" who wiped out middle management has collected his bonus and moved on.
Re: (Score:2)
...the young king's heroic character is shown by his decision to wander around the English camp at night, in disguise, so as to comfort his soldiers and determine what they really think of him...
Re:My metrics are superior. (Score:5, Insightful)
"Seriously? That's the only way to evaluate? You can't think of a single other way?"
I take umbrage with this sort of response. Your whole purpose is trying to make the parent look stupid, without any sort of constructive input of your own.
By all means, if you know how to perform the job of 10 middle managers with just one top level manager without using metrics, please tell us, otherwise I'm calling your bluff. The parent poster's whole point is that you need hands on management to do proper evaluations. They need to observe you, talk to you, critically evaluate your work, etc. A single top level manager, managing 100 employees can't do that, and if you try to ask the other employees, most decent ones tend to not want to sell out their co-workers.
Metrics was invented for this exact purpose. But a number based on some work stats can't replace critical evaluation.
Re: (Score:3)
Re: (Score:2)
Re: (Score:2)
Re: (Score:3)
Lets take the metric of issues resolved for an IT department.
Agent 1 - Issue 1 - Faulty keys on keyboard, Replaced keyboard - Resolved
Issue 2 - Faulty monitor, reconnected loose cable - Resolved
Issue 3 - User locked out of account, restored account - Resolved
Agent 2 - Inventory database inaccessible, troubleshoot server