Catch up on stories from the past week (and beyond) at the Slashdot story archive


Forgot your password?

The Four Fallacies of IT Metrics 223

snydeq writes "Advice Line's Bob Lewis discusses an all-too-familiar IT mistake: the use of incidents resolved per analyst per week as a metric for assessing help-desk performance. 'If you managed the help desk in question or worked on it as an analyst, would you resist the temptation to ask every friend you had in the business to call in on a regular basis with easy-to-fix problems? Maybe you would. I'm guessing that if you resisted the temptation, not only would you be the exception, but you'd be the exception most likely to be included in the next round of layoffs,' Lewis writes. 'The fact of the matter is it's a lot easier to get metrics wrong than right, and the damage done from getting them wrong usually exceeds the potential benefit from getting them right.' In other words, when it comes to IT metrics, you get what you measure — that's the risk you take."
This discussion has been archived. No new comments can be posted.

The Four Fallacies of IT Metrics

Comments Filter:
  • Business planning (Score:5, Insightful)

    by InsightIn140Bytes ( 2522112 ) on Wednesday December 14, 2011 @09:03PM (#38378728)
    It's bad business planning, but it's also the way any big name linux distroy works. Something not working on your Red Hat Linux? No problem, call us! And that's how they make money. They make money on the promise of fixing problems, and that includes saying that their OS is broken.
    • by fsckmnky ( 2505008 ) on Wednesday December 14, 2011 @09:08PM (#38378764)
      SCO was famous for this. $5,000 minimum support contracts, with $1,000 per incident fees, whether they fixed it or not.

      "Thank you for calling SCO may I help you ?"

      "Yeah, my manufacturing plant just shut down because your kernel panic'd."

      "We're sorry to hear that, but you have the newest version, so there are no updates you can apply to resolve the issue. ($1,000 cha ching)"
      • Fools that signed such contracts deserved the level of support they got. My company would laugh such a contract right out the door.
    • by AdamWill ( 604569 ) on Wednesday December 14, 2011 @09:21PM (#38378882) Homepage

      Support doesn't just mean 'fixing bugs'. It also means 'helping you set things up right', 'helping you optimize your configuration', 'helping you figure out what tool you need for the job at hand', and so on.

      Selling support does not require that the underlying product be broken.

      • by sleigher ( 961421 ) on Wednesday December 14, 2011 @10:49PM (#38379582)
        Wasn't there just a story recently about having gurus on site? Yet, in this example, the production system shut down because of a kernel panic? Hmmmm...

        Any company that is relying on support to keep their production up and running deserves to be down and losing money. Just because some analyst uses big words in a conference room doesn't mean you don't need good people on staff that know their shit!

        I hate IT and I am so sick of it. Here's an ask slashdot! What the hell can I do now? I hate this place! All I know is *NIX and enterprise storage. Load balancing and JBOSS. Virtualization and a decent amount of PERL and Assembly. Maybe flipping burgers ain't so bad after all...
        • by Anonymous Coward on Wednesday December 14, 2011 @10:57PM (#38379610)

          Flipping burgers ain't bad if you own the restaurant.

        • Re:Business planning (Score:5, Interesting)

          by nahdude812 ( 88157 ) * on Thursday December 15, 2011 @10:55AM (#38382984) Homepage

          The major fallacy many big companies fall into is that some of these systems have been running flawlessly for years, because they hired a competent IT staff. They look at the price of those paychecks and shiver. Why are we paying so many high priced engineers when we've never had a problem, they think.

          So they reduce staff and start to rely on support contracts instead of on-site gurus. The gurus are still there to solve any oh-shit moments. But that back investment in good engineers has produced a stable infrastructure that runs with few problems for years. So they reduce staff more, pay for more support contracts, and eventually the system critical mass is greater than the engineers who can support it. It's no problem until it's a problem.

          Eventually something minor goes wrong, but nobody notices or if they do it's not really their field of expertise so they don't understand it's minor now but could escalate. When it does, something else goes wrong, and a cascade effect takes out more and more systems. With a full staff, you have enough guys that when the critical mass is reached, they can start defensive measures and get things back in working order in no time. With support staff only, things are going wrong faster than they can deal with it.

          "Call on our support contracts," shout the bosses! So now your on-site staff are all on hold instead of troubleshooting. When they get through to someone, they have to spend the first hour or two describing their infrastructure to the technician on the other end, who starts making random suggestions that maybe help, but probably don't.

          My anecdote on this front is a company I used to work for. It's a long read, but demonstrates the failures at several levels which is the direct result of this kind of thinking. The Oracle transaction log disk was getting full. Some warnings came in, but disks running low on space was an every day occurrence, we'll send an email to the person on record as being responsible for those servers, and troubleshoot why the "Executive Dashboard" is responding a bit slow today (it's for the execs, it's automatically high priority). Except that person is currently aboard an airplane on his way to help reduce staff in east Asia, he'll be incommunicado for the next 19 hours or more.

          It seems like an innocent enough problem, it's just a log disk, the worst thing that could happen is we lose some logs, right? Whoops, transaction logs are pretty important for Oracle. The fact that the disk is filling up at all is itself an indicator that something bigger is wrong; this shouldn't happen. But critically once the disk does fill up, Oracle will enter read-only mode. Or it should. This time it doesn't, it shuts down. BOOM, offline. So down goes SAP. With SAP down, our entire business is offline. We can't take orders, we can't ship orders, we can't pay bills, we can't pay paychecks, the hourly workers whose shift is starting can't even clock in. Some buildings with tighter badge access can't even be entered unless someone inside opens an emergency door to let someone in.

          Once the transaction log disk was full, Oracle will no longer start up, it needs some space on the log disk to log startup-related transactions. Two hours on hold with Oracle Gold Pressed Latinum level support they finally get an engineer. Wow, this is something he's never seen before, Oracle should have gone into read-only mode before this happened! The only solution anyone can seem to think of is to get some bigger disks for the transaction logs, clone the data over to these new disks and give the startup another go. We have hot spares on a shelf, but nobody knows this. Finding disks requires a different support contract, they can have disks out to us tomorrow. Yeah, that's not going to cut it. Someone literally drives out to a distribution warehouse. Two more hours down (they actually send two different guys in different cars with instructions to take different routes in case one runs into traffic or gets in an acciden

      • by Anonymous Coward on Wednesday December 14, 2011 @11:19PM (#38379704)

        As someone that works in support it would be nice if customers understood how to use their support system.
        All the screaming and yelling that it's broke will not tell me what 'it' is or what is 'broken' about it.

      • Re:Business planning (Score:5, Interesting)

        by FairAndHateful ( 2522378 ) on Thursday December 15, 2011 @01:44AM (#38380444)

        Support... ... also means 'helping you set things up right', 'helping you optimize your configuration', 'helping you figure out what tool you need for the job at hand', and so on.

        Worked at a support center... I was a "talk to them until they understand" guy, playing the long game... I figured while it might not take every time, if I got people to understand, they could get back to work and not break things for just a little bit longer. You know, it costs two people money if they have to talk to me while I help them.

        One of my coworkers got huge amounts of management praise for processing lots and lots of cases... My management was too dumb to run numbers on how many callbacks he had, that the rest of us were fixing...

        Yeah, sure I was spending too much time with each person, but half of my time was fixing this jerk's mistakes. There's probably some of that at every support center. It takes 10 minutes to fix a problem, but 5 minutes to get them to go away. You can look very busy by making them go away, if management isn't clever enough.

        I'm rather happy with my new position... I get to review other people. And I do it fairly.

    • It's bad business planning, but it's also the way any big name linux distroy works.

      Don't you normally include the phrase "nobody can deny"?

      They make money on the promise of fixing problems, and that includes saying that their OS is broken.

      So insurance companies make money on the promise that your house will burn down, and that includes saying that houses are flammable?

      • by gutnor ( 872759 )

        So insurance companies make money on the promise that your house will burn down, and that includes saying that houses are flammable?

        Well yes - insurance companies tell you all the time that houses are flammable. They even list everything that you will lose if your "so flammable, it could be considered burning already" house do actually burn in their advertisement. They also tell you that you will lose your phone, have your car stolen, crashed and destroyed. They also tell you have a good chance to die unexpectedly and leave your family in misery if you don't get their life insurance policy.

  • by russotto ( 537200 ) on Wednesday December 14, 2011 @09:09PM (#38378788) Journal

    Losers realize this simple fact, instantly think of several ways to game the metric, then don't do it figuring that "obviously" the decisionmakers realize the metric is horribly broken. Then they get laid off. Winners spend hours, days, or weeks coming up with one way to game the metric, pat themselves on the back for being so clever, and do it. Then they get promoted, eventually to a position where they come up with metrics of their own.

    • by Hatta ( 162192 ) on Wednesday December 14, 2011 @09:12PM (#38378802) Journal

      Unfortunately, this is true. Evil will always triumph because good is dumb.

      • Re: (Score:3, Insightful)

        by ShieldW0lf ( 601553 )

        Unfortunately, this is true. Evil will always triumph because good is dumb.

        Evil will never truly triumph over Good because if it does it will have nothing left to eat next season.

        • by GNU(slash)Nickname ( 761984 ) on Thursday December 15, 2011 @07:05AM (#38381572)

          Unfortunately, this is true. Evil will always triumph because good is dumb.

          Evil will never truly triumph over Good because if it does it will have nothing left to eat next season.

          No, Least Evil becomes next season's Good, and then the spiral continues.

        • by Kjella ( 173770 ) on Thursday December 15, 2011 @07:22AM (#38381628) Homepage

          Evil will never truly triumph over Good because if it does it will have nothing left to eat next season.

          Unfortunately, this isn't very true. Even if you know the job is a ripoff where someone else is taking 99% of the profits, in the choice between a job and no job people will work under sweatshop conditions to put food on the table. That is something that's happening today. To take an even more extreme historic example, the slave trade lasted for centuries. You are a slave, your children will be slaves, your grandchildren and great-grandchildren will be slaves. If anyone did the starving, it wasn't the slave owners. Yes, the circle of slavery was eventually broken, but you'd need a really, really long perspective to say that Good will triumph some day. I don't see that the future holds any guarantees either, there's plenty sci-fi futures where you have robotic or unmanned weapons system that are utterly loyal. They will never desert, never refuse to obey orders, never let their feelings get in the way of killing unarmed civilians and imposing a reign of terror. Technology is no guarantee you will have more freedom, even though some technologies have obviously helped.

    • Almost. (Score:5, Insightful)

      by khasim ( 1285 ) <> on Wednesday December 14, 2011 @09:17PM (#38378848)

      Winners understand that tech support is a stepping stone and treat it as such. Which means that they move up as soon as possible.

      Tech support managers are under pressure to keep their costs down. So unless you're okay with working for less money than the others there (but still solving as many problems / answering as many calls) you will be replaced with a new, cheaper person as soon as they can find one.

      The metrics are just there to justify replacing you.

      • Re:Almost. (Score:5, Insightful)

        by jonamous++ ( 1687704 ) on Thursday December 15, 2011 @08:54AM (#38381930)
        I manage a support environment (granted, they are not script-readers) and we pay our support staff very well. I also have one of the lowest turnover rates in the industry, despite the fact that we are busy and the job can be stressful. In fact, most of my turnover is losing support reps to our DBA and Development groups as the folks advance in skill (win/win for the company and for the rep).

        It really depends on what you are supporting and your skill level. I'm willing to pay more for someone who is a great problem solver; someone who can connect to a client's environment and do whatever it takes to solve a problem - take risks, explore, and find solutions. I'm certainly not going to replace that person with someone cheaper who can't resolve the problems we face.
    • by AdamWill ( 604569 ) on Wednesday December 14, 2011 @09:23PM (#38378900) Homepage

      Yeah, this is pretty much the problem. Performance evaluation should really be done by crazy, high-tech methods such as you and your peers and manager sitting down and discussing what you've achieved, but that kind of thing is way too hard to stick into an Excel macro, after all...

      Another classic example: call centres which measure 'performance' mainly by the average call time metric. Which gives tech support workers all the incentive in the world to give out any piece of bogus advice that'll get the customer to hang up as fast as possible. Or just hang up on them, if the phone system isn't sophisticated enough to detect it.

      • by Ethanol-fueled ( 1125189 ) on Wednesday December 14, 2011 @09:46PM (#38379096) Homepage Journal
        As a widget-fixer, our analog is obviously how much shit we can fix in a given time frame. One of the biggest mistakes I've seen in multiple companies(ranging from laptop to medical device repair) is that the PHB keeps a board or chart showing how many widgets each tech fixed during a given timespan.

        Any idiot can see that the misguided sweatshop-style metrics cause the following problems:

        Cherry-picking - Techs choosing and even stashing away (!) the returns with the easiest and quickest problems to fix. It matters not that your expensive gadget has been sitting there for a month, there are numbers to be made and we'll get to yours when we want to regardless of the order they came in.

        Racing - When there are no "easy" ones to be cherry-picked, then the techs will race to fix your item. They will ignore problems and cut corners on others. Stripped screw hole? Super-glue the screw in. Low output? Game the settings so the tests will pass. Part shortage? Cannibalize and rob Peter to pay Paul in a hardware-sort of Ponzi-scheme.
        Status Quo and mediocrity - The top performers will become accustomed to the attaboys and will continue to produce slipshod repairs, even if there is a slowdown in work when they can do their job right. Meanwhile, the low performers will become used to it and feel no need to better their work.

        My idiot boss in the company I'm in now considered it and was shot down by every tech. In this company, due to the variety of products, one person could make tens of thousands of dollars with 1-2 days work while another tech working on a different product will have to spend more labor and overhead juggling external vendors and all the headaches it involves only to make a couple thousand dollars. Yeah.

        Fortunately, the consultants we brought in are smart. They listed generic milestones and a cheeky "100%" as the goal with the smiling disclaimer that it will probably never happen.
        • by DigiShaman ( 671371 ) on Thursday December 15, 2011 @12:48AM (#38380206) Homepage

          I work for an MSP (Manage Service Provider). We account for time every 15 minutes. Inactive, internal department active, billable active, and non-billable active. All of this logging of time gets calculated out as metrics that define our bonus. So the outcome is pretty much as you've stated. But that's ok, we know how the metric get calculated and thus we game the system of metric without cheating our clients out of money. Naturally, that would be dishonest to do otherwise. But I'll be damned if I sit back and be judged and taken advantage of by some MBA that can't even interoperate the concept of what those numbers are supposed to mean in the first place. They only need to know two things. Is the work billable to the client, and how much. They're free to speak to a manager if they wish to contest the hours performed and/or quality of work. The point is, we want their business. So it serves no point to lose clients for us.

          It will get worse I hear. Rumor has it we will be timed every 5 minutes with a USB activity button. Sort of like a Chess timer or some such. Also, our keyboards will be logged for activity and application fields will track mouse moment and other activity. It's absolutely nuts. At this rate, they'll need to hire me a secratary just to do the logging for me while I focus on actual work. Hey, now that's cost effective right? I bet they didn't think of that, did they. Doh!

    • by perpenso ( 1613749 ) on Wednesday December 14, 2011 @10:03PM (#38379266)
      Its not just the losers. Talented and rational technicians and engineers bend to the rules of the system too. Basically you get what you incentivize, what your reward. If you reward people for complying to some metric then they will generally comply. It does not matter what everyone agrees is right, it does matter if management says quality is important. If the metric decides whether you get to keep your job or get that raise then the metric is what the company gets regardless of what the company asks for or whether the company's goals are actually advanced.
      • Typo: it does *not* matter if management says quality is important
      • by CAIMLAS ( 41445 ) on Thursday December 15, 2011 @05:25AM (#38381198) Homepage

        Talented or not, those people are not ethical.

        On the third hand, you've got those of us who put our effort into work instead, bill accurately, and don't dick about with meaningless excel spreadsheet metrics. We do our work, ignoring the rest, and end up getting a lot more done than ticket munching assholes.

        Then we quit for a substantially higher paid job when the idiots start thinking you're doing nothing. Maybe we steal our previous boss's clients (because they like our work). Maybe we start our own company.

        Metrics are what kill small IT companies. IT (and programming) are some of the least trackable/accountable things you can think of - it's how people get by doing absolutely nothing for years and years while still making close to $100k a year: their lack of work and fuck ups fall in someone else's lap, while they spend their time attempting to look important and knowledgeable (while they are neither).

        The problem is that the people doing these things - metrics - don't know better. That's what they were taught in management school, and that's what so-called industry knowledge says you should do. (It works in India, right? So it's gotta work here. Except, it doesn't really work in India, and that's why their product is shit.)

        Ultimately, it will be their end. No company can thrive without actually paying attention to what the smallest unit within their organization is actually producing and accomplishing, to some level. Metrics just attempt to abstract actual work into a number discernible by those who don't understand anything but numbers. It doesn't end well.

    • by inviolet ( 797804 ) <> on Wednesday December 14, 2011 @10:34PM (#38379474) Journal

      Losers realize this simple fact, instantly think of several ways to game the metric, then don't do it figuring that "obviously" the decisionmakers realize the metric is horribly broken. Then they get laid off. Winners spend hours, days, or weeks coming up with one way to game the metric, pat themselves on the back for being so clever, and do it. Then they get promoted, eventually to a position where they come up with metrics of their own.

      It's not just IT. Our entire society has converted over to metrics. An easy example comes to mind: the stock market versus a company's quarterly performance. Another set of particularly nasty examples is found in our justice system: police officers evaluated by their number of citations, prosecutors by their number of convictions, prisons by their dollars per inmate per day.

      I get the financial impetus to switch to metrics. Where it used to be one skilled manager overseeing per 5-7 employees, it can now be one schmuck manager with an Excel spreadsheet overseeing 30 employees.

      I even get the psychological impetus. Numbers give us that all-important feeling of certainty, and at low cost too... while the traditional alternative requires legwork, mindwork, judgment, contemplation, and mistakes.

      But it's wrecking our society.

      • If we were dumb enough to turn it all over to bean counters and business grads, we deserve what we got. Now how do we change it. Because that's the only other alternative to a system you don't like living in. Bitching about it won't get you, me, or anyone else anywhere. Of course we could help the bean counters by living with it and chewing our own tails when it gets to us. Kind of like these kinds of stories and the reactions we see here.
      • I dunno, I'm a paramedic, and the only metric I've ever been judged by is the number of patients I kill (Still 0!). Note, that's not the same thing as the number of patients I fail to save.

        All told, I think that's a pretty fair metric to hold me up to.
      • by CAIMLAS ( 41445 )

        It's wrecking more than our society.

        Where it used to be one skilled manager overseeing per 5-7 employees, it can now be one schmuck manager with an Excel spreadsheet overseeing 30 employees.

        This is short-sighted thinking. Sure, they make a lot in 6 months, a year, maybe two years. But then something goes horribly wrong (in their eyes), and shit starts to fall apart.

        I suppose it's great if you plan to pump and dump, as the inclination is in this culture... a bit of a catch 22, I suppose.

        God save the children, because they're not going to inherit much of anything.

  • by jhoegl ( 638955 ) on Wednesday December 14, 2011 @09:14PM (#38378816)
    I am going to get a little mean here, but if a company is doing this they are looking to outsource you because they dont understand.
    So fuck em and dump em.
    Anyone worth their salt will look at downtime, stability, and resolutions before they look at resolution time.
    • Re:Who does this? (Score:5, Insightful)

      by wisnoskij ( 1206448 ) on Wednesday December 14, 2011 @09:32PM (#38378976) Homepage

      Probably the same people who consider number lines of code written per hour as a good metric to evaluate their employees productivity.

      • That's almost asking for a shitload of dead code to be pasted into a routine that just adds two numbers.

        The use of metrics to measure peoples performance is usually implemented incorrectly. You want to measure my performance? Okay, I did 1 thing all day. It was rebuilding a $50,000 test stand. Yesterday, I built 3 custom test fixtures for the production line, and worked on maintaining some of my equipment.

        The number of events being measured is correct, but what about the value to the company of each event?

      • At least give credit where it belongs for the idea that that is a bad metric:

        -2000 lines of code []

        -- Terry

      • by CAIMLAS ( 41445 )

        *gasp* What're you talking about? Programmers with a high LOCPH (lines of code per hour) are rock stars. MOTHERFUCKING ROCK STARS, you hear? It doesn't matter if it's good code, it doesn't even matter if it's maintainable. They're putting out the features and making the big dollars.

        Oh, I see what you did there.

    • Re:Who does this? (Score:4, Interesting)

      by thegarbz ( 1787294 ) on Thursday December 15, 2011 @02:17AM (#38380566)

      Anyone worth their salt will look at downtime, stability, and resolutions before they look at resolution time.

      Ahh so number of resolutions are better than resolution time are they?

      My recent experience with our IT call centre at work (Company is top 10 in the Fortune 500) I needed access to a shared drive. We've been asked to email the call center with specific detailed requests if we know what the exact problem is to save the phone support for problem identification sessions.

      Anyway my email went along the lines of: "I have recently for some unknown reason lost access to network share with no explanation. Access to the drive is necessary for "

      I got not 1 but 3 replies from the service centre:
      Email 1 (autogenerated): Your Incident INC xxxxxxx1 has been raised for access to a network share.
      Email 2 (typed): Dear User, Requests for access to network shares need to go through .
      Email 3 (autogenerated): Your Incident INC xxxxxxx1 has been closed with successful resolution.

      Errr no it hasn't. They generated a case number and closed it successfully but I still have no access to the network folder. Anyway off I go to the other system and request access through it. I get 3 emails again:

      Email 1 (autogenerated): Your request has been received and has been forwarded to the service centre.
      Wait for real? The same schmucks who I just requested this through and been sent away get the request?
      Email 2 (autogenerated): Incident INC xxxxxxx2 has been raised for access to a network share.
      Email 3 (autogenerated): Incident INC xxxxxxx2 has been closed with successful resolution, please wait up to 1 hour for permissions to propagate.

      Well there you go two incidents were raised and closed, and in neither case was the end user asked if it actually worked. I wonder what happens if they gave me read only access instead of read-write as per my original email, given that the system we were supposed to use didn't specify. I guess that would raised a 3rd case.

    • by Wovel ( 964431 )

      In most medium+ organizations the hel dek woul have nothing to do with stability or downtime. Resolutions would not be any better than resolution time. First call resolution are tracked a lot. They are also subject to eBay manipulation. There are not any help desk metrics not subject to what is discussed in the article.

      The view is cynical. Unfortunately, it is also accurate.

  • by tedgyz ( 515156 ) * on Wednesday December 14, 2011 @09:14PM (#38378818) Homepage

    This problem was aptly portrayed in the classic dilbert comic strip in 1995.

    I'm going to code myself a minivan. []

    • You have an amazing memory. You've managed to remember every single Dilbert comic strip since 1995 (or beyond). You then picked the one most relevant to this article. You should be the one getting the mini-van!

      • I'd pity him instead. Being able to remember something like that means it's probably applicable to where he works. I'll bet it's pinned to his wall.

  • ain't pretty. (Score:5, Insightful)

    by Caerdwyn ( 829058 ) on Wednesday December 14, 2011 @09:17PM (#38378842) Journal

    Such metrisc also disincentivize people taking proactive steps to reduce the number of incoming tickets (i.e. making the system/environment more robust or your users more educated), and disincentivizes managers for so doing by reducing the number of people needed to service incoming tickets (thus reducing the size of the empire and the pay grade of the manager).

    I've seen both "disincentives" in action. It ain't pretty.

    • by Muros ( 1167213 )

      Such metrisc also disincentivize people taking proactive steps to reduce the number of incoming tickets (i.e. making the system/environment more robust or your users more educated)

      Tell me about it. The job I'm in is support/maintenance/administration/installation etc. for companies too small to have their own IT depts. I'm constantly saying "we should do this because it will prevent xxxxx in 6-12 months time". I'm sometimes listened to. "Has the customer logged a problem? No? Well we're flat out, there are more important things that need to be done". Usually true, but if we always did the things I recommend we'd have far fewer sudden customer crises demanding immediate attention. Of

  • that it always sounds like a good idea when you're thinking of implementing it and few people go beyond the "this sounds like a good idea" phase to the "how can I game the metric I just thought up?" phase.
  • This makes me sad (Score:5, Interesting)

    by multiben ( 1916126 ) on Wednesday December 14, 2011 @09:22PM (#38378894)
    Metrics are great for some things. For making sure that your employees are working they are terrible. I used to work in a metric free environment and there was a great team atmosphere. Then metrics came along and it all went to hell. Now everyone is so focussed on making their numbers look good that the whole organisation is suffering from a weird sense of internal competitiveness. People no longer collaborate on difficult problems because there is no measure within the metrics system to reflect that this occurred. People who used to be innovative are no longer so, because they are not rewarded for spending time innovating. It has achieved nothing good that I can see.
    • Re:This makes me sad (Score:5, Informative)

      by Trepidity ( 597 ) <delirium-slashdo ... g ['kis' in gap]> on Wednesday December 14, 2011 @09:46PM (#38379098)

      Sounds like academia, actually. It's all about impact factor, citation count, and grant dollars these days...

    • Re:This makes me sad (Score:5, Interesting)

      by bunratty ( 545641 ) on Wednesday December 14, 2011 @10:16PM (#38379374)
      I once worked at a company that used exactly one metric for determining employees' bonuses -- company profit. That got everyone to work together to generate more revenue and cut costs. The first year it was in place, everyone in the company got a 25% annual bonus. The downside was that the next year the economy went sour and no one got a bonus.
      • Re:This makes me sad (Score:4, Interesting)

        by Trepidity ( 597 ) <delirium-slashdo ... g ['kis' in gap]> on Wednesday December 14, 2011 @11:51PM (#38379874)

        I have no large-scale study, but my anecdotal information on those kinds of schemes is that they can have the opposite problem in large companies: rather than promoting uncooperative individual greed, they instead promote a sort of feeling that, "well, as one peon I can't possibly move the needle on this gigantic corporation". People then get demotivated when their bonus depends on things totally divorced from what they actually do, e.g. you do a great job in IT this year, but your bonus goes down because you work for an oil company and the price of oil went down, something you don't have any control over even remotely.

        • Re:This makes me sad (Score:4, Interesting)

          by Kjella ( 173770 ) on Thursday December 15, 2011 @07:43AM (#38381664) Homepage

          People then get demotivated when their bonus depends on things totally divorced from what they actually do, e.g. you do a great job in IT this year, but your bonus goes down because you work for an oil company and the price of oil went down, something you don't have any control over even remotely.

          Not to mention that the slacker next to you got exactly the same bonus. I don't mind it as part of my bonus though, because usually they wouldn't risk increasing your fixed pay that much. It's a lot easier to announce a small or no bonus than it is to announce pay cuts. I know a company that in desperation cut pay by 15%, it saved their bacon then and there but the moment the market lightened up just a little then boom, mass resignations of the good people and it didn't matter what offers they'd make at that point - nobody would risk another downturn employed there. Bonus is bonus, sure if you usually get a nice bonus you'll miss it but it doesn't have nearly the same destructive effect.

      • by CAIMLAS ( 41445 )

        I once worked at a company that used exactly one metric for determining employees' bonuses -- company profit.

        I'm pretty sure I work for a company that does that. Guess what? "The company isn't profitable this quarter, no bonuses" all while the owner pockets a million dollars in pay for the year.

  • Unless you are the only support person, then your friends have as much chance of getting you as anybody else, and your friends would only be helping other peoples' metrics and not yours. In fact, if everybody on the tech team isn't doing this, you would probably be harming yourself more than helping, since fewer of your friends' calls getting to you means more of the calls that aren't from your friends getting to you.
    • As long as you and your coworkers have enough work to keep you busy, your boss won't play Russian Roulette with the team. So increasing everyone's numbers helps.
      • by mark-t ( 151149 )

        Only if the boss isn't looking at individual metrics to determine who to fire. If one person's metrics are substantially lower than everybody else's, then that person can and usually will be let go.

        If all your coworkers are doing this, and you are on a team of more than 3 people, then you will actually be benefitting yourself the most if you *DON'T* do this, because since the tickets are usually assigned randomly, if you don't try to game the system, then you lower the chance for everybody else getting

    • If it's not your friend, you hang up, wait a minute, then call back.

  • by stephencrane ( 771345 ) on Wednesday December 14, 2011 @09:30PM (#38378958)
    ..but I'm not so keen on /.'s article description here. "...the use of incidents resolved per analyst per week as a metric for assessing help-desk performance..." Having worked in this area for decades, I can tell you that I can't think of a single IT support org that uses this as a metric. It's a straw horse, of which there are many when it comes to metrics. The three most common metrics are: Cost per incident Customer Satisfaction Resolution on First Contact (sometimes FC is defined as 'resolved at/within tier 1, even if it means') There are usually two more, but those tend to vary on your business and priorities, if you have SLAs/OLAs, and what service channels you offer. Average speed of answer/Time to Respond to Client is usually next. Average Time to Resolution sometimes. People sometimes care about Abandon Rate, but only within the context of the customer satisfaction metric. A nice place may poll for employee satisfaction. A nicer place does it more than 1-2/year. I've never even seen 'resolved/analyst/week' come up in discussions, forums or books going back to the early 90s. And seriously - NOBODY running anything but a penny ante 100 call/week call center would ever try to regularly cook the stats by having friends and family calling in to boost the customer contacts. It's too much work for too little bang, and it's too easily caught. Any place with a real ACD system, eventually, will notice that a not-insignificant number of calls/emails are coming from the same 10 addresses/numbers. It's just not worth it. The description implies the exact opposite. If you don't have a real ACD system and a real incident-management/ticket-tracking software, you're not really measuring anything anyway and you're probably working at a place that's not complicated enough to care about metrics in the first place.
    • by bill_mcgonigle ( 4333 ) * on Wednesday December 14, 2011 @10:12PM (#38379344) Homepage Journal

      Yeah, I used to work tech support for what was then the largest Mac products reseller in the US, and that's the kind of metric they used (just calls per hour, not even resolved issues).

      There was one tech so bad that people would just hang up and call back. When asked about my long call times, I showed them a dozen calls from the logs where they talked to 'Hank' for 5 minutes, got back in the queue, and then talked to me for 20-50 minutes (the source phone # was in the logs with destinations and timestamps). I never left a customer with an unresolved problem, but that's not what was being measured.

      They did understand that the real waste of money was the guy who had 'great' call times, but they also had no way to measure our actual performance, so they used the reports they did have as proxies.

    • by crath ( 80215 ) on Wednesday December 14, 2011 @10:21PM (#38379400) Homepage

      I can tell you that I can't think of a single IT support org that uses this as a metric

      Some years ago, I had a help desk in my organisation that did use this metric as part of how its analysts kept tabs on their performance. It was one metric in an overall package, and the whole team (all the analysts) reviewed the package every week. As I recall, other metrics in the package included Customer Satisfaction, Average Call Length, Number of Calls Back to Users per Agent, Incidents Resovled on First Contact, Incidents Escalated to Second Level, and others.

      The help desk team very successfully used the overall metrics package as part analyst self motivation and peer motivation (as well as management oversight). Bob Lewis's piece is provocative journalism: devoid of concrete detail and full of high level innuendo. It doesn't contain sufficent detail (say, by way of actual detailed examples) to allow a typical reader to apply the thoughts he has expressed.

    • Back when I did phone tech support for an ISP their metric was the number of calls per day. It's been a long time since I worked there, but if memory serves, techs were expected to handle at least 20 calls each shift. I don't know what the average tech thought of the idea, but I never could see how anybody could judge our performance simply by counting calls. Every now and then we'd hear talk about judging us by how many customers had their issues completely resolved in one call (Not a bad idea, as we ke
    • by CAIMLAS ( 41445 )

      Oh? I've got friends who are judged by that very metric. It incentivises their coworkers to cut corners and do things poorly, if not at all: close the ticket and get credit. Someone calls in with the same issue "again", it's another ticket. Nobody protests, because that's another resolution for them, too... (The other big one is "number of calls ber hour, but that's basically the same thing, isn't it? You're supposed to be an automatron, getting off the call faster than a telemarketer gets hung up on.)


  • have sales metrics and even the tech side has a big pull to sell stuff even when people don't need it and the extended warranty and accidental damage warranty are even lied about what they cover to push more sales.

  • by hawguy ( 1600213 ) on Wednesday December 14, 2011 @09:35PM (#38378998)

    Metrics work if you are comparing two workers on an assembly line doing the exact same work - you can compare their widgets built-per-hour rate (offset by any QA problems).

    But when you're dealing with a helpdesk team, the work is no longer homogeneous. The more senior helpdesk person usually gets the hard problems... and he spends more time mentoring his peers (at least he'll do that in a well run team). But tell him that his time-to-resolve metric will determine his bonus and suddenly he'll focus on solving tickets as quickly as possible and instead of volunteering to track down that intermittent printing problem reported by the finance team, he'll leave that for his cohorts and instead will jump on the fast easy tickets.

    • The more senior helpdesk person usually gets the hard problems...

      But they probably get paid more, too! So if I lay him off, my costs go down and my average number of calls finished per hour go up (because people get tired of trying to talk with idiots) and I get promoted and I win!!!

      That's what they call management, my friends!

    • by CAIMLAS ( 41445 )

      You do realize that widget building assembly line work is what most managers think IT is, right? Same for programming.

      If it were any other way, they'd not instill assembly line logic (x per y) as a guide for requirements, and the West wouldn't have to contend with Indian accents anytime we actually need to get something done.

  • by bfwebster ( 90513 ) on Wednesday December 14, 2011 @09:35PM (#38379004) Homepage

    The quote above is from Jerry Weinberg, and it is true.

    There's an entire brilliant, short book about this problem: Measuring and Managing Performance in Organizations [] by Robert Austin (1996). It's actually a fairly rigorous, somewhat philosophical work, but it is pretty unrelenting to documenting that, indeed, trying to manage by metrics almost always introduces distortions, which in turn are almost always counter-productive. The problem isn't just with IT, it's with any type of effort that seeks to reward or punish based on metrics.

    The only metrics that I've found actually useful in IT are those that are predictive -- for example, aiding to estimate the actual delivery date of a project under development. The metrics that seek to somehow measure "accomplishments to date" solely for the purpose of reward or punishment are always gamed and are almost always useless. ..bruce..

    • Re: (Score:2, Interesting)

      by Anonymous Coward

      Actually, I'd recommend something outside the business field. Someone's going to cry BASTARD LIBERAL ARTS MAJOR on this, but what about Foucault's Surveiller et punir: he traces the development of the modern concept of uniform regimentation -- and the assessment process that accompanies it -- in the prisons and schools that shape or re-shape populations since the nineteenth century. I'm not sure he gets it quite right, since he focuses on France and tries hard to pretend that Prussia doesn't exist, and th

    • Back in manufacturing, the quote was: "What gets measured gets done. What gets rewarded gets done well."
  • by Joe_Dragon ( 2206452 ) on Wednesday December 14, 2011 @09:38PM (#38379028)

    even non tech call centers suffer from this and then that's way so many times you get people who don't care are fast to get you off of the phone.

  • by Joe_Dragon ( 2206452 ) on Wednesday December 14, 2011 @10:13PM (#38379348)

    unions in other jobs poor metrics can do have done worse and unions are big help to fix bad metrics.

    I saw a old post hear or it was linked I think it was about about a glass factory where one one shift was doing more units then the other but the quality was slipping and management was pushing the other shits to do more. So the union pushed the one shift to slow down. In the end they did slow down and quality when up. There is more to it but I don't remember the full post.

    Also I think there was this fork lift job where people where by pass safety locks to hit metrics.

    Now maybe a IT union can you know step in to talk back about poor metrics to they can do a better job and not just do whats needed to hit numbers.

    • >unions in other jobs poor metrics can do have done worse and unions are big help to fix bad metrics.


      • by grcumb ( 781340 )

        >unions in other jobs poor metrics can do have done worse and unions are big help to fix bad metrics.


        Just wait for his newsletter. All will be explained. You'll find it fascinating.

    • by u38cg ( 607297 )
      My favourite example of this was working in a fish factory. Removing the head of a large fish is done with what is effectively a large mechanical guillotine. It's controlled by two buttons, which you activate at the same time, one with each hand to ensure you don't chop your hand off.

      The problem is, fish are slippery, and positioning them so as not to chop off half a head or several inches of tasty fillet is tricky. So the technique was to hold them in place near the neck with one hand and use the elbo

  • I used to work at a cell phone call centre and many times I got calls from people who have had the same billing issue month after month. The previous call takers had credited the bill for the amount charged but never fixed the reason for the charge. They fixed the symptom rather than the problem. Sometimes it took half an hour to fix the issue instead of the five minutes to issue the credit. My call times went up where their call times stayed low. When they moved me up to Tier two I also got many issues tra

  • QA at our company would be rated better or worse by the number of bugs we filed that got fixed. So it made more sense to focus on small issues (e.g. typo in some text) than bigger issues (e.g. data corruption in a stress environment).
    I could file 5-10 easy bugs that get fixed in a day, while someone else suffers trying to explain how to reproduce the data corruption issue.

    This wasn't the absolute deciding factor keeping your job, but it had quite some weight to it.
    I also wrote the report that the managers u

  • by dbIII ( 701233 ) on Wednesday December 14, 2011 @10:30PM (#38379456)
    It's not just a problem in IT. It's a problem anywhere that managers are out of their depth. When they are very badly out of their depth poor attempts to observe what is going on can have very bad effects on an operation.
    Some years ago I worked for few months at a steelworks, and it was managed as badly as the most rabid Libertarians imagine that the worst of government is run. Management never saw the operation or the city it was in. They just saw numbers, and the most important number graphed on noticeboards everywhere around the plant was "tonnes of steel per man hour".
    Now the hours were only counted for permanant employees, so contractors were shuffled in and out by the thousands to skew that number. Since they were not employees and were theoretically off the books there was no training for those that came in from outside of the industry, which of course given the number of people involved and the nature of the workplace resulted in some very serious accidents with multiple deaths. Quality suffered from untrained staff and a desire to increase the tonnage above all else. Revenue went down becuase a lot of material had to be sold as a lower grade of steel, in addition to increased amounts of scrap which still counted in that magical number even though it had to be remelted. Nobody on site had the authority to make any major changes and any reports beyond the interesting numbers were ignored.
    It went from being a profitable operation to almost completely shut down within two years - of over 16,000 employees only 300 remained to operate a small rod rolling mill that could get steel shipped in from elsewhere. The losses exposed the company to a takeover bid and they are now owned by Swiss Bankers that have some odd remote control management quirks of their own (which has created a billionaire that picked up one of their discared operations for just about nothing).

    Performance metrics are just a simple model and you have to make sure that model actually fits the situation. Trying to change reality to fit an inappropriate model can result in the opposite to what is intended.
  • The Human Element (Score:5, Interesting)

    by snero3 ( 610114 ) on Wednesday December 14, 2011 @10:56PM (#38379604) Homepage

    Stats by themselves will only ever be an indicator what is happening. You really need managers on the ground that are trust worthy to give you feed back on how things are actually going.

    Taking humans out of the loop when rating other humans is always a mistake

  • by BrianRoach ( 614397 ) on Thursday December 15, 2011 @12:18AM (#38380038)

    I worked as an engineer at a company the professed to be "agile" (the quotes are because really, not so much). They started judging performance by "cards closed per week".

    You'd be amazed at the number of cards that will be created and closed under those conditions. Our productivity *soared* (according the graph that showed productivity as a measurement of cards closed per week ... ).

  • by ILongForDarkness ( 1134931 ) on Thursday December 15, 2011 @01:02AM (#38380262)
    on Raymond Chen's site (The old new thing) a sporting goods store wanted to increase the upsales of "shoe protecting" sprays. They offered the staff a kickback since they had a ridiculously high margin on the spray (like 80%). Anyways the sales people were allowed to use discounts at their discrecion and the store had coupons frequntly. So ... smart employees gave away the spray and used coupons or "discounts" to make up the difference so they'd get the kick back. In general any behavior you offer a reward for that isn't exactly what you want as a company will result in you getting what you are incentivizing regardless of whether or not you get what you want out of the deal. The only solution: tie the reward directly to what you want, eg. if you want more profit for the company than give profit sharing. You still have to work hard to remove disincentives, ie crappy employees that make the goal unachievable and so make even your good employees just take it easy since there is no reason to put in the extra effort, but at least you make your employees incentives tied to your organization wide goals.
  • because there are no metrics used.

  • by msobkow ( 48369 ) on Thursday December 15, 2011 @01:41AM (#38380434) Homepage Journal

    Be warned: my example is way off topic, but a pet statistic I keep track of.

    There is no such things as bad statistics, only bad layman statisticians who don't understand what the numbers actually measure.

    Take lines of code, for example. Some people hate it because you can bloat the numbers by adding comments, neglecting to consider how useful those comments are for future maintenance, and thereby a useful application of a developer's time. If you use a consistent formatting style for two projects, you can get a fair grasp of their complexity from the line count, though that will gloss over details about how the code actually works.

    The most interesting pattern I've notice in line counts over the years is that the use of templates and other code abstraction facilities really hasn't decreased the size of code much at all, though it's improved readability, maintainability, and programmer API usability substantially. So line counts only give you an approximation of complexity with a language like Java, but do nothing to measure the quality of the code.

    One other thing I've found is that complex code looks fat and heavy from it's sheer size, but often compiles to very reasonable executable size and runs rings around supposedly "tight" code that makes heavy use of dynamic techniques like introspection. As only one image of an executable is loaded by a reasonably competent OS, a fat binary does not mean a fat application at runtime.

    Big code is only scary if it's not following recognizable patterns and is instead a mishmash of different developer's pet syntax, algorithms, style conventions, naming conventions, and even preferred APIs. If you manufacture it predictably, fat source code becomes a joy to maintain, enhance, and use.

    But back to the core topic: help desk performance.

    The only help desk stat I care about is a low number on customer complaint reports about the quality of information and assistance provided by the tech team. If it's my company and my budget, I'd rather hire more technicians to handle the load and produce happy customers in the end than I would saving money by overworking and burning them out by even thinking about useless numbers like "calls handled per week."

    In the end, if you care about your business, the only thing that truly matters are happy customers who want more services or products in the future, and who will gladly tell others about their good experiences in dealing with you.

    There is no substitute for a good word-of-mouth reputation and repeat business. No one ever got fired for buying IBM not because they're perfect, but because their people will go the extra mile to make things work.

  • by os2fan ( 254461 ) on Thursday December 15, 2011 @05:03AM (#38381094) Homepage

    It should be remembered that efficency and effectiveness generally are unrelated.

    Efficiency is something that can be measured: responces to calls, forms processed, etc, the sort of thing you can count. It's pretty easy to do this sort of thing, and often the PHBs will take some metric and use it as a measure of activity. Because of this, one often sees things like proformance indicators, and the process and often salary, becomes connected to the indicator. The industry stops being what it is and starts producing 'red beans' for the bean counters. The indicator changes, and one produces blue beans.

    Effect is something that is about getting the right job done, both for the customer and for the system. It's not even about what the customer wants, since this supposes that it is the role of the customer to diagnose the problem and the solution, and simply ask for the solution to happen. One needs to think of what happened with the system that responded to cyclone Katrina in New Orleans, which the responce was based on customer wants, rather than pre-assessment by those who should have done this. A call for help is an indicator to a problem, not a proposed solution.

    Of course, even though an indicator might be proportional to effect in the wild, when it is proportional to money, the indicator becomes more important to the effect. A doctor, who might have an indicator on consultations, will split several illnesses to several consultations. On a help desk, one is more intent on creating calls, then on providing effect. A call that seeks three problems would be terminated at the first, and new calls needed for the second and third. Also, the process might be extended to several calls to create extra indicator traffic.

    In the main, help desk traffic is not a really good indicator of effect, since there are things that effect this. Response time, time to fix, etc, all serve to alter traffic, in some cases, it might be better served by the section guru rather than the help desk. The effectiveness of the guru's solutions may well impede the help desk's overall issues, since it might make matters worse.

    One should also note that recording the help calls is also an impediment. It serves no effect, and in many cases, might take as much to make happen as the call does in nature. One might answer say, 90% of the calls first up, yet spend more than 50% of the times making the necessary beans for the counter. A good deal of issues can be condensed into a few batch files (yes, i did this: system configuration is a good candidate for script files), so that while the call is terminated relatively fast, the actual recording might be tedious.

    My experience of help desk is that particularly Microsoft rograms (eg Word, Access, Windows), use common names, which makes them very hard to grep for in the system. This reduces the effectiveness of any sort of 'search the job tables' for help. To this end, i used Wart, Abcess, Windoze, much to the annoyances of the PHBs.

  • by Mysticalfruit ( 533341 ) on Thursday December 15, 2011 @08:11AM (#38381784) Homepage Journal
    I worked in a helpdesk many years ago where we were all measured on the number of calls per week we closed. There was no consideration towards the complexity of the call given.

    Our boss at the time, started giving a $100 incentive to the most number of closed calls. One of the guys in there consistently got the prize. One day, while looking up a call I fat fingered a digit and found myself looking at one of his tickets... it was a ticket, opened and closed about receiving a phone call from X. $ticketnum +1 was the actual ticket for X.

    In a nutshell with some sorting/filtering I saw that the guy was not only gaming the system, but hiding the fact that he was grossly incompetent. I wrote everything up and showed it to our boss. Needless to say, he was less than happy not only with this guy, but with me. He was being pushed on from his boss to generate metrics and basically was complicate.

    Long story short, I went to his bosses boss i.e. the CIO and voiced my frustration. I pointed out that fallacy of this metric that me imaging a laptop (which back then took hours) vs. Answering the phone both being basically equal to the same measure of productivity made the metric useless. Not to mention the fact that it provided zero incentive to provide better support, just incentive to close tickets.

    Obviously, this caused some huge changes. Not the least of which was a much more comprehensive analysis of what people were actually doing. This made quite a few people unhappy because it exposed them for being the incompetent hacks they were. Not the least of which were my boss at the time and that employee.

Don't tell me how hard you work. Tell me how much you get done. -- James J. Ling