Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Businesses IT

IT Crises vs. Vacation: Sometimes It Isn't Pretty 352

Posted by timothy from the could-put-strychnine-in-the-guacamole dept.
CWmike writes "It's true that IT systems have become essential to business operations, but the successful functioning of the IT department shouldn't rest on any one person's shoulders. All told, vacations serve as mini tests to prove if a department can function when key players are away. That's the theory, anyway. In reality, IT departments sometimes flunk. The results can either be comical or turn out to be a serious wake-up call to organizations that need a better Plan B. To prime your mental pump before your own vacation, Computerworld compiled anecdotes about good vacations gone bad."
This discussion has been archived. No new comments can be posted.

IT Crises vs. Vacation: Sometimes It Isn't Pretty More

IT Crises vs. Vacation: Sometimes It Isn't Pretty

Comments Filter:

  • The real problem (Score:5, Insightful)

    by rsilvergun ( 571051 ) on Sunday July 10, 2011 @11:36AM (#36712152)
    at least in the states is, companies have figured out they can get one person to do the work of two and pocket the other guy's salary. I'm seeing this everywhere in the form of longer wait times for services. It's also really screwing the economy because it means there's 1 less job available, so higher unemployment and less money circulating. We're heading back to the 1800s, when our masters argued that idle hands were the devil's playthings, and the lower class would just spend the time drinking anyway... Kiss your vacations (and your 40 hour work week) goodbye.

    • and that lead to the hit by bus problem what do you do then hot shot?
      Make person work from the hospital room high on pain killers? What if they are to out of it to work?

      Hire some one real fast and hope they can get up to speed real fast on what your setup is like? and you better hope that there is some one there who knows how to hire tech people.

      Have some other person fill in the roll + work there own job and hope they can do the tech stuff? And how long can you get by with that before burn out or there own

    • Re:The real problem (Score:5, Interesting)

      by Ephemeriis ( 315124 ) on Sunday July 10, 2011 @12:07PM (#36712398)

      at least in the states is, companies have figured out they can get one person to do the work of two and pocket the other guy's salary.

      This.

      I'm working at a small hospital. Our entire IT department is just three people - one clinical liaison, me, and our manager.

      The clinical liaison is awesome at what she does, but she can't build a server or fix a network issue or any of that. Her job is to train the nurses and explain the issues they have in terms we can understand and things like that. She isn't supposed to be technical support.

      My manager is certainly skilled... But he's stuck in meetings most of the day, or working on grant proposals, or putting together purchase orders, or whatever. He's rarely available to fix technical issues.

      Which means that all of the day-to-day support, and most of the longer-term projects, fall on my shoulders. I've been begging for another technical person for months, and it just isn't happening.

      So I'm getting stuck working longer hours... And support is still suffering. It takes me longer to get to the little things, which gives them time to grow into bigger things. And the bigger things are getting fixed as quickly as possible, which means corners get cut. There's not enough time to properly plan/implement/train on new projects, which results in more things going wrong...

      The end result is that I'm doing sloppy work, and causing more problems, which means more sloppy work... I can see what is happening, but I can't really do much about it. There's only so many hours I can work before becoming absolutely useless (As the 17-hour day I put in last week showed very clearly. I was completely useless the next day.)

      I've got vacation time coming up next week... Which I've had to cut short, due to go-live for a new product... But I've still got a few days off. And, while I'm really looking forward to the break, I'm kind of dreading what I'll face when I come back to work.

      • Re:The real problem (Score:5, Insightful)

        by JoeRandomHacker ( 983775 ) on Sunday July 10, 2011 @12:19PM (#36712504)

        Here's a thought: stop working longer hours; you are just reinforcing management's bad behavior. At this point you are clearly too important to sack, so no worries there. Do your excellent job at a reasonable pace, and keep a backlog of things you have to do, making it available to interested parties who wonder why things aren't getting done faster. And when new work arrives, let them know that while you would be more than happy to fix their problems right away, there is a pile of other stuff to get through first, so it will have to wait. And most of all, stop worrying about it. It may be that nobody ever wises up and get you some technical help, but at least you'll be less overworked, and maybe a bit happier.

        • Re: (Score:2)

          by TarPitt ( 217247 )

          You can choose to stop working longer hours

          And your company can choose to eliminate your position the next time they are thinking about layoffs.

          Refusing to work insane hours gets you tagged as "not a team player". People so tagged get on the short list for reduction in force.

          In most cases, employment protections do not exist in the USA, especially if a termination can be made to look like a layoff.

          • Re:The real problem (Score:5, Insightful)

            by bigstrat2003 ( 1058574 ) on Sunday July 10, 2011 @12:48PM (#36712740)

            It's like this. I work to enable me to enjoy the rest of my life, not for its own sake. If my life is really shitty as a result of my job, then I might be better off without it.

            This is one of those things that happens because we allow it. If IT employees were willing, across the board, to demand proper respect and consideration from their employer, then there would be no problem. Until we stop cowering in fear of losing our jobs, we're going to be screwed unless we happen to get lucky and have a nice manager.

            • you're right, it's happening because we allow it, but it's broader than IT. The only way to stop cowering in fear of losing our jobs is to force our society to put up a permanent and complete safety net. That means unemployment benefits that don't run out, ever, and it means not getting our panties in a bunch if some lazy bloke doesn't want to work at all.
          • what it's like in the rest of the world, do you? In India, China and the 2nd and 3rd world, you don't got to 'choose to stop working longer hours'. It's a rough scramble for bare existence. That's what the super wealthy want here. Right now they're allowing the the luxury. For one thing, you've forgotten what unemployment insurance is really for. It's to keep the unemployed from desperately flooding the job market and removing that choice from you. For another thing, you've forgotten that people in this cou

      • Re: (Score:3, Insightful)

        by todrules ( 882424 )
        It sounds like you are really in a downward spiral here. You need to start looking after yourself more and putting your foot down. It is commendable that you work so hard at your job, but at what cost? See about taking your original, planned vacation. Fill your boss and coworker in on the upcoming deployment, and let them handle it.

      • Re:The real problem (Score:5, Insightful)

        by EdIII ( 1114411 ) on Sunday July 10, 2011 @06:19PM (#36715284)

        Quit. Find another job, or at least start looking right now. Even if you safely move one foot to the other like a stepping stone, DO IT.

        I am speaking from experience here. Salaried and was working 20 hours a day for over a year and half. I finally did a re-fi and cashed out the equivalent of 3.5 years worth of my salary (I was not compensated well in the first place).

        First two weeks I slept 14 hours a day and woke up after having nightmares where I dreamed about cronjobs, databases, and SQL statements. I would wake up screaming with panic attacks wondering if certain processes were done.

        Quit.

        What it did was completely shot my adrenal glands. I had nearly killed them. Doctors put me on stuff for a year, and it took about that long for me to get back to normal sleeping patterns and feel better.

        It truly is not worth it, life is too fucking short. Don't waste your life on this.

        If you have children, a wife, or a girlfriend that IS suffering right now. If you have children you are performing a great sacrifice and being a good father, but you could be a better one being there. I can honestly say that I would have rather been a little more poor, eaten a little less, had less conveniences and video games if I got to see my father more.

        The reason why this continues in this environment is that WE let it. An awful lot of IT people are not lazy at all. Some are driven to bad behavior and apathy, but more are driven to keep things alive at all cost. It is our purpose. I took it that seriously. It was like the Path of the Warrior and shit like that. The SQL server WILL not fucking do down BECAUSE I WONT LET IT. I'll figure out a way to work with what I have to make things as redundant as possible.

        Quit. Quit. Quit.

        Find another job, even if it pays only 80% as much, even if it is a different type of job. Quit.

        If you continue down this path, it will be you writing this post to a possibly younger IT guy on some website in the future trying to tell him the same thing. Although, honestly, I would have ignored the advice back then and pushed on like a solider anyways... till I physically could not do it anymore.

        Good luck.

      • Re:The real problem (Score:5, Informative)

        by Archangel Michael ( 180766 ) on Sunday July 10, 2011 @06:55PM (#36715494) Journal

        First off, it sounds like you don't value your life. Until you value your life, it will be made miserable by your colleagues. You and your boss need to figure out the scope of what is "normal" duties and everything that exceeds those duties comes at a budget cost to other departments. The trick is to get the OTHER guy to say "no".

        Them: "We need a new ______ to do _________"

        You: "That sounds Great. I'd love to do that, but I'm currently doing _______ (list) for the foreseeable future, which of those things would you like me to stop doing so that I may tackle your project? Alternatively, could you budget an extra $150K for our department so that we can hire someone to do ________ (list) while I start some of these more interesting projects?

        I don't ever say "no", I tell them what it will take and ask them if they have budget codes for overtime, and the complete project work schedule. It sounds like you don't do bill backs for things and this is where metered work and a nice Help Desk System comes into play. "Can you put in a work order ticket for me" works wonders, because, as you'll find out, that if it isn't important enough for them to do what is needed, then it isn't enough for you to do it. You let them set their priorities (in writing) and you just do what they want you to do.

        In short, let them figure out how to get you more help when it is too busy by making them choose what is important to them, all the while you're saying "sure, we can do that for you".

    • Re: (Score:2)

      by mickwd ( 196449 )

      "companies have figured out they can get one person to do the work of two and pocket the other guy's salary".

      http://dilbert.com/strips/comic/2011-07-07/ [dilbert.com]

    • Re:The real problem (Score:5, Insightful)

      by innerweb ( 721995 ) on Sunday July 10, 2011 @12:31PM (#36712576)

      As always, the problem is management and management not knowing enough about what they are managing,

      Most management has two issues they contend with when it comes to IT. One is they can not see how IT contributes to profit, and therefore see it as nothing more than cost. Another is they do not see how IT can help enhance and manage work flow.

      For an IT person to be successful, they need to learn how their management hears things, and learn to talk to them in a way they will hear. Which means to get where you need to be and to get what you need you have to sell it by talking to them at their level. That may not be easy. Sometimes it involves golf games, sometimes fishing, whatever it takes to get to know management to understand what it is they see and hear (we all have filters). Once you learn how to communicate with them, then you have to start to educate them. Once they learn how IT truly can work, they will start to let you have projects that they would not have let you have before. Choose these projects carefully. The ones they can see and feel the success of get you *karma* points. These points become spendable for projects you need that they will not understand the benefit of. Do enough stuff they can see the benefit of, and eventually, they will see the justification of having another person.

      Should it be this way? Probably not, but it is, so we in IT have to learn to deal with it. Remember that most management is ignorant about IT. And they want to stay that way . Management typically thinks it has too much on its own plate as it is. They manage things like IT by looking elsewhere and saying, hey see what they are bragging about, why are we not like that? Kind of like all those *investment* bankers who collapsed the economy on bad loans and derivatives. Many said, "Not smart", but their bosses ignored them, saying, "Look others are doing it and profiting , we need that profit as well."

      We know that is not real management. But management does not care. If it does not bite them today, it is a good thing today.

      That is why it is our job in IT to stealth educate our management. It is our job to know these things. It is also our job to communicate those needs effectively. That is where most in IT fall down. It is very hard to communicate IT effectively. It is even harder to do so when you do not have a grasp on the whole of the company's operations. To be able to explain IT in terms the rest of the company will understand, you have to know their jobs and how IT is used to help them. So, one of the reasons IT management is so extraordinarily tough is that you have to know everything about how the company works to be able to do the job effectively. That means not just running the IT department. It means knowing in full detail how the IT department impacts the company as a whole in every nook and corner. It actually means you wind up knowing more than anyone else in the company about how the company works. IT management is the hardest job in any company.

      And that is a natural things when you think about what IT truly is in a company. It is everywhere in a company. The phone system, the desktops, the printers, the servers, the network, the data, the data sharing, the personnel, ... Companies work or don't work because of their Information flow. Information flows because of IT. IT becomes the lifeblood of the company. Wrong numbers in inventory, parts are not made. Wrong field size for an import, data lost. Wrong version of software, job might not get done. Nothing in a company is as pervasive as IT.

      We could all go back to pen and paper to track things. We could remove all the digital IT in every company. The job could and would get done (well, most would). But, as what cost? This is probably the thing that an IT person has to understand the most to manage the company (not just IT). This cost reduction from using data systems is where management will understand you. But you have to understand it first. Only then can you demonstrate why hiring another person in IT is profitable.

      • Re: (Score:2)

        by Greyfox ( 87712 )
        I prefer to use the "everyone is going to die*" allegory when talking to management. We need to upgrade Windows or everyone is going to die. We need a new server or everyone is going to die. We need to hire another admin or everyone except me is going to die next time I go on vacation. It's surprisingly effective, and it works every time.

        * Of course technically everyone is STILL going to die, but that's one of those management blind spots you can gloss over with a vague hand wave.

  • Everything the article says is true, but ... (Score:4, Insightful)

    by Daniel Dvorkin ( 106857 ) on Sunday July 10, 2011 @11:47AM (#36712232) Homepage Journal

    ... it won't make any difference, I suspect. There are (at least) three big problems here:

    • Techie-macho. A lot of IT people have the idea that if you're not working at least twelve hours a day, six days a week, you're not really working. They don't just put up with sleep deprivation, irregular eating schedules, and the total lack of a social life when it's necessary to get the job done; they actively take pride in it. Vacation isn't even on their radar.
    • Management ignorance. IT is seen as a cost sink, so there's a reluctance to hire any more than the bare bones minimum staff to get the job done (and often, not even that.) When the inevitable problems occur as a result of this practice, it's seen as proof of IT's inefficiency and an excuse to cut their budget and resources even more.
    • (Often justified) paranoia. "If the boss knows the guy in the next cubicle over can take over for me when needed, I'll be the first one out the door in the next round of layoffs." This encourages "job security through obscurity," the creation of needlessly complex and poorly documented systems that only one person in the entire company can understand. It may not actually work, but people think it does, so they keep doing it.

    And no, I don't know what the solution is. Just pointing out that it's a structural issue, and collecting anecdotes about how badly things tend to go wrong doesn't seem to be doing much to motivate anyone to try to fix the underlying problems.

    • That's destructive behavior. Any outside vendor can act as the body double. They would work to preserve their own role as emergency contact while reinforcing the value of the day-to-day admin.
    • The solution is a degree of trust and the maturity to realise that you won't get thrown out on your ear if you are delivering the goods and you're good at your job. You don't need to be a superhero - and you almost certainly aren't. Especially if you're regularly working more than 40 hours a week, the chances are a large proportion of those extra hours are spent fixing screwups caused by poor quality decisions or mistakes; made through tiredness.

      Sure, in some places I've done the work of 3 people (though

      • Indeed, and there are plenty of organizations which have that trust and maturity as part of their culture. Unfortunately, there are also a lot that don't. Personally, I've been extraordinarily lucky in my working life; the couple of times I've found myself working at places that pushed the "everyone has to be a superhero" mentality, or had such poor communication that it was nearly impossible to find out what people actually needed, have also been when the economy was doing well and there were better jobs

  • When I leave, I look my fellow co-workers in the eye and say, "If anything happens while I am on vacation. Whatever you do, don't call!" then I turn off my pager and put it on my desk and laugh manically!

  • My wife came up with a solution (Score:3)

    by mbourgon ( 186257 ) on Sunday July 10, 2011 @11:50AM (#36712268) Homepage

    We started going more exotic places. Places that don't have cell signal or internet access. If they want to call a boat to the tune of $5-$10 a minute they're welcome to.

    • +1 - Exactly, this is why I go to a cabin in the mountains. No electricity and, by proxy, no internet, no phone, and certainly no cell service.

  • vacation... or delayed punishment? (Score:3)

    by v1 ( 525388 ) on Sunday July 10, 2011 @11:53AM (#36712288) Homepage Journal

    I know anytime I go on vacation, it causes major headaches for those that try to prevent me from being completely buried by the time I get back

    But I'm always buried when I return. Then I have at least a week of torture trying to catch back up. People say someone else is going to get trained and certified to serve as a backup for me, but it never materializes.

    I think most companies just have to experience the lesson the hard way, by a bus or a plane ticket. And even then, half of them don't learn anything from the lesson.

    They're just too short-sighted. All they see is the cost of investment today, not the return of tomorrow. I find it amazing that business majors, managers, and CEOs don't have that skill. They are blind to the benefits to all involved, and are comforted in the peace they find in keeping their heads in the sand.

  • "On the night before Thanksgiving last year, T.J. Whelan .. phone started buzzing with texts .. The messages said there was no connectivity to the Microsoft Exchange cluster .. That meant that attorneys in the firm's two U.S. offices and two overseas offices were completely cut off from email .. The network manager contacted Dell support, which confirmed that the disks had failed but also reported that it might be a while before replacement parts could be located" ..

    This beggers belief, the IT department of

    • Re:Law firm fails because of single disk failure? (Score:4, Insightful)

      by paitre ( 32242 ) on Sunday July 10, 2011 @12:00PM (#36712350) Journal

      I'm not at all surprised by this, actually.

      In my experience, there is a significant percentage (IME, most, but others may differ) of businesses that rely on Best Buy for hardware replacements. They see additional hardware lying around as a waste, and will not keep spares handy.

      I can EASILY see this happening. EASILY.

    • Re: (Score:2)

      by TarPitt ( 217247 )

      This is a law firm

      If they can't bill it to a client, they spend as little as possible on it.

      The ultimate judge of "as little as possible" is a managing partner completely clueless about IT

    • This beggers belief, the IT department of a major law firm don't keep a single harddrive as backup and don't have a standby system in place for just such an eventuality as a failed harddrive ..

      I wouldn't be shocked by this at all. I've seen several companies keep all their financial records on a 10+ year old PC with NO backups of any kind. This sort of behavior is not uncommon. You would be shocked at how many companies (even big ones) are playing a game of Russian roulette with their IT systems.

  • In the end, everyone agreed that the easiest solution would be for Laping to physically fix things himself. "I had to drive two hours back to push a power button," says Laping, recalling that he turned right around and got back on the road once the NIC was up and running again.

    Lesson learned: Even with smart devices, wireless services and VPN technology, not every problem can be dealt with remotely; make sure your backups know the basics -- like how to power down

    Lesson learned: Hire IT staff who have the me

    • Agreed, if this is mission critical stuff and you don't have staff living onsite there is no reason why they could not justify purchasing an IP KVM and a remote PDU for just this type of emergency. I'd imagine it would take at least thirty minutes to a few hours to have your on call person drive into the office and push a power button, whereas remote access would take what like 10 minutes?

    • I had a support issue similar to this which required only a few mouse-clicks to solve.

      Unfortunately my verbal description of the clicks that were required seemed to be getting lost in translation to the "click-er". Fortunately both of our cell phones had video chat capability, so in the end, he was using his phone's camera to show the screen of the computer, and I was telling him where to click. Problem solved and it saved me an hour of driving in heavy traffic.

      One of the first troubleshooting techniques

  • I wonder if part of the culture of 'we can get by without backup for our IT guy' is down to total length of vacation time that someone has? In the UK its something like 25 days minimum (as opposed to the 10 or so including National holidays that were on offer in the US) and so maybe being able to cope with the systems administrator being away for a couple of weeks at a time is forced upon the organisation.
    • Comment removed based on user account deletion

    • I wonder if part of the culture of ' ...

      I don't think so. I think a lot of the time the techies get caught up in their own self-image. They are often quite impressionable types and see techies in films and on TV - which almost always involves a lone uber-geek who single-handedy runs a billion $$ operation. Just like cops watch cop shows and "learn" how to behave from them, so it is with techies: they try to emulate what they see on TV or in films in some delusional idea that the programme shows how people think they *should* behave. Basically, th

  • Another reason for vacations: crooked employees (Score:5, Interesting)

    by John Jorsett ( 171560 ) on Sunday July 10, 2011 @12:01PM (#36712360)

    I'm personally aware thru my late father, who was an accountant, of two companies that had employees embezzling funds for years. One telltale sign was that they never took vacations, because their replacements would have discovered what they were up to. Businesses should insist that their personnel take time off, just to make malfeasance easier to detect.

  • My wife and I are expecting to test our company next month when we're due to have our first child. She's the senior programmer and many help desk calls get forwarded to her every day and sometimes at 3AM. We've been joking that we're going to have photos of her taking support calls during labor.

    In all seriousness, our options aren't good. We always bring our laptops on vacation with us knowing that our adventures might get put on hold to handle support calls. We're a company of five people, so we're stressi

    • Comment removed based on user account deletion
    • I feel for you. In situations like these, I just want to give the giant middle finger to IT and open up a Parking Lot or something.

  • This is a problem with the downsizing of companies. They try to push as much work as possible onto as few people as possible, often burning out the good people because they never get any time off, are constantly on outage calls, etc., and then nobody listens to them because they've identified a myriad of problems... but fixing them would require not putting out that extra new feature so they use operations to hold things together while disregarding their importance.

  • It's worse the other way (Score:3)

    by DNS-and-BIND ( 461968 ) on Sunday July 10, 2011 @12:10PM (#36712432) Homepage
    OK, let's say you go on vacation and nothing happens. Next time layoffs come around, you'll be perceived as non-essential.

    • If you are essential, you'll have a week or two of clearly observed mania and making arrangements with coworkers for the two weeks off, followed on your return by a similar week or two catching up and following up with co-workers all over.

      And even if you can avoid doing anything during vacation, if you are essential, people will have found your absence a hindrance.

  • Vacation? we don't allow that nonsense here! (Score:5, Interesting)

    by green1 ( 322787 ) on Sunday July 10, 2011 @12:19PM (#36712508)

    At a previous company I worked for I went away for 3 days (friday-sunday) at a company that only worked mon-fri, so it was essentially just a single day off. I made everyone well aware that I would be out of town and unreachable.

    I returned on Monday to find my boss had flown in from out of town and was sitting cross legged on the server room floor with one of our servers in pieces all around him. He informed me that there had been a hardware failure over the weekend and that I should have been there to deal with it. After I finished fixing the original problem, and the problems he had created by trying to fix things, and once everything was up and running again, he asked me for my security pass and escorted me off the premises citing "budget cuts".

    Probably much better that I don't work there anymore...

    • sounds to me like you have a good case for a wrongful termination suit.

      • It depends upon the state. A lot of states are simply, "at will," meaning that you can legally be terminated for farting in the wrong direction. Okay, I know that was an embellishment. In this case, sounds like getting unemployment would be a slam dunk.

      • Re: (Score:3)

        by green1 ( 322787 )

        Well... hard to say, their official reason for terminating me was due to budget cuts, and they did not fill my position once I was gone, it was termed a lay-off and not a firing. (It was actually somewhat satisfying to hear from a co-worker I ran in to almost a year later that the whole IT infrastructure in the company basically fell apart after that.) There's no way I could prove otherwise, so a wrongful dismissal case would have been an uphill battle to say the least. I suppose it's possible that it was a

  • The same sort of management short-sightedness happens in engineering and software development all the time.

    Case in point, at a Fortune 200 companythe senior technical staff all left the project I was on left over the last three years, leaving me as the last senior person. Management saw it as an opportunity to save money and rarely backfilled, and never with senior people. I saw the writing on the wall and started looking over two years ago. Last summer I took a three week vacation and absolutely nothing go

    • That is their fault. This posting is a nice segway from the earlier slashdot posting on MBAs. This is a prime and classic example of two many MBAs and not enough engineers/technical people.

      • Right. I don't blame it on MBA though per se, but PHB. My undergrad was Bus. Admin, emphasis on MIS (then, now I suppose it would be IT) basically the undergrad equivalent of an MBA. One of the things I remember from a Management Theory class I took is that one part of a manager's job is to be able to do their employees' jobs when the employee can't, e.g. because they're on vacation.

        The anecdotes in this thread about managers who couldn't do their employees jobs, e.g. reboot a server, have no business being

      • Re: (Score:2)

        by Gramie2 ( 411713 )

        This posting is a nice segway

        "segue". I blame Dean Kamen for corrupting our language.

  • Vacation? How about WEEKENDS? (Score:5, Interesting)

    by natoochtoniket ( 763630 ) on Sunday July 10, 2011 @12:29PM (#36712568)

    My current company, has no vacations. You simply tell them when you are not going to be there, and they decide if they want to fire you for the absence.

    They also do not have weekends. On the Friday before each customary "3-day" weekend the owner declares an emergency that, somehow, MUST be finished by Tuesday.

    No one wants to work there for very long. Turnover is very high. Projects don't get finished, precisely because of the turnover. Other projects do get "finished', but don't work, also because of the turnover

    The owner doesn't seem to realize that he is sabotaging his own projects.

    • Just wait until the economy improves, this guy is going to be in a world of hurt when there is a mass exodus.

      • Re: (Score:2)

        by JoeWalsh ( 32530 )

        Just wait until the economy improves, this guy is going to be in a world of hurt when there is a mass exodus.

        Eh, that'll be in 10 years if we're lucky, at this rate.

  • The real lesson is... (Score:3)

    by seifried ( 12921 ) on Sunday July 10, 2011 @12:34PM (#36712600) Homepage

    If you are "indispensable" in the sense that without you the IT services can't be maintained/fixed then the company is f**ked regardless. You may go on vacation, get sick, get hit by a car, have a heart attack or simply get a new job. This is true of any job function, but seems especially true of IT, I suspect in large parts because each IT build-out is pretty much a custom job with all sorts of gotchas, exceptions and internal workarounds to address issues, and the system is rarely documented properly.

    I find this especially strange since most companies now rely upon IT to carry out basic functions (telephone, email, workflow, etc.) but fail to treat it as a critical service (single points of failure, especially with respect to personal are more the rule than the exception). Oh well.

  • Every time (Score:5, Interesting)

    by ledow ( 319597 ) on Sunday July 10, 2011 @12:40PM (#36712664) Homepage

    Every time I go on holiday, something weird will happen with a system that's been running perfectly for years. It's guaranteed. And it won't just be because I've been kicking that system back into action all the time and "would get around to fixing it", I would get the really esoteric interconnected problems that suddenly crop up out of nowhere and you're never entirely sure you've solved until months afterwards.

    However, my employer knows I'm on the end of a phone if it is indeed an emergency. They have called on me in Italy several times. The trick is to take holidays FAR AWAY from your place of work, and then they can't do anything but cope without you. Flying back to fix a company server? No thanks. Not unless you provide DOUBLE the time I'd taken off in lieu as compensation for ruining my long-planned holiday through poor planning / hiring. If a company can't cope without any single individual, then its hiring policies suck. What would you do if he went under a bus and was *never* coming back?

    The worst that's ever happened to me is that I lined up my own brother to go into the company should the emergency they were having not be solved by my instructions. It was, however it would have be after-hours, because he works too, but they would at least have someone there who knew the right switch to press, could be talked through a RAID rebuild, etc. and not have to be led every step of the way and incur only a single day's downtime without making things worse.

    Think the downtime wasn't important? It was a school with automated billing, parental contact, phone system, heating controls, registration, medical records, salaries, you name it, not to mention IT lessons and exams. Without registration, etc. the school is legally not allowed to open because they have no records of which children are where, no medical records, etc. Guess what? They coped for the day because they had contingency plans (i.e. cancel all IT lessons and do something else instead, catching up again next week, manual financial control, manual registration, etc.). There are very few companies that *can't* carry on if the IT dies. It might be inconvenient, it might mean harder and more work, but it's rarely impossible unless you're something like an ISP or a datacentre.

    If there was nobody else suitable to come and fix the problem? Not hard - hire an IT guy to come in. You do have support contracts for your gear and software, yes? Or you could organise an emergency contractor to visit and fix your problem? It's not hard and the only problem there is finding the right guy (i.e. someone who *can* walk into the middle of a mess and at least get something working enough to last until the "real" IT guy gets back).

    If you honestly, genuinely can't cope without an employee - you need him to train an assistant, or even two. It won't be perfect but it's better than nothing. Failing that, you need a large enough team that you can do something on the guy's instructions. Failing that, you need your support contracts which pretty much come as standard with business-level hardware/software. Failing that, you need a contractor at short-notice. If you can't do those four and get to a working system of some fashion within 24 hours, you were always going to be in deep shit whenever anything went wrong anyway. What would you do if the guy left and you had to find a replacement? What if he died? What if he suffered amnesia and forgot all the passwords? What if he was arrested? What if, what if, what if. Or you could just do the normal IT-thing and have backups - lots of them.

    Nobody is that invaluable that they have to abandon holidays and drive away from their kids to come back after-hours. Sorry, it's just not true, and if it "is" then that's only the company's fault. It's purely a money saving solution rather than hiring someone else to fix the mess - get the guy who's away on holiday and pay him for a few extra hours - it's cheaper than calling on your support contracts or call-out fees for an emerge

  • When I am on vacation, that is MY time, no one else's. It is a cherished benefit of the job, and as such, I make it clear that I will have LIMITED access to email only. That said, I usually take vacations in the mountains where there is no internet and no cell signal. I like it that way! My idea of a vacation is getting away from what is modern to what is simple.

  • Common issues (Score:3)

    by meerling ( 1487879 ) on Sunday July 10, 2011 @01:00PM (#36712826)
    You wouldn't believe how many people I've talked to in a panic because they are having an issue and need to access the server, but the ONLY person with the key or password is unreachable. (On vacation with no contact number, not responding for some reason, or in a couple of cases, recently deceased.)

    I know security people will often tell you to limit these things, especially passwords, so that only one person has it and it's not written down. Ignore that. You need to control access, but not so tightly that if one thing goes wrong your company is screwed. Always have a password log, and have it stored in a safe and fireproof location. Same with duplicate keys. It's actually safest if there are 2 backups, and at least one kept at a separate location. (In case of fire, flood, building blowing up, etc.) Obviously keep those secure, like in a safe. Is this 100% security on those things? No, but there's no such thing as 100% security, but it will allow you to keep reasonable security and acceptable ability to respond to emergencies. Both are important, and ignoring one to favor the other will eventually leave you screwed.

    And follow the same advice for backups, you need them, they may fail, and they can get destroyed just like everything else. (Easier in a lot of cases.)

    • The company I'm at has a hardcopy list of passwords in a bank safety deposit box. The owner and manager, as well as myself, have access to that box. If I'm unavailable and something needs to be done, the passwords are available, and are secure.

      • Re:Common issues (Score:4, Informative)

        by Glonoinha ( 587375 ) on Sunday July 10, 2011 @06:14PM (#36715242) Journal

        Print them out.
        Put them in an envelope.
        Seal the envelope, and sign across where it is sealed.
        Let 'whoever' know where they are, and if the envelope is opened you want a damn good reason for it.
        If the envelope disappears or is compromised without a good reason, you know to change the passwords.

  • The CEO and the power user were mortified that they couldn't figure out which button to push, says Laping, but this particular machine was a Dell rack server with a flat design rather than the tower configuration with which the men were more familiar.

    The two kept pushing a button that was for adjusting the display, not turning the unit on and off. When nothing happened, they panicked.

    Wait! The "Power User" has never seen the system restarting? He did not even know how to find the manual? The CEO and the pow

    • Re: (Score:2)

      by PPH ( 736903 )

      He did not even know how to find the manual?

      We had a IT supervisor (we were in engineering) that was tasked to support us and our web server in the event engineering staff wasn't around to fix it. I had a 3 ring binder* with detailed instructions on how to maintain practically anything on the system. So he asks why it isn't in in 'digital' format, loaded onto the very web server that his people are going to be trying to fix in the middle of the night.

      *You can read the contents of a notebook even if the lights are out if you have a flashlight.

  • Happened once (Score:4, Interesting)

    by PPH ( 736903 ) on Sunday July 10, 2011 @01:47PM (#36713116)

    They panicked at Boeing when they couldn't get me after hours. So I told them to hire backup. They transfered a guy in from IT who (supposedly) knew Perl (and other stuff). So, on the first day when I was showing him the ropes, I opened up one of the CGI files to show him some of our coding style and conventions. He stared at the file for a few minutes
    (with #! /usr/bin/perl staring him right in the face) and asked, "What language is this written in?"

    Management had a bad habit of calling everyone down to the shop floor at any hour if any little thing went wrong. I had remote login capability and could generally fix anything from home. One night, I get a call. Panic! All hands to the shop! So I ask, "What's the problem? Maybe I can fix it from here." I'm told I've got an attitude problem. So I get dressed and drive in. An hour and a half later, after reading the error message on the ATE equipment console, sure enough, a 15 minute fix from any terminal in the world.

    Often, crisis are engineered to make people look important. Build a system that never dies and nobody notices you. Build one that causes trouble (or hire people who move their lips when they code to maintain it) and you get noticed. And promoted. We had a saying at Boeing: Heads roll uphill.

    • The funny thing is, there would have been less down time had you been allowed to fix the issue remotely and you get accused of having an attitude problem. You just can't win anymore.

      • Re: (Score:3)

        by PPH ( 736903 )

        Yeah. But I think management has a problem with the mental image of someone working a problem while sitting at their home PC in ratty boxer shorts. Or worse yet .... naked.

        Seriously, when managers don't really have any power they need to create the image of having it. They do so by seeing how many people they can get to jump at their command.

  • It's called CMM Level 3.

    If a service is business critical, it had best be at least the following:

    1. Documented

    This means the process is documented well enough so that a reasonably experienced person coming in off the street should be able to muddle through it successfully. This also means that there is budget for documentation.

    2. Trained

    This means that all people responsible (and their backups) are trained on how to perform these critical functions. This means that there is budget for training.

    3. Consistent

  • My advice to avoid this (Score:4, Informative)

    by epseps ( 39675 ) on Sunday July 10, 2011 @02:34PM (#36713462)

    Before I was in charge of hiring for my team, my managers employed a guy who didn't know anything because they did not check his references for some reason. He would always call me while I was on vacation because a simple procedure that he should have known was confusing him. Later our company cell phones were switched from Verizon to AT&T and AT&T had no signal in the Aosta Valley in the Italian Alps.

    That is where I would go on vacations.

    So if you have AT&T, go to the lovely Aosta Valley but do not cross over into France or else your voicemail will be filled with messages.

  • Road Trip (Score:3)

    by jacobsm ( 661831 ) on Sunday July 10, 2011 @09:09PM (#36716370)

    I'm going on a 16 day road trip later this month. I'll have my cell phone but I'm not going to be able to VPN into work while driving. Last year I took some time off and received work related phone calls, including requests to join conference calls 7 of my 9 days off.

    We're understaffed and there's very limited backup support for the high level technical support staff.

  • IT is always short staffed worse than nursing. (Score:3)

    by CAIMLAS ( 41445 ) on Sunday July 10, 2011 @10:13PM (#36716724)

    Good fucking luck.

    Chances are, most functional organizations under 400 hundred people have only one or MAYBE two people who can effectively troubleshoot a bad outage. Sure, they may have an IT staff of 3, 4, 5, 10... but chances are, they're not of the 'sysadmin' type. They're frontline support, most likely, and deal mostly with Windows workstations and servers. For any crucial role, there is no more than one capable person on hand in most IT organizations. The pay masters wouldn't hear of duplicated functionality (that's inefficient!).

    After all, if something in IT breaks, the worst management sees that can happen (unlike a dead body from neglegence/overworking your staff) is for there to be a fired employee. They don't see the big picture.

    Sure, it's nice to feel needed. It's job security. But it's better to be needed and have someone else who can help pull the weight while you're sitting on a beach with a drink in hand, live a little longer, and have your resume ready to go. Being unemployed for a long period of time isn't half as bad as month after month of high-stress environments where you're pressed with "fix it now under pressure" or "I'm completely burnt out and can't maintain this level of service".

  • Vacation not as bad as quitting (Score:4, Interesting)

    by tompaulco ( 629533 ) on Monday July 11, 2011 @12:30AM (#36717346) Homepage Journal
    We had one IT person who handled keeping about 50 servers up and running, 3 or 4 domains, multiple websites, 30 laptops, 3 different FTP servers, communications with the outside agencies that were sending us data, RFPs, customer support, and who knows what else. He quit for another company where he will probably have 1/4 of the job duties, will not have to work weekends, and will get paid more. Our company has not hired a replacement yet. The first IT related issue that came up was a failed backup that occurred about 12 hours after he quit (failed on a weekend). The next one occurred about 24 hours later when a client locked their FTP account because they can't remember their password, and nobody could figure out how to reset it. They eventually had to call him and ask. That debacle resulted in about 200 hours of unpaid overtime.
    Supposedly, they are interviewing replacements, but so far I think they are patting themselves on the back for saving money (about half an IT person's salary, as a guess) and spreading around his work mostly to the overworked development group, including myself, who are now getting surprisingly little development done.

  • perception vs. reality (Score:5, Insightful)

    by Tom ( 822 ) on Monday July 11, 2011 @04:45AM (#36718210) Homepage Journal

    Giving up vacation days because you couldn't use them, interrupting vacations or cutting them short for work - if you find yourself anywhere near that list, you're a fucking idiot, and the one you're fucking is yourself. And that was the polite way of putting it.

    A few years ago, I've had to become a bit of an expert in vacations for business reasons (negotiations regarding vacation times, rules, company procedures, etc.). Two things are absolutely frightening when you do that.
    One is that we need vacations at all - for thousands of years, there was no such thing. That's because work has become condensed to a point where it's detrimental to health at good times.
    Two is how little almost everyone, employers, employees, even most union people, realize how important vacations and other free times are. I've seen many people crash and burn in those years and lack of vacations, interruptions of off-work time and not being able to "shut down" when you leave work were almost always present and at least contributing factors.

    In all those years, I have encountered one group of people who can do that, who can go on without vacations and free days and suffer no ill consequences. These people share two important characteristics: One is that their time is self-determined to a large degree. In other words: They could close up shop and go away for a few days at any time if only they wanted. They have no boss pressure and no customer pressure that would stop them, because they've organized their work so that if they ever need to, they can. Obviously, most of them are self-employed, but not all. A great example is a cobbler who has his shop down the street from where I live: The official opening times of his shop, as posted in the window, are: "When I'm here."
    Two, these people work their dream. They do what they want to do, they have meaning in their jobs, and they've cut out as much of the crap as possible, and some that other people thought would not be possible to cut. They never ask themselves "what the fuck am I doing here?".

    None of them make a killing. But they make a living. And I try to be one of them over being rich, but hollow. I've worked with too many so-called "successful" people and seen their dull eyes. Five minutes with someone from the group above and you can never go back into the machinery.

Slashdot Top Deals

By working faithfully eight hours a day, you may eventually get to be boss and work twelve. -- Robert Frost

Close