Forgot your password?
typodupeerror
IT

Deferred IT Maintenance Is a Ticking Time Bomb 186

Posted by samzenpus
from the we'll-do-it-later dept.
snydeq writes "The underfunding of routine hardware replacement purchases and the degradation of aging enterprise apps pose systemic risk for many IT organizations, thanks to a ballooning 'deferred IT maintenance debt' in the decade since Y2K fears pushed enterprises to invest heavily in essential system upgrades, InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud."
This discussion has been archived. No new comments can be posted.

Deferred IT Maintenance Is a Ticking Time Bomb

Comments Filter:
  • by mschaffer (97223) * on Wednesday January 05, 2011 @05:26PM (#34769866)

    Deferring any maintenance can have calamitous effects.

    I fail to see why this is newsworthy? Is it just because IT people whine louder?
    If you are in the US---just look around. Infrastructure systems are crumbling away because of "deferred maintenance". It's not just IT. It's roads, bridges, state governments, municipalities, houses, businesses---it'severything!

    • by Desler (1608317)

      InfoWorld needed some adclick revenue so they posted this completely duh story

      • by Anonymous Coward on Wednesday January 05, 2011 @06:28PM (#34770612)

        Well, it is sort of a "duh" story the way it is written, but OTOH the subject is not without merit.

        I have been involved with infrastructure assessment of companies prior to acquisition and some stuff is just shocking. Publicly owned companies are driven by return to the shareholders; one way to keep the dividends flowing when the economy is in a downturn or when the business plan isn't working is to reduce operational expense.

        Releasing employees is very effective to reduce the spend side but usually that means there is less available effort to work on maintenance. It looks good to have all employee time capitalized on projects but who is keeping stuff working? Also, each person out the door takes expertise with them that is lost to the company. After a while, the company may not even have enough knowledge internally to understand that their boat has holes in it and that patching isn't happening.

        This isn't smoke; I've seen it. Data centers with overheating problems and with inadequate standby generators. Power is distributed unwittingly to cause a cascading failure if one breaker trips. Leaking roofs over financial servers (plastic tarp and bucket gave that away). Licensing that has not been kept up to date because no one has a good inventory and no one wants to look-see. So... Oracle enterprise instances running in non-secure network zones and without proper licensing ( potentially million$ in back costs). A database server being used as a network monitoring node and firewall because funds were not available to separate the functions.

        Deferred infrastructure investment and maintenance investment happens and it is a ghastly mess to clean up. I am not surprised that more of this is happening.

        • by AmiMoJo (196126)

          Some admins are just scared of things going wrong during maintenance do don't do it. My boss at the last place I worked was like that. He never did updates on servers if he could possibly avoid it. The fear that the server might not reboot and he would have to drive all the way out and spend part of his weekend fixing it stopped a lot of preventative maintenance getting done. It was easier to blame users when the RDP server was hit with a 2 year old virus using a vulnerability that was long since patched. D

    • by I8TheWorm (645702) * on Wednesday January 05, 2011 @05:31PM (#34769952) Journal

      I think it's a setup for the "IT Industry Invaded by Incompetent Idiots" and "CIOs Found Replacing Working Systems with Crap Made By Their Hunting Buddies" articles.

      A large portion of /. readers are in IT and already knew this. However, seeing it in "print" in a newsrag you might find in a CIOs office is a little noteworthy. It means it's only a matter of time before someone comes rushing to your desk to say "Our CIO just read an article about infrastructure and we need an ans..."

      Hang on, someone's at my desk.

      • Re: (Score:3, Interesting)

        by KublaiKhan (522918)
        Oh, is -that- the way to get your boss to authorize expenditures?

        I need to make some friends in the news media.
        • by I8TheWorm (645702) *

          Careful what you wish for... the CIO at the Fortune 100 company I just left still thinks DLink routers without redundancy are the way to go. He still approves purchases for replacements and new ones.

          You could be having fun with that.

          • by hawguy (1600213)

            Sometimes cheap routers are the way to go -- I replaced 2 Cisco routers at a remote site in a seaside warehouse (one due to a power surge when a generator failed, one due to water from a leaky roof) before switching to cheap Netgear routers at about 1/10 the cost. Redundancy? We had a spare configured and ready to go the foreman's truck toolbox and another at his house.

            One of the Netgears even survived a similar water deluge to the one that took out the Cisco (but then the Netgear didn't have a fan to suck

            • Try a pelican case, a drill and some rubber grommets.
          • by CAIMLAS (41445)

            I can't say I particularly care for DLink, but for the cost of a (used) Cisco, I can pick up two Procurves with better specs and a management interface which anyone with basic networking comprehension can master in a couple hours. I'm not hearing anyone say the Procurves are shit (because they aren't). With d-link, I'm sure the same is true - and I can have a stack of hot spares for the same cost, too - Just In Case.

    • by fuzzyfuzzyfungus (1223518) on Wednesday January 05, 2011 @05:35PM (#34770012) Journal
      I suspect two reasons: 1)(and most important): This is being published by Infoworld, ergo it focuses on IT stuff. 2) Much of the worst rot in IT is largely invisible to the layman.

      Slow computers with styles that were pretty neato back in 2000 are obvious to the poor office drones who have to endure them; but anything that new can, largely, be forklift upgraded for the cost of the new systems and some grunt labor. Turning a 3 year desktop refresh cycle into a 5 year(or 7 year, *cough* *cough*) desktop refresh cycle doesn't make anybody happy(particularly once warranties run out, the scavenging and improvising begins); but is architecturally a small problem. You don't really accrue much "debt" over time. The cost will be "1 forklift upgrade to present day PCs" whether that upgrade takes you one generation ahead or three.

      It's the complex software, the highly specialized proprietary industrial controller cards, and suchlike widgetry where there is real hell to pay, and most of that is invisible...
      • by arivanov (12034) on Wednesday January 05, 2011 @06:11PM (#34770392) Homepage

        Depends on the level of "bespoke" in your house.

        Scavenging for desktop parts is the "little devil". Scavenging for people who know how the bloody things work more than 3 years after for IT systems is the real nightmare.

        If a system has been in the field for 3+ year nobody knows what are its real dependencies and what does it really take to augment, add capacity or do any changes. The people who knew have left, gone to pastures new or have forgotten what the problems used to be and no documentation can help you here (even if there is any suriviving docs on the design of the system in question). This is valid for almost all classes of IT and telecoms systems and is the real cost factor in IT "maintenance debt". If we use a real-life analogy IT maintenance debt is like a discounted mortgage. You pay virtually nothing for 2-3 years and after that the lender skins your hide.

        • by puto (533470)
          I agree wholeheartedly with you. I think another large part of the equation which our fellow IT workers fail to admit is that our ilk are incredibly stubborn about replacing and fixing things. IT workers are notorious for telling management that they can make things work with a hodge podge of coathangers and toenails, not because it is the best solution, but because they can. The problem lies on both sides of the coin. Management not wanting to spend money and IT workers not setting a realistic expectat
        • Hey now, don't know Y2K vintage hardware. Place I worked spent a lot on some beige and white Dell workstations that year. They were spiffy at the time. The boss/owner was ultra proud of his Dells and made sure they sat on TOP of the desks so clients and whatnot would see them and be impressed. Woooh he spent money on Dell.

          Boring beige Dell was actually an improvement. Prior to this, the place was run off an old infected Vision PC they got from a radio ad. I upgraded them with almost no budget to DIY s

        • by dbIII (701233)
          It's not just bespoke unfortunately. If, for example,the vendor decided some time after 2003 to lay off their programmers, support staff etc and get a huge suite of software rewritten in India in java but have not finished yet then you just have to have the in-house expertise that the vendor can no longer provide. It sucks having to edit install scripts of expensive commercial software just to get it onto the machines and then alter undocumented scripts to get it to do plot outputs or do remote tape acces
      • IE6 based intranet apps for the win!
    • by Toe, The (545098) on Wednesday January 05, 2011 @05:36PM (#34770026)

      Because much of this IT is stuff that affects individuals who have no influence over it.

      When a company puts off investing in security, for example, and when they also collect and store my credit card info / medical info / personal demographics / shopping history / etc., they are putting me at risk.

      I have to trust that their IT department is on the ball. Something I am beginning to think is never a good idea. But it's impossible to not give companies some info on me and still be a normal modern human, and thus I am forced to trust them all the time.

      So if they're further neglecting their IT, it means my data is more vulnerable. Not that's there's a damn thing I can do about it.

    • Yes, it is "common sense" to you.

      But how many "IT managers" understand that part of their job involves pruning? Killing old systems. Deprecating other systems. Weeding out the "one-offs" that pop up when no one is looking.

      The last place I worked has 7 different database servers. Because they were running 7 different database platforms.

      Virtualization means that you can reduce the floor space needed. But management also needs to look at reducing the systems needed.

      • Just one minor gripe with the parent - a lot of times, what should be weeded out isn't the "one-offs" (which are often times built way under budget with way more capacity and way less maintenance cost), but the actual official enterprise standard that got put in because some CIO was buddies with some sales rep. "One-offs" are a signal that the current standards (either of technology, or product development), are having problems. While not all "one-offs" may be worthy of keeping, when going through the wee

        • by hawguy (1600213)

          I think the one-off's he's talking about are things like the purchase tracking system that Finance installed on a workstation under someone's desk. IT was not aware of it being purchased and of course it has no way to export data into any of our other systems so we can integrate it with the rest of our financial systems.

          Eventually we sucked it into our VMWare infrastructure to get it off of the desktop machine (no RAID disks, and the backup consisted of someone in accounting copying the database to a flash

          • by pasamio (737659)

            You think that's bad. A former organisation I worked at had a similar thing happen where the Finance group bought a software package which was sold on "no IT department required" however the way we found out was even before they deployed it. It was of course a computer program so instead of the Finance department using their own budget allocation, the bought it out of the IT departments budget. When the CIO and the CFO had a meeting the shouting could be heard from the other end of the building.

            Suffice to s

    • by puto (533470)
      I live and work in Latin America. My residence is in Colombia, but for work I am in most LATAM countries, trust me, the US has wonderful infrastructure compared to us. And any latin country i have been in.
    • by c0lo (1497653)

      Deferring any maintenance can have calamitous effects.

      I fail to see why this is newsworthy?

      Probably the estimated amount of money it would be needed to catch-up with the backlog (the so called "IT debt") and a few words on how the estimation has been made? Also, where these money would be needed? TFA:

      $500 billion -- it's a number so big you'd assume it's a component of the national debt. It isn't. Instead, it's what Gartner analyst Andy Kyte calls the IT debt. "

      The "debt" really has two major components: One is underfunding and even neglect of routine but important hardware replacement purchases and software upgrades. The other is the slow degradation of enterprise applications.

      ...

      Is that $500 billion number too high? Kyte says he derived it by analyzing several large Gartner clients that generally do a good job of keeping applications up to date. That led him to estimate that a typical Fortune 2000 company would require upgrades costing more than $200 million each.

      I admit, these as scraps of "meat", but they are nevertheless still meat.
      I imagine that this may be the first step (as an argument) in a push the big IT houses (hardware/software/services) will make to get so more revenue... Something on the line of "well... Apple and Android tablets and phones (that

    • by CAIMLAS (41445)

      I'm guessing it has something to do with the fact that IT managers read InfoWorld. Some read slashdot, too. Many of them are dropping the ball on this, big time. "We don't have the money this quarter" comes up time and time again, and they don't realize that the decreased maintenance is going

      The only people who really realize this kind of thing are sysadmins. Unless the sysadmins have a huge amount of power within the organization (including their own ability to prioritize the budget and 'vision' of the org

    • Now how is pushing services to a cloud exsaserbating the problem? Sure going to the cloud has it pros and conns. But if you are retiring your old servers to a cloud in terms you are Upgrading your infrastructure and removing your old stuff.

  • And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud.

    The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things. Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem. YMMV, of course.

    • by sycodon (149926) on Wednesday January 05, 2011 @05:34PM (#34769992)

      CIOs and organizations blissfully march towards disaster while quietly chanting to themselves, "The Cloud will save us all".

      • by jeffmeden (135043)

        Don't blame the cloud... If it weren't around they would simply chant about outsourcing, virtualization, or right-sizing whilst marching to their doom.

    • by Nikker (749551)
      Something to do with eggs and baskets comes to mind, just because you rent the Limmo you are still the one responsible for the drivers mistakes, etc, etc.
    • Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem.

      "Cloud" (as in, dynamic server provisioning) has very little to do with it.

      Outsourcing IT functions to a firm that is contracted to actually perform the maintenance that was being deferred on the in-house systems (whether hardware, infrastructure software, application software, etc.) obviously can address problems related to deferred maintenance, not because of the outsourcing itself, or because the vendor to whom the operations are outsourced happens to use "cloud" technology to power its offerings, but be

    • by turbidostato (878842) on Wednesday January 05, 2011 @07:16PM (#34771218)

      "The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things."

      Yes, because it's a demonstrated hard fact that those companies providing infrastructure for the cloud can't lower their operational costs by neglecting maintenance; of course they wouldn't do that anyway since it's those infrastructure companies' very valuable data what is at risk if maintenance is neglected instead of their customers'.

      Oh, wait!

  • by GPLDAN (732269) on Wednesday January 05, 2011 @05:35PM (#34770006)
    Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation. Pick up any copy of CIO magazine and look at the toadies who make the top CIOs in the nation, and ask yourself - what innovation did they bring to make that list? What business process did they improve with tech? Only a handful make the cut. Most are there because they are good at pinching out costs, kicking out the older IT workers and either outsourcing or bringing in college grads.

    I routinely see job ads for experienced Java developers, people with hard core experience in integration, esp. with telephony or security technologies, need 5-10 good years, offering $70k tops. Good luck with that, but again it is the CIOs who get the jobs telling people they can staff cheaper, run leaner, cut the corners - that get the job because it is the CFO who is doing the hiring and the performance reviews.

    The big corporation IT C-level execs are a fear driven lot, there are no Gates or Zuckerburgs in their midsts. The action is being with the cloud providers, or the web service providers themselves. Enterprise IT is really a shit place to be outside China. It's a world full of EDS consultants and chickenshit CIOs who won't think how a business could use IT to expand. And the social media space is going to tear a bunch of them new assholes, because none of them know how to leverage it. The startups do.
    • by mlts (1038732) * on Wednesday January 05, 2011 @06:14PM (#34770450)

      Nail, head hit. I have worked for people who had bosses who had zero interest in anything other than cost. The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

      What is ironic is that even though this makes the quarterly figures look good, it is killing competitiveness long term. Another example is R&D. Not product research, but true R&D where people discover something, shrug, put it on a shelf for 20 years, then it becomes immensely marketable (think Corning and Gorilla Glass as the prime example of this.) Instead any groundbreaking research ends up being done overseas or by small firms which are bought out, or have their IP infringed upon. If research is done, it is for a product to cough up this quarter or perhaps this FI, and usually it is how to add a gewgaw to something existing and palm that off on the market.

      You mention China. Chinese companies know how to lock their doors down. They know what happens if you run your company with your fly open, and most of the companies over there wouldn't even be in business had it not been for "borrowed" IP from the West. Take Foxconn as an example. For a company that size making so many Apple products (including ramping up production on unannounced items), they are quite airtight about what their factories produce even with the hordes of workers they have. Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.

      I wish the US would start borrowing from China in this regard. Even with security aside, just because a company's IT infrastructure works today doesn't mean it will work 5 years, or perhaps even six months from now without major issues. Taking a charge off quarterly earnings to fix problems now means a lot of less wasted money when the upgrades have to be done post-haste.

      • by gtall (79522)

        "I wish the US would start borrowing from China in this regard." I wish you were right. However, if an American firm did this, they'd be sued by Chinese companies in American courts using American lawyers...and their Imaginary Property would be protected and the American firm set up for serious damages. And the Chinese know this.

      • by IICV (652597)

        The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."

        To be fair, on the normal USA business timescale (that is, the next two or three quarters), it really doesn't. Security only matters if you're still heading the company when something bad happens, which to most of these people is pretty much unthinkable - because they know they'll be gone in a few years.

      • by jimicus (737525)

        Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.

        I wish the US would start borrowing from China in this regard.

        Give them a few years, they will. It'll start to happen when the first PHB demands evidence that a chinese factory has had no more than three information leaks in the previous 2 years, and they'll get a very bemused email back asking them what sort of information leaks would they like engineered because that's the only way anything's getting out.

      • Not only is the nail on the head, it is worse. The CIO reports to the CFO but doesn't actually know any of the terms that the CFO uses, so all they have are "saved X by not doing Y". A REAL CIO would have the CFO terms down and be able to say "by doing T we can do S things more productively".

        I remember when this hit me about a dozen or so years ago, when I was the business manager for a small regional ISP (when they still existed), and had to process some logs for billing purposes. I had one computer that I

    • by hedwards (940851) on Wednesday January 05, 2011 @06:17PM (#34770488)
      That's not entirely the CFO's fault, a lot of that has to do with the US tax code. If the capital gains rate for investors didn't start to kick in until 2 years down the road and took another year to fully kick in, you'd see longer holding periods. Also, if the short term tax rate was higher for individuals holding for less than a month.

      As it is there's very little reason for corporate executives to think down the road by more than a couple quarters, as chances are the people who own shares now won't still own any much beyond that.
    • by c0lo (1497653)

      Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation.

      Assuming the corporation is not in the IT business but only uses IT, I thing that's only fair to use risk management as the second term of the contrast (instead of tech innovation).

  • Maybe the cloud will mean less in-house IT stuff, which means the IT debt won't even need to be paid off.

    Now if Amazon or Microsoft is putting off the IT work for their cloud systems, that might.. be a problem..

    • by sjames (1099)

      One way or another, it's going to be paid. Either buy and deploy internally or pay an outsourcer to buy and deploy. Meanwhile, you can't just export re-writing your old creeky VB6 apps to the cloud. Even if you find a cloud vendor offering a suitable replacement app, you'll have to pay for that and pay to have your vital business data pried from the cold dead fingers of your old app and imported to the new app.

      It may come out a bit cheaper having the cloud provider do it, but be sure of the business arrange

  • The price of replacing things may be getting cheaper faster than the implied cost of the risk of not replacing it is going up. It may be cheaper to wait until it breaks than to buy something with a rapidly depreciating value. The extra cost of dealing with an emergency may be paid for by the lower cost of the gear and labor. And for those risk events that never happen, the ratio of preventative replacement cost to emergency replacement cost will be infinite.

    Only your CIO knows for sure. But I bet he's p

  • In other news: The sky is blue!

  • "The underfunding of routine exercise programs and the degradation of aging overweight sysadmins poses systemic risk for many IT organizations, thanks to a ballooning 'deferred weight loss program' in the decade since Y2K fears pushed enterprises to invest heavily in dudes who live in their parents' basements", InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep their bodies up and running with foods that barely cover the nutiritional basics',' this 'IT chub' promises only to increase in
  • With hardware prices dropping every year, the longer you can defer hardware upgrades, the less money it will cost you. Given that basic piece of information, it's hardly a surprise that companies don't upgrade until they absolutely have to (anyway: why would they until there's a need?). If they can give their kit a mid-life kicker with some more memory or swapping in some faster CPUs, isn't that better than spending 10s of thousands or more on a new box. Better, that is for everyone except the hardware manu
    • ...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?

      I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part. From HP it was OMFG-are-we-really-spending-this-kind-of-money expensive, a "re-certified" part from a third party was still way more than the $300 you'd spend for your home system as the part and its corresponding VRM were proprietary.

      And the only reason it was done at all

      • by magarity (164372)

        ...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?

        I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part..

        I did and while the parts from HP were outrageously expensive we ordered it from a third party. HP does not have an exclusive market on Xeon chips or the plug-in voltage regulators they used to use (maybe still do?). The tricky part was the heatsink which needed to be a funky size to squeeze in the space allotted but that too was solved from a third party vendor. Total cost was about a third what HP wanted. We did it as a test case to make a single CPU machine with 6 total slots into a dual. It was even

      • by petes_PoV (912422)
        Yes, I've personally seen it done on two seperate occasions. In both the situation was the same. A software vendor that licenced on a per-processor basis (Oracle). One box was a 12 processor macine, the other was a 16 processor job. In both cases it was far, far cheaper to swap the CPUs for faster ones than to add more of the same speed. In both cases that was my recommendation - based purely on cost savings and in both cases it worked very well.
    • by dangitman (862676)

      If they can give their kit a mid-life kicker with some more memory or swapping in some faster CPUs,

      That's just stupid. Simply "dropping in" a faster CPU doesn't do all that much, because you keep the same bus speeds on the motherboard that limit bandwidth. As for memory, haven't you noticed that buying older, slower memory tends to be more expensive than buying the new faster memory, because the old stuff is no longer in production?

      Aside from those factors, there's the added labor in installing those components and testing them. On top of all that, most mass-market desktops that are used in this space ar

      • by petes_PoV (912422)
        Those who have no experience and say it's "stupid" should look and learn from those who are actually doing it and getting their employers good value for money in proven, real-life situations. It *does* work. It's a quick, reliable and scalable solution in many, many cases. Obviously you need a background in performance analysis so that you've correctly identified and quantified the bottleneck - and no, merely playing with a "point and shoot" GUI "tune your system" toy program that you downloaded for free o
  • I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC. Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire. I have documented our needs, but my boss the CIO is afraid to push the issue.
    • by Herkum01 (592704)

      I see replacing equipment as less of issue than purchasing new equipment for additional capacity. Assuming you are doing backups, you can always move hard drives from old-system to a new-system. Yeah that is some work to get it back in running, but it is still a possibility.

      However, if you have servers running at capacity, and they refuse to purchase new equipment, you are asking for problems.

      • "Assuming you are doing backups"

        And that's exactly the problem, my friend. Too many people round here seem to imply that "deferred it maintenance" means not replacing servers when the guarantee period ends up. But maintenance means having two sysadmins when you formerly had three or maintaining the three sysadmins when capacity has grown 50%.
        Lowering maintenance costs means that it has been a year that nobody has the time for a test restore so nobody has noticed that the backups are failing since six mont

    • by c0lo (1497653) on Wednesday January 05, 2011 @06:47PM (#34770848)

      I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC. Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire. I have documented our needs, but my boss the CIO is afraid to push the issue.

      My advice: add some risk analysis argumentation. You know? Something on the line of:
      1. probability of equipment failure [omdec.com] over time - use the "cumulative hazard function" not the "probability distribution function".
      2. impact the server crash will have on the business (make sure you slip-in some "lost face" apropos - after all it would be the manager's face to be lost). If you can express the impact in $$$ and plot the "risk x impact", chances are the budget analyst will "get the picture" easier.

      • by CAIMLAS (41445)

        Forget the risk analysis arguments. If people refuse to see simple logic, hitting them with more of it will do no good. These people will dismiss your reasoning with superstition, thinking along the lines of, "My VCR is 10 years old. It works fine. I see no reason to upgrade our network equipment until my vcr dies."

        For things like that, the only reasonable solution is a loose mains wire and some bridging rod to the offending equipment in question. Then, when you get your new equipment overnighted at 20% mar

  • ... it could seriously reduce 2nd quarter earnings for Cisco, Juniper, Microsoft, Peoplesoft, Oracle, Sun.....

    ~Sticky

  • by jolyonr (560227) on Wednesday January 05, 2011 @06:04PM (#34770336) Homepage

    How, exactly, do enterprise apps degrade?

    Do they suffer from bit-rot, and have some kind of half-life?

    I understand that eventually apps will fail to be supported by the developers, won't potentially work on more modern operating systems, and in some cases require updating in order to be able to work correctly with the rest of the world.

    But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.

    • by Maxo-Texas (864189) on Wednesday January 05, 2011 @06:27PM (#34770610)

      1) The languages, special hardware, libraries and controls become unsupported on new hardware.
      2) The languages, special hardware, libraries and controls become unsupported on new versions of the operating system.
      3) The operating system becomes unsupported.
      4) The hardware becomes unsupported.

      Example: VB6 program uses bar code scanners.

      2004? VB6 unsupported as a language.

      2008, VB6 unsupported for security patches (so any required security patch could kill VB6 program)

      2009 bar code scanners unsupported (change to optical recognition with new software interface)

      2009 VB6 Outlook/Word integration fails.

      2010 Hardware and operating systems to support VB6 start becoming unavailable. All are unsupported by vendors.

      Cost to redevelop VB6 program-- about 1.6 million dollars.

      At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

      • At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.

        Larry Ellison? Is that you? That's been standard Oracle-speak for nearly a decade, now.

        I think you may have meant it sarcastically... but I think it's a good idea.

        How much benefit do you really derive from your uni

        • by Skal Tura (595728)

          Sometimes the differentiating business practices is your competitive edge. Infact, it many times is just that.

          I should know - we do loose some business because we cannot adapt the software to suit our needs (closed source portions/original design approach), and are forced to have our business to fit within the software specs. That sucks.

          For startups the standard software package might allow cheaper start up, but eventually there will be things which does not suit the business.

          Biggest competitive advantages

      • by CAIMLAS (41445)

        There is a solution for 1 through 4, and it's not one you'd like (unless you have a shotgun):

        Virtualization. Virtualization will give those ancient systems another stay on life, and you will be able to "phase them out" (not really) over the following decade, as other products start to get used.

        Virtualization turns your old crap systems into ever-living zombies. "This is old crap, we'll just virtualize it on new hardware". Doing this should last the system itself 1-2 hardware refreshes. By the time it's time

    • Maxo gave a great answer, but there are matters of internal compatibility also. Generally an IT app relies on lots of other IT apps to function, all of them are changing, and if YOUR app does not change you are retarding what can be done in other applications, or forcing maintenance of a backwards compatibility layer.

      There are some systems that can really just sit there and don't interface with much, really core systems... but then you run into that systems support issue and there is nothing corporate IT p

    • by c0lo (1497653)

      But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.

      Hmm... While theoretically you are right (the app doesn't change, the environ/data changed), it doesn't make the application run properly over time, the effort/cost to correct the application is still needed.

      Examples:
      a. application/system that don't function as expected anymore due to a security patch applied on the OS (damn;d if you don't apply the patch, damn'd if you do).
      b. applications/system that don't scale anymore with the number of concurent users
      c. applications/system with performance strongly a

    • by sjames (1099)

      It's not so much that the app rots, but it's support structure and fitness do. Requirements slowly grow until they exceed the app. The OS the app depends on gets a stream of updates until one of them introduces an incompatibility. People who actually know that app move on, demand for knowledge of that app declines and so less bother to learn it. Vendor EOLs the app or one of it's dependencies and support for it. Ancient install media gets damaged or lost.

      How quickly the lessons of Y2K are forgotten! Everyon

  • Everyone I've been talking to my field is telling me that corporations are spending like crazy on IT in the last two quarters, and are going to continue spending large amounts for at least the next year. There has been some slow down after the economy tanked, but from everything I've seen, the cash flow spigots are opening up.

    In my own experience I just got a new job six months ago and it has been non-stop, balls to the walls busy since I walked through the door. We're hiring new people and spending milli

  • by Script Cat (832717) on Wednesday January 05, 2011 @06:11PM (#34770394)
    Cost savings is the biggest expense to any large organization that does it.
  • I'm sensing many organizations may defer it until 2014. No special reason.
  • life of a sysadmin (Score:5, Interesting)

    by br00tus (528477) on Wednesday January 05, 2011 @06:22PM (#34770554)
    I have been a UNIX sysadmin for years, and have seen this at many companies. For a long time. Back in 1997 I worked at a company that had a lot of critical applications on an old, old Sun Netra. Lots of organizations have a few of these machines around - they are out of maintenance, the people who built it have left the company, it is out of warranty, and people generally don't touch them and hope they keep running. After years of meetings, presentations, budgets etc. we finally got a new machine shipped - and then a more politically connected department heard the machine had arrived, and had it stolen for their department on the day it arrived. I guess everyone thought it would be cheaper to page me in the middle of the night if the old machine failed and blame me the next day. The replacement machine being pulled is one of the main reasons I quit the company.

    I worked for another company that had a lot of money, but one thing we had to deal with was printing. Print jobs would come into our machines from strange places (IBM mainframe machines, from programs that were written 40 years ago) and go out to strange places (old dot matrix printers in a field office out in some obscure city in India). Thus I was sometimes left to puzzle why some program written in PL/I, coming from a mainframe which I don't have access to, is not printing to some ancient printer in Bangalore which is hooked to some ancient PC's parallel port.

    My former company from 2009 had some machines like this. Two very old Ultras running StoryServer and who knows what else. The StoryServer license had long fallen out of use, the machine firmware and Solaris OS had not been upgraded or patched for years. It sent e-mail through, for some reason, four Macintoshes. The Macs did not even run MacOS X, they were previous MacOS versions. E-mail starting with the letters A-F went to Mac1, G-M went through Mac2 etc., if a Mac crashed, mail to those letters would stop going through. The developers did not want to spend the time migrating to a new system, and I don't blame them, the oldest long-time developer there who dealt with such arcana was laid off, while the people building the latest new and shiny that the business wanted had the most secure jobs. Aside from this, we did not ever patch or upgrade our Red Hat Dell servers or firmware, we had no scheduled system downtimes etc. Our major Java application server had had its license run out. As I was leaving, the operations boss (soon to be fired) was considering not re-upping our Red Hat licenses.

    If a sysadmin goes on a job interview, and is not desperate, these are the types of questions they should ask, at least on the second round of interviews. Are all of the machines, OSs and applications I'll be responsible for under license? Are they all fully patched and upgraded for firmware, OS and application on a regular basis? What is the oldest machine still under responsibility - is it older than three years? Because all servers should be phased out every three years - at the very least. Try getting Dell/HP to support a 7 year old server decently. Also, do you have scheduled downtime once a week? Meaning do you have the option of rebooting and patching your main database machine, even if it is early Sunday morning? If they want 100% uptime it would necessitate paying for the infrastructure for high availability.

    Why should they spend the money when they can just call you in the middle of the night, to continue keeping it running with duct tape? Then they can blame you the next day after it broke. And you get no credit for it continually running either - the time you spend keeping it running is not counted, only time you devote to the latest shiny they want to implement. In fact, too much time devoted to keeping the machines they decided not to spend money on keeping up can cost you your job - if there's a choice between laying off the guy maintaining legacy stuff, and the guy who makes the new shiny for the business group and management and who deals with the

    • I think anyone who works in IT long enough comes to think of printing as the biggest waste of money in corporate America. How many forests have ended up as paper jams in a printer because a manager wanted to print his email.

      • by Lifyre (960576)

        So true. It gets even more fun when you're in an environment that is unfriendly the many small moving parts of a printer. Such as areas with large amounts of dust or sand in the air. We used to have printer parties . We'd take a bunch of old (mostly broken) printers out back and find various creative and entertaining ways to disassemle them so they would fit in the disposal barrels.

    • by CAIMLAS (41445)

      My GOD, man! Get out of my head!

      Thank you for your knowledgeable and reflective post. You made my a little warmer inside (maybe it was the whiskey?). It sure has been hell, lately, with archaic

      Note: the newest machines I've got are in that "3 year" ballpark. They're all critical infrastructure hosts. I can't get replacements.

  • I work in enterprise tech support, and finding a setup with unsupported hardware or code is the highlight of my day. No problem is easier for me to close than one where the customer has let equipment drop out of warranty and/or software support. Just did one today... box has been in continuous service (without poweroff) for six years; the model was discontinued in '05 and dropped out of support on 12/31/09. They have a difficult problem that likely would have taken me a day or two to solve... instead I m

  • ...'scrambling to keep systems up and running with budgets that barely cover the basics'...

    How is this different than it's ever been?

  • Thus

    Do more with less

    becomes

    Doing nothing with nothing

    as all systems fail and bankruptcy court takes your now worthless assets.

  • by GodfatherofSoul (174979) on Wednesday January 05, 2011 @08:22PM (#34771850)

    In the age of BS corporate leadership, who *doesn't* want to be the guy who cuts costs by 25%, gets promoted up into the suites, then lets one of his successors take the fall when the shit hits the fan? I'm more concerned with our public infrastructure BTW.

  • Geeks are handicapped in social areas, they have trouble advocating for bigger budgets, they get shafted by their bosses on funding, something goes wrong, they get blamed for it.

  • that a lot of businesses we work with every day control their own "fortunes".

    It seems like the companies that fall into this "duct tape" IT maintenance mentality are just buying time until a stroke occurs and the company essentially dies.

    And companies want me to believe that I should "trust" them with my data, my orders and my trust.

    Sheesh.

If you think the system is working, ask someone who's waiting for a prompt.

Working...