Deferred IT Maintenance Is a Ticking Time Bomb 186
snydeq writes "The underfunding of routine hardware replacement purchases and the degradation of aging enterprise apps pose systemic risk for many IT organizations, thanks to a ballooning 'deferred IT maintenance debt' in the decade since Y2K fears pushed enterprises to invest heavily in essential system upgrades, InfoWorld's Bill Snyder reports. And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud."
How is this newsworthy? It's just common sense. (Score:5, Insightful)
Deferring any maintenance can have calamitous effects.
I fail to see why this is newsworthy? Is it just because IT people whine louder?
If you are in the US---just look around. Infrastructure systems are crumbling away because of "deferred maintenance". It's not just IT. It's roads, bridges, state governments, municipalities, houses, businesses---it'severything!
Re: (Score:2)
InfoWorld needed some adclick revenue so they posted this completely duh story
Re:How is this newsworthy? It's just common sense. (Score:5, Interesting)
Well, it is sort of a "duh" story the way it is written, but OTOH the subject is not without merit.
I have been involved with infrastructure assessment of companies prior to acquisition and some stuff is just shocking. Publicly owned companies are driven by return to the shareholders; one way to keep the dividends flowing when the economy is in a downturn or when the business plan isn't working is to reduce operational expense.
Releasing employees is very effective to reduce the spend side but usually that means there is less available effort to work on maintenance. It looks good to have all employee time capitalized on projects but who is keeping stuff working? Also, each person out the door takes expertise with them that is lost to the company. After a while, the company may not even have enough knowledge internally to understand that their boat has holes in it and that patching isn't happening.
This isn't smoke; I've seen it. Data centers with overheating problems and with inadequate standby generators. Power is distributed unwittingly to cause a cascading failure if one breaker trips. Leaking roofs over financial servers (plastic tarp and bucket gave that away). Licensing that has not been kept up to date because no one has a good inventory and no one wants to look-see. So... Oracle enterprise instances running in non-secure network zones and without proper licensing ( potentially million$ in back costs). A database server being used as a network monitoring node and firewall because funds were not available to separate the functions.
Deferred infrastructure investment and maintenance investment happens and it is a ghastly mess to clean up. I am not surprised that more of this is happening.
Re: (Score:3)
Some admins are just scared of things going wrong during maintenance do don't do it. My boss at the last place I worked was like that. He never did updates on servers if he could possibly avoid it. The fear that the server might not reboot and he would have to drive all the way out and spend part of his weekend fixing it stopped a lot of preventative maintenance getting done. It was easier to blame users when the RDP server was hit with a 2 year old virus using a vulnerability that was long since patched. D
Re:How is this newsworthy? It's just common sense. (Score:5, Funny)
I think it's a setup for the "IT Industry Invaded by Incompetent Idiots" and "CIOs Found Replacing Working Systems with Crap Made By Their Hunting Buddies" articles.
A large portion of /. readers are in IT and already knew this. However, seeing it in "print" in a newsrag you might find in a CIOs office is a little noteworthy. It means it's only a matter of time before someone comes rushing to your desk to say "Our CIO just read an article about infrastructure and we need an ans..."
Hang on, someone's at my desk.
Re: (Score:3, Interesting)
I need to make some friends in the news media.
Re: (Score:3)
Careful what you wish for... the CIO at the Fortune 100 company I just left still thinks DLink routers without redundancy are the way to go. He still approves purchases for replacements and new ones.
You could be having fun with that.
Re: (Score:3)
Sometimes cheap routers are the way to go -- I replaced 2 Cisco routers at a remote site in a seaside warehouse (one due to a power surge when a generator failed, one due to water from a leaky roof) before switching to cheap Netgear routers at about 1/10 the cost. Redundancy? We had a spare configured and ready to go the foreman's truck toolbox and another at his house.
One of the Netgears even survived a similar water deluge to the one that took out the Cisco (but then the Netgear didn't have a fan to suck
Re: (Score:2)
Re: (Score:2)
Sure, the rule may have been bogus, but we had to obey nonetheless. That's one of the joys of dealing with a government bureaucracy where the guy who manages the facility gets to interpret the rules no matter how capricious and arbitrary they may appear. Things were a lot smoother at that facility after we found out that the harbor master liked jelly donuts.
Re: (Score:2)
Things were a lot smoother at that facility after we found out that the harbor master liked jelly donuts.
I have to wonder why this wasn't your first strategy.
Or, am I the only one where the phrase "harbor master" conjures up an image of someone like Homer Simpson or Louie DePalma?
Re: (Score:2)
I have to wonder why this wasn't your first strategy.
New York management. It wasn't until they hired a local guy to manage the facility that he was able to get things done.
Re: (Score:2)
Re: (Score:2)
I can't say I particularly care for DLink, but for the cost of a (used) Cisco, I can pick up two Procurves with better specs and a management interface which anyone with basic networking comprehension can master in a couple hours. I'm not hearing anyone say the Procurves are shit (because they aren't). With d-link, I'm sure the same is true - and I can have a stack of hot spares for the same cost, too - Just In Case.
Re: (Score:2)
D-Link routers are just fine...
FOR ME TO POOP ON!
FTFY
Re: (Score:2)
He already said he was using AT&T. Your fix was redundant.
Re:How is this newsworthy? It's just common sense. (Score:4, Insightful)
Slow computers with styles that were pretty neato back in 2000 are obvious to the poor office drones who have to endure them; but anything that new can, largely, be forklift upgraded for the cost of the new systems and some grunt labor. Turning a 3 year desktop refresh cycle into a 5 year(or 7 year, *cough* *cough*) desktop refresh cycle doesn't make anybody happy(particularly once warranties run out, the scavenging and improvising begins); but is architecturally a small problem. You don't really accrue much "debt" over time. The cost will be "1 forklift upgrade to present day PCs" whether that upgrade takes you one generation ahead or three.
It's the complex software, the highly specialized proprietary industrial controller cards, and suchlike widgetry where there is real hell to pay, and most of that is invisible...
Re:How is this newsworthy? It's just common sense. (Score:5, Insightful)
Depends on the level of "bespoke" in your house.
Scavenging for desktop parts is the "little devil". Scavenging for people who know how the bloody things work more than 3 years after for IT systems is the real nightmare.
If a system has been in the field for 3+ year nobody knows what are its real dependencies and what does it really take to augment, add capacity or do any changes. The people who knew have left, gone to pastures new or have forgotten what the problems used to be and no documentation can help you here (even if there is any suriviving docs on the design of the system in question). This is valid for almost all classes of IT and telecoms systems and is the real cost factor in IT "maintenance debt". If we use a real-life analogy IT maintenance debt is like a discounted mortgage. You pay virtually nothing for 2-3 years and after that the lender skins your hide.
Re: (Score:3)
Re: (Score:3)
Hey now, don't know Y2K vintage hardware. Place I worked spent a lot on some beige and white Dell workstations that year. They were spiffy at the time. The boss/owner was ultra proud of his Dells and made sure they sat on TOP of the desks so clients and whatnot would see them and be impressed. Woooh he spent money on Dell.
Boring beige Dell was actually an improvement. Prior to this, the place was run off an old infected Vision PC they got from a radio ad. I upgraded them with almost no budget to DIY s
Re: (Score:2)
Re: (Score:2)
Comment removed (Score:5, Insightful)
Re: (Score:2)
I'm pretty sure you meant to say "CNC"
http://en.wikipedia.org/wiki/Cnc [wikipedia.org]
Re:How is this newsworthy? It's just common sense. (Score:4, Funny)
You want to watch a "real" programmer wet his pants in fear you hand him a huge 14 page VB mess written by a half a dozen guys over the years ...
Do you want to see real bedwetting? If you're most anywhere in America, your healthcare depends on a few gigabytes of VB6. That it works speaks to the value of good development practices.
Re:How is this newsworthy? It's just common sense. (Score:4, Insightful)
Silly young grasshoppah. There is no "just do..." in IT. The mythical solution you're referring to is a cruel joke told by vendors.
huge 14 page
So, a little script, then?
Something on my table right now: 15k (in-file - and probably significantly more on the printed page) lines of PHP3 with nasty embedded SQL up the wazoo. It ties into half a dozen (literally, 6) other 'mission critical' applications and is customer facing as well as providing internal network management functionality. And this is small fry compared to some of the crap out there.
You want to know why there is an ever increasing IT debt I'd say that is a BIG part of it. All across the country you have this huge mess of apps written by some Joe Schmo that was bought ages ago and nobody knows how to live without and it DON'T run on anything but what it was written for and even then it is fussy as hell
I couldn't agree more. We've had entirely too many Boy Geniuses in decision making places in IT who think they've got something special and unique which will have Totally Awesome Results. They don't bother to think through their decisions.
The proper approach to something like this isn't to fix it. It's to replace it outright with something that does 90% of the task, better, with 50% fewer inter-dependencies by modularizing things as much as possible. Re-implementing, feature for feature, is quite often quicker. Just make sure you don't make the mistake of so many before you and re-implement it - poorly. If you can't do it, find someone who can.
IMO, the key to a successfully maintainable software infrastructure is to KISS and leave things as White Box as possible. When you can't keep things generic, you keep things isolated and modular. When you need something custom purpose that your users rely upon, you make damn sure it's standards based and that there are alternatives available.
(I don't even want to THINK about where we will be with things like Sharepoint in 3-5 years. Likely, another lengthy, drawn out, and costly migration project. This time, maybe back to something like, oh, NFSv5.)
Comment removed (Score:5, Insightful)
Re: (Score:3)
Ya, well I'm the Joe Schmo that gets tasked to fix this sort of stuff.
The conversation usually goes like this:
Me: "To fix this properly you will need to do this, buy this software, redesign this system, it will take X amount of dollars and Y amount of time."
Boss: "We don't have the money. You need to fix it as best as possible using the materials at hand with no budget, and a limited time."
Thus, slaps together whatever I have to get it working, cheaply and likely crudely. Forget documentation, you don't hav
Re:How is this newsworthy? It's just common sense. (Score:4, Interesting)
Because much of this IT is stuff that affects individuals who have no influence over it.
When a company puts off investing in security, for example, and when they also collect and store my credit card info / medical info / personal demographics / shopping history / etc., they are putting me at risk.
I have to trust that their IT department is on the ball. Something I am beginning to think is never a good idea. But it's impossible to not give companies some info on me and still be a normal modern human, and thus I am forced to trust them all the time.
So if they're further neglecting their IT, it means my data is more vulnerable. Not that's there's a damn thing I can do about it.
Many people don't understand it. (Score:2)
Yes, it is "common sense" to you.
But how many "IT managers" understand that part of their job involves pruning? Killing old systems. Deprecating other systems. Weeding out the "one-offs" that pop up when no one is looking.
The last place I worked has 7 different database servers. Because they were running 7 different database platforms.
Virtualization means that you can reduce the floor space needed. But management also needs to look at reducing the systems needed.
How about weeding out enterprise standards? (Score:3, Insightful)
Just one minor gripe with the parent - a lot of times, what should be weeded out isn't the "one-offs" (which are often times built way under budget with way more capacity and way less maintenance cost), but the actual official enterprise standard that got put in because some CIO was buddies with some sales rep. "One-offs" are a signal that the current standards (either of technology, or product development), are having problems. While not all "one-offs" may be worthy of keeping, when going through the wee
Re: (Score:3)
I think the one-off's he's talking about are things like the purchase tracking system that Finance installed on a workstation under someone's desk. IT was not aware of it being purchased and of course it has no way to export data into any of our other systems so we can integrate it with the rest of our financial systems.
Eventually we sucked it into our VMWare infrastructure to get it off of the desktop machine (no RAID disks, and the backup consisted of someone in accounting copying the database to a flash
Re: (Score:2)
You think that's bad. A former organisation I worked at had a similar thing happen where the Finance group bought a software package which was sold on "no IT department required" however the way we found out was even before they deployed it. It was of course a computer program so instead of the Finance department using their own budget allocation, the bought it out of the IT departments budget. When the CIO and the CFO had a meeting the shouting could be heard from the other end of the building.
Suffice to s
Re: (Score:2)
Re: (Score:2)
Touche, but to be fair that doesn't mean we shouldn't still inspect our bridges.
Re: (Score:2)
Deferring any maintenance can have calamitous effects.
I fail to see why this is newsworthy?
Probably the estimated amount of money it would be needed to catch-up with the backlog (the so called "IT debt") and a few words on how the estimation has been made? Also, where these money would be needed? TFA:
$500 billion -- it's a number so big you'd assume it's a component of the national debt. It isn't. Instead, it's what Gartner analyst Andy Kyte calls the IT debt. "
The "debt" really has two major components: One is underfunding and even neglect of routine but important hardware replacement purchases and software upgrades. The other is the slow degradation of enterprise applications.
Is that $500 billion number too high? Kyte says he derived it by analyzing several large Gartner clients that generally do a good job of keeping applications up to date. That led him to estimate that a typical Fortune 2000 company would require upgrades costing more than $200 million each.
I admit, these as scraps of "meat", but they are nevertheless still meat.
I imagine that this may be the first step (as an argument) in a push the big IT houses (hardware/software/services) will make to get so more revenue... Something on the line of "well... Apple and Android tablets and phones (that
Re: (Score:2)
I'm guessing it has something to do with the fact that IT managers read InfoWorld. Some read slashdot, too. Many of them are dropping the ball on this, big time. "We don't have the money this quarter" comes up time and time again, and they don't realize that the decreased maintenance is going
The only people who really realize this kind of thing are sysadmins. Unless the sysadmins have a huge amount of power within the organization (including their own ability to prioritize the budget and 'vision' of the org
Re: (Score:2)
Now how is pushing services to a cloud exsaserbating the problem? Sure going to the cloud has it pros and conns. But if you are retiring your old servers to a cloud in terms you are Upgrading your infrastructure and removing your old stuff.
How does that work again? (Score:2, Insightful)
And with sysadmins 'scrambling to keep systems up and running with budgets that barely cover the basics,' this 'IT debt' promises only to increase in the coming years, especially as IT continues to defer routine maintenance in favor of new 'cost-saving' initiatives, particularly around the cloud.
The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things. Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem. YMMV, of course.
Re:How does that work again? (Score:5, Insightful)
CIOs and organizations blissfully march towards disaster while quietly chanting to themselves, "The Cloud will save us all".
Re: (Score:3)
Don't blame the cloud... If it weren't around they would simply chant about outsourcing, virtualization, or right-sizing whilst marching to their doom.
Re: (Score:2)
Re: (Score:3)
No one gets my jokes around here.
Re: (Score:2)
Re: (Score:3)
Theoretically, taking advantage of the cloud where it fits your organization will offset the "maintenance debt" problem.
"Cloud" (as in, dynamic server provisioning) has very little to do with it.
Outsourcing IT functions to a firm that is contracted to actually perform the maintenance that was being deferred on the in-house systems (whether hardware, infrastructure software, application software, etc.) obviously can address problems related to deferred maintenance, not because of the outsourcing itself, or because the vendor to whom the operations are outsourced happens to use "cloud" technology to power its offerings, but be
Re:How does that work again? (Score:4, Insightful)
"The point of using "the cloud" (a hollow buzzword, I admit) is that you can offload the servers, software, and maintenance to a firm that specializes in such things."
Yes, because it's a demonstrated hard fact that those companies providing infrastructure for the cloud can't lower their operational costs by neglecting maintenance; of course they wouldn't do that anyway since it's those infrastructure companies' very valuable data what is at risk if maintenance is neglected instead of their customers'.
Oh, wait!
It's a governance issue - plan and simple (Score:5, Insightful)
I routinely see job ads for experienced Java developers, people with hard core experience in integration, esp. with telephony or security technologies, need 5-10 good years, offering $70k tops. Good luck with that, but again it is the CIOs who get the jobs telling people they can staff cheaper, run leaner, cut the corners - that get the job because it is the CFO who is doing the hiring and the performance reviews.
The big corporation IT C-level execs are a fear driven lot, there are no Gates or Zuckerburgs in their midsts. The action is being with the cloud providers, or the web service providers themselves. Enterprise IT is really a shit place to be outside China. It's a world full of EDS consultants and chickenshit CIOs who won't think how a business could use IT to expand. And the social media space is going to tear a bunch of them new assholes, because none of them know how to leverage it. The startups do.
Re:It's a governance issue - plan and simple (Score:5, Informative)
Nail, head hit. I have worked for people who had bosses who had zero interest in anything other than cost. The same people that I lambast for putting basic security precautions as an extreme low priority, due to their attitude of "security has no ROI."
What is ironic is that even though this makes the quarterly figures look good, it is killing competitiveness long term. Another example is R&D. Not product research, but true R&D where people discover something, shrug, put it on a shelf for 20 years, then it becomes immensely marketable (think Corning and Gorilla Glass as the prime example of this.) Instead any groundbreaking research ends up being done overseas or by small firms which are bought out, or have their IP infringed upon. If research is done, it is for a product to cough up this quarter or perhaps this FI, and usually it is how to add a gewgaw to something existing and palm that off on the market.
You mention China. Chinese companies know how to lock their doors down. They know what happens if you run your company with your fly open, and most of the companies over there wouldn't even be in business had it not been for "borrowed" IP from the West. Take Foxconn as an example. For a company that size making so many Apple products (including ramping up production on unannounced items), they are quite airtight about what their factories produce even with the hordes of workers they have. Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.
I wish the US would start borrowing from China in this regard. Even with security aside, just because a company's IT infrastructure works today doesn't mean it will work 5 years, or perhaps even six months from now without major issues. Taking a charge off quarterly earnings to fix problems now means a lot of less wasted money when the upgrades have to be done post-haste.
Re: (Score:2)
"I wish the US would start borrowing from China in this regard." I wish you were right. However, if an American firm did this, they'd be sued by Chinese companies in American courts using American lawyers...and their Imaginary Property would be protected and the American firm set up for serious damages. And the Chinese know this.
Re: (Score:2)
To be fair, on the normal USA business timescale (that is, the next two or three quarters), it really doesn't. Security only matters if you're still heading the company when something bad happens, which to most of these people is pretty much unthinkable - because they know they'll be gone in a few years.
Re: (Score:2)
Had this been a company run by the typical PHB here in the US (with their usual lack of interest in security), everyone and their brother would know what the iPhone 5 looks like, and perhaps even have the source code for that rendition of iOS.
I wish the US would start borrowing from China in this regard.
Give them a few years, they will. It'll start to happen when the first PHB demands evidence that a chinese factory has had no more than three information leaks in the previous 2 years, and they'll get a very bemused email back asking them what sort of information leaks would they like engineered because that's the only way anything's getting out.
Re: (Score:3)
Not only is the nail on the head, it is worse. The CIO reports to the CFO but doesn't actually know any of the terms that the CFO uses, so all they have are "saved X by not doing Y". A REAL CIO would have the CFO terms down and be able to say "by doing T we can do S things more productively".
I remember when this hit me about a dozen or so years ago, when I was the business manager for a small regional ISP (when they still existed), and had to process some logs for billing purposes. I had one computer that I
Re: (Score:2)
My point got lost here. The point is that Chinese companies actually seem to devote resources to computer security and IT infrastructure because they are seeing the mistakes companies are making on the other side of the Pacific. Even the Chinese government is laying fiber like there is no tomorrow.
Here in the US, we don't need corporate dorms -- we escaped that stuff in the Gilded age (but with the political climate, we may be heading back towards that direction.) Instead, more than just a token effort i
Re:It's a governance issue - plan and simple (Score:5, Informative)
As it is there's very little reason for corporate executives to think down the road by more than a couple quarters, as chances are the people who own shares now won't still own any much beyond that.
Re: (Score:2)
Too many CIOs of too many western corporations report to the CFO, not the CEO. There are WAY too many CIOs who come into organizations with an eye, or a reputation, for cost cutting instead of tech innovation.
Assuming the corporation is not in the IT business but only uses IT, I thing that's only fair to use risk management as the second term of the contrast (instead of tech innovation).
Re: (Score:3)
Re:It's a governance issue - plan and simple (Score:4, Insightful)
It's amazing how right this is. The problem also stems from companies that could or should be using IT as a way to improve their core competencies and improve their competitive position aren't because of the recent economic issues. Many people are getting power and influence by riding the penny pinching wave instead of making good long term decisions. We're going to be facing the aftermath of having these people over promoted for a long time to come.
Re: (Score:2)
Companies which have been improving their services and products through better IT practices are actually not doing too poorly. And no, I'm not wholely talking about IT industry companies, either: I know of manufacturing, architectural, and a handful of general 'service' companies (including banks) which have significantly improved their IT utility, both internally and as they face the customers.
No, XP isn't off the desktops at many of these places. (They are at the architectural place.) But many of the mach
Re:Low cost strategy (Score:2)
Low cost needs to be balanced against getting the job done, and reliably done.
Because if your IT starts to have frequent outages or lose valuable data, it can be more expensive than investing in decent equipment and competent employees. /. likes car analogies:
Since
In the 90s Opel, a German branch of General motors, was a bit too aggressive in cutting manufacturing costs. The resulting quality problems were quite damaging to the brand and customers started to look elsewhere for their next car.
Maybe it's ok.. (Score:2)
Maybe the cloud will mean less in-house IT stuff, which means the IT debt won't even need to be paid off.
Now if Amazon or Microsoft is putting off the IT work for their cloud systems, that might.. be a problem..
Re: (Score:2)
One way or another, it's going to be paid. Either buy and deploy internally or pay an outsourcer to buy and deploy. Meanwhile, you can't just export re-writing your old creeky VB6 apps to the cloud. Even if you find a cloud vendor offering a suitable replacement app, you'll have to pay for that and pay to have your vital business data pried from the cold dead fingers of your old app and imported to the new app.
It may come out a bit cheaper having the cloud provider do it, but be sure of the business arrange
on the other hand (Score:2)
The price of replacing things may be getting cheaper faster than the implied cost of the risk of not replacing it is going up. It may be cheaper to wait until it breaks than to buy something with a rapidly depreciating value. The extra cost of dealing with an emergency may be paid for by the lower cost of the gear and labor. And for those risk events that never happen, the ratio of preventative replacement cost to emergency replacement cost will be infinite.
Only your CIO knows for sure. But I bet he's p
*yawn* (Score:2)
In other news: The sky is blue!
Re: (Score:2)
In other news: The sky is $500 billions-worth of blue!
FTFY
I read it like this: (Score:2)
The longer you leave it, the cheaper it gets (Score:2)
Have you ever? (Score:2)
...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?
I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part. From HP it was OMFG-are-we-really-spending-this-kind-of-money expensive, a "re-certified" part from a third party was still way more than the $300 you'd spend for your home system as the part and its corresponding VRM were proprietary.
And the only reason it was done at all
Re: (Score:2)
...worked anywhere where someone actually swapped CPUs in a server from a real vendor (ie, not some BS whitebox)?
I *added* a CPU to an HP server once (single to dual CPUs) and it was super expensive and not all that easy to get the part..
I did and while the parts from HP were outrageously expensive we ordered it from a third party. HP does not have an exclusive market on Xeon chips or the plug-in voltage regulators they used to use (maybe still do?). The tricky part was the heatsink which needed to be a funky size to squeeze in the space allotted but that too was solved from a third party vendor. Total cost was about a third what HP wanted. We did it as a test case to make a single CPU machine with 6 total slots into a dual. It was even
Re: (Score:2)
Re: (Score:2)
If they can give their kit a mid-life kicker with some more memory or swapping in some faster CPUs,
That's just stupid. Simply "dropping in" a faster CPU doesn't do all that much, because you keep the same bus speeds on the motherboard that limit bandwidth. As for memory, haven't you noticed that buying older, slower memory tends to be more expensive than buying the new faster memory, because the old stuff is no longer in production?
Aside from those factors, there's the added labor in installing those components and testing them. On top of all that, most mass-market desktops that are used in this space ar
Re: (Score:2)
I feel the pain...in state Gov... (Score:2)
Re: (Score:2)
I see replacing equipment as less of issue than purchasing new equipment for additional capacity. Assuming you are doing backups, you can always move hard drives from old-system to a new-system. Yeah that is some work to get it back in running, but it is still a possibility.
However, if you have servers running at capacity, and they refuse to purchase new equipment, you are asking for problems.
Re: (Score:3)
"Assuming you are doing backups"
And that's exactly the problem, my friend. Too many people round here seem to imply that "deferred it maintenance" means not replacing servers when the guarantee period ends up. But maintenance means having two sysadmins when you formerly had three or maintaining the three sysadmins when capacity has grown 50%.
Lowering maintenance costs means that it has been a year that nobody has the time for a test restore so nobody has noticed that the backups are failing since six mont
Re:I feel the pain...in state Gov... (Score:5, Informative)
I am having trouble getting basic hardware replaced - I can't get a 500-750 dollars to replace some network switches let alone enough scratch to update my primary DC. Our Budget Analyst does not see the need to plan for future needs, or periodic replacement of vital equipment as warranty cycles expire. I have documented our needs, but my boss the CIO is afraid to push the issue.
My advice: add some risk analysis argumentation. You know? Something on the line of:
1. probability of equipment failure [omdec.com] over time - use the "cumulative hazard function" not the "probability distribution function".
2. impact the server crash will have on the business (make sure you slip-in some "lost face" apropos - after all it would be the manager's face to be lost). If you can express the impact in $$$ and plot the "risk x impact", chances are the budget analyst will "get the picture" easier.
Re: (Score:2)
Forget the risk analysis arguments. If people refuse to see simple logic, hitting them with more of it will do no good. These people will dismiss your reasoning with superstition, thinking along the lines of, "My VCR is 10 years old. It works fine. I see no reason to upgrade our network equipment until my vcr dies."
For things like that, the only reasonable solution is a loose mains wire and some bridging rod to the offending equipment in question. Then, when you get your new equipment overnighted at 20% mar
It's such a Ticking Time Bomb that... (Score:2)
~Sticky
degradation of aging enterprise apps? (Score:3, Insightful)
How, exactly, do enterprise apps degrade?
Do they suffer from bit-rot, and have some kind of half-life?
I understand that eventually apps will fail to be supported by the developers, won't potentially work on more modern operating systems, and in some cases require updating in order to be able to work correctly with the rest of the world.
But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.
Re:degradation of aging enterprise apps? (Score:5, Interesting)
1) The languages, special hardware, libraries and controls become unsupported on new hardware.
2) The languages, special hardware, libraries and controls become unsupported on new versions of the operating system.
3) The operating system becomes unsupported.
4) The hardware becomes unsupported.
Example: VB6 program uses bar code scanners.
2004? VB6 unsupported as a language.
2008, VB6 unsupported for security patches (so any required security patch could kill VB6 program)
2009 bar code scanners unsupported (change to optical recognition with new software interface)
2009 VB6 Outlook/Word integration fails.
2010 Hardware and operating systems to support VB6 start becoming unavailable. All are unsupported by vendors.
Cost to redevelop VB6 program-- about 1.6 million dollars.
At some point- basically find a new packaged product (cost $100k + $500k user licensing & support + loss of ability to differentiate business) which provides 80% of functionality of the VB6 program and toss it. Can't be changed to match your business - you must change business to match it.
Re: (Score:3)
Larry Ellison? Is that you? That's been standard Oracle-speak for nearly a decade, now.
I think you may have meant it sarcastically... but I think it's a good idea.
How much benefit do you really derive from your uni
Re: (Score:2)
Sometimes the differentiating business practices is your competitive edge. Infact, it many times is just that.
I should know - we do loose some business because we cannot adapt the software to suit our needs (closed source portions/original design approach), and are forced to have our business to fit within the software specs. That sucks.
For startups the standard software package might allow cheaper start up, but eventually there will be things which does not suit the business.
Biggest competitive advantages
Re: (Score:2)
There is a solution for 1 through 4, and it's not one you'd like (unless you have a shotgun):
Virtualization. Virtualization will give those ancient systems another stay on life, and you will be able to "phase them out" (not really) over the following decade, as other products start to get used.
Virtualization turns your old crap systems into ever-living zombies. "This is old crap, we'll just virtualize it on new hardware". Doing this should last the system itself 1-2 hardware refreshes. By the time it's time
Connections (Score:2)
Maxo gave a great answer, but there are matters of internal compatibility also. Generally an IT app relies on lots of other IT apps to function, all of them are changing, and if YOUR app does not change you are retarding what can be done in other applications, or forcing maintenance of a backwards compatibility layer.
There are some systems that can really just sit there and don't interface with much, really core systems... but then you run into that systems support issue and there is nothing corporate IT p
Re: (Score:2)
But it's a bit disingenuous to call this "degradation". The app continues to do what it always did. You're just wanting more out of it than you did before. The app didn't change, you did.
Hmm... While theoretically you are right (the app doesn't change, the environ/data changed), it doesn't make the application run properly over time, the effort/cost to correct the application is still needed.
Examples:
a. application/system that don't function as expected anymore due to a security patch applied on the OS (damn;d if you don't apply the patch, damn'd if you do).
b. applications/system that don't scale anymore with the number of concurent users
c. applications/system with performance strongly a
Re: (Score:2)
It's not so much that the app rots, but it's support structure and fitness do. Requirements slowly grow until they exceed the app. The OS the app depends on gets a stream of updates until one of them introduces an incompatibility. People who actually know that app move on, demand for knowledge of that app declines and so less bother to learn it. Vendor EOLs the app or one of it's dependencies and support for it. Ancient install media gets damaged or lost.
How quickly the lessons of Y2K are forgotten! Everyon
Load of crap (Score:2)
Everyone I've been talking to my field is telling me that corporations are spending like crazy on IT in the last two quarters, and are going to continue spending large amounts for at least the next year. There has been some slow down after the economy tanked, but from everything I've seen, the cash flow spigots are opening up.
In my own experience I just got a new job six months ago and it has been non-stop, balls to the walls busy since I walked through the door. We're hiring new people and spending milli
Cost savings is really expensive. (Score:4, Insightful)
hmm (Score:2)
life of a sysadmin (Score:5, Interesting)
I worked for another company that had a lot of money, but one thing we had to deal with was printing. Print jobs would come into our machines from strange places (IBM mainframe machines, from programs that were written 40 years ago) and go out to strange places (old dot matrix printers in a field office out in some obscure city in India). Thus I was sometimes left to puzzle why some program written in PL/I, coming from a mainframe which I don't have access to, is not printing to some ancient printer in Bangalore which is hooked to some ancient PC's parallel port.
My former company from 2009 had some machines like this. Two very old Ultras running StoryServer and who knows what else. The StoryServer license had long fallen out of use, the machine firmware and Solaris OS had not been upgraded or patched for years. It sent e-mail through, for some reason, four Macintoshes. The Macs did not even run MacOS X, they were previous MacOS versions. E-mail starting with the letters A-F went to Mac1, G-M went through Mac2 etc., if a Mac crashed, mail to those letters would stop going through. The developers did not want to spend the time migrating to a new system, and I don't blame them, the oldest long-time developer there who dealt with such arcana was laid off, while the people building the latest new and shiny that the business wanted had the most secure jobs. Aside from this, we did not ever patch or upgrade our Red Hat Dell servers or firmware, we had no scheduled system downtimes etc. Our major Java application server had had its license run out. As I was leaving, the operations boss (soon to be fired) was considering not re-upping our Red Hat licenses.
If a sysadmin goes on a job interview, and is not desperate, these are the types of questions they should ask, at least on the second round of interviews. Are all of the machines, OSs and applications I'll be responsible for under license? Are they all fully patched and upgraded for firmware, OS and application on a regular basis? What is the oldest machine still under responsibility - is it older than three years? Because all servers should be phased out every three years - at the very least. Try getting Dell/HP to support a 7 year old server decently. Also, do you have scheduled downtime once a week? Meaning do you have the option of rebooting and patching your main database machine, even if it is early Sunday morning? If they want 100% uptime it would necessitate paying for the infrastructure for high availability.
Why should they spend the money when they can just call you in the middle of the night, to continue keeping it running with duct tape? Then they can blame you the next day after it broke. And you get no credit for it continually running either - the time you spend keeping it running is not counted, only time you devote to the latest shiny they want to implement. In fact, too much time devoted to keeping the machines they decided not to spend money on keeping up can cost you your job - if there's a choice between laying off the guy maintaining legacy stuff, and the guy who makes the new shiny for the business group and management and who deals with the
Printing, the bane of every IT worker (Score:2)
I think anyone who works in IT long enough comes to think of printing as the biggest waste of money in corporate America. How many forests have ended up as paper jams in a printer because a manager wanted to print his email.
Re: (Score:2)
So true. It gets even more fun when you're in an environment that is unfriendly the many small moving parts of a printer. Such as areas with large amounts of dust or sand in the air. We used to have printer parties . We'd take a bunch of old (mostly broken) printers out back and find various creative and entertaining ways to disassemle them so they would fit in the disposal barrels.
Re: (Score:2)
My GOD, man! Get out of my head!
Thank you for your knowledgeable and reflective post. You made my a little warmer inside (maybe it was the whiskey?). It sure has been hell, lately, with archaic
Note: the newest machines I've got are in that "3 year" ballpark. They're all critical infrastructure hosts. I can't get replacements.
I love outdated equipment and code (Score:2)
I work in enterprise tech support, and finding a setup with unsupported hardware or code is the highlight of my day. No problem is easier for me to close than one where the customer has let equipment drop out of warranty and/or software support. Just did one today... box has been in continuous service (without poweroff) for six years; the model was discontinued in '05 and dropped out of support on 12/31/09. They have a difficult problem that likely would have taken me a day or two to solve... instead I m
I'm confused. (Score:2)
...'scrambling to keep systems up and running with budgets that barely cover the basics'...
How is this different than it's ever been?
Debunking the new adage (Score:2)
Do more with less
becomes
Doing nothing with nothing
as all systems fail and bankruptcy court takes your now worthless assets.
Same holds true for *all* maintenance (Score:4, Insightful)
In the age of BS corporate leadership, who *doesn't* want to be the guy who cuts costs by 25%, gets promoted up into the suites, then lets one of his successors take the fall when the shit hits the fan? I'm more concerned with our public infrastructure BTW.
Simple (Score:2)
Geeks are handicapped in social areas, they have trouble advocating for bigger budgets, they get shafted by their bosses on funding, something goes wrong, they get blamed for it.
Slashdot Writers Don't Give Me Hope... (Score:2)
that a lot of businesses we work with every day control their own "fortunes".
It seems like the companies that fall into this "duct tape" IT maintenance mentality are just buying time until a stroke occurs and the company essentially dies.
And companies want me to believe that I should "trust" them with my data, my orders and my trust.
Sheesh.
Re: (Score:2)
Just because iMacs made better aquariums than computers doesn't mean all Macs have that problem. A few of them actually have uses and the new MacBooks make really good stands for my iPad.
Re: (Score:2)