How Overhauling IT Was a Life-Saver For the American Cancer Society 51
Lemeowski writes: American Cancer Society CIO Jay Ferro lets readers peek behind the curtain of the nonprofit's IT organization, saying that when he took on the role a little over three years ago, the nonprofit had 12 different divisions — each with its own independent IT set-up and more than 600 independent applications in its portfolio. In the past three years, Ferro has aligned the entire IT organization into one global entity, consolidating dozens of data centers into three; increasing spending on strategic projects from 5 percent to 40 percent, and reducing 600 core systems down to fewer than 200. His journey is a powerful reminder that while streamlining IT can often be painful upfront for IT managers, the payoff for sticking with it, especially for nonprofits, can feed into saving more lives.
First Post with good info (Score:5, Interesting)
IT leadership is lacking in most companies.
The idea a good manager can manage anything is wrong.
After many years, I had a friend in manager ask me a question or two outside the office.
He had been in many management roles, and was working as a project manager for some major Oracle DB projects.
After explaining and doing a short lesson on all the major tech terms the DBs & system admin use.
I asked what does management understand.
On time (or not), Manpower Head count & budget.
They almost never have a clue about the interaction of the systems, quality, maint, network (or any infrastructure) & replacement (timing)
One manager I worked for needed a powerpoint to explain why upgrading the memory in the main unix server was needed,
after all we had just install 8gb in his PC.
Buy another tape cabinet with every project - BOFH (?)
Re:First Post with good info (Score:4, Insightful)
Often they're looking for career progression, and not caring about the software development changes or support changes, but more about the "is it done yet?" - meaning, forget about it being right, refactored, consolidated, any kind of documentation, etc, and testing is assumed that the moment code is up and running, that it's fixed. So they ask you "how you're getting on" - and the only thing word they hear is "working", and they report that. Then when they find out it isn't, they immediately pile on the pressure. "you told me it was working!" - "you better get it done, quickly!, and get it in!" (meaning source control and released) so as to cover up their lack of understanding.
I've actually confronted a number of project managers, and their managers on such issues - even taking it as high as contacting board members through email (risking my contract!) to tell them they've got a communication problem. Once they hear this, I'm shuffled into an office for a long chat, where they say they're aware of, and working towards improvement, and it just so happens the software developers are always the last department to see these changes filtering in from "above".
Once I suggested they go and visit a place I previously contracted at, as they were actually a supplier - making some excuse about a project they could work on jointly, and learn the way they develop, thus, bringing their style of project management in house.
I ended up with a previous work colleague contacting me asking who the hell xxxx was, and so it seemed they may actually of listened to me. Whether that brought about positive change, I don't know. But the moral of the story is, managers love to talk, and pore over financial figures, but don't hear you when you talk about technical debt, need for refactoring, or consolidation, or offer new ways to do things that could prove beneficial. They only hear "risk". - and managing that doesn't seem appealing when they don't know that you're talking about.
Re: (Score:2)
It's not just management to blame. Every mid-level peon who is attempting to play the game instead of focusing on delivering value for their customer is subverting the goals. When Manager Mike asks Joe Lineworker for a specific result that's conducive to his politics, Joe can say "sure, I'll scratch your back, just don't forget I did this for you." It's corrosive in both directions.
What happens in this case is the outcome is as the manager inflates his project, the organization is wasting money on a proj
Re: (Score:2)
If management sets up incentives to game the system they have no one to blame but themselves. Of course employees will do what the system is setup to reward. Duh.
Re: (Score:2)
If management sets up incentives to game the system they have no one to blame but themselves. Of course employees will do what the system is setup to reward. Duh.
And that's my point. A non-engineer manager won't know what their engineering employees do, so they won't incentivize the right behaviors.
Manager - Systems Architect - Project Manager (Score:3)
The problem is there is a hierarchy structure while for IT projects the Manager, Systems Architect and Project Manager really need to be on the same level, and not try to take over each other's roles.
The Manager: On Time, Manpower, and Budget is a lot more work than it sounds, doing such work means you cannot really focus Technical Details, even if you take a good tech and give him a manager job, he will either make bad tech decisions, xor fail to manage effectively. The same with having the manager trying
Re: (Score:1)
I like having a manager and I can understand the Systems Architect role. As a programmer, I view the Project Manager role as an unnecessary cost in IT. The project manager in IT isn't the same as one in construction and they often don't minimize communication issues. If I have to follow the project manager into every meeting, then they're not shielding me from that communication. And if they can only pass messages back and forth without actually creating a project plan (independent from just asking the
Re: (Score:2)
"You don't have to know the ins and outs of router configuration, or maintaining Exchange servers to understand who in your organization knows what and to consult them on technical matters"
Kinda, but if you don't have ( or get ) the beginnings of a clue, you wont be any good at knowing who in your org *really* knows what.
Then you will consult the wrong people, and manage your way to somewhere between mediocrity and disaster.
In tech, there will usually be some "glad hands" around who kinda know, then the ant
Re:First Post with good info (Score:4, Insightful)
Yep, I had a very interesting talk with my wife's friend.
She's an accountant and she was just complaining about IT.
So I talked to her about our side, and it's amazing the disconnect there.
We talked and I mentioned how every IT project needs a maintenance budget. You need knowledge retention in case the service needs to be updated in the future. You need someone to support it... You know, just like any other project. You wouldn't build a washroom and budget someone to clean it.
So she asked, well why don't you include that in your estimates? So we can have proper costing for the project.
Then I thought about it. There is a huge gap in accounting and IT. In IT, we often view accounting as something in the way. Stupid time tracking software. Bah, I'll just charge all my time to my assigned project. I have to provide an estimate. I'll just provide an estimate for the development. It's not my job to think about knowledge maintenance or operations...
Now of course the average IT person should not think about this, but the upper IT folks definitely should by setting up the right groups and proper leadership on how projects are structured. It's their job to get the funding so to speak.
It's not so much a problem that all the heads want to know is budget, on time...
It's largely that IT accounting is very poor to not account for all the costs. And we're so used to delivering something even if it does not account for maintenance that they get used to it.
Re: (Score:2)
Development is often capitalized. Maintenance has to be out of cash flow (accounting rules etc).
So companies trying to get acquired often hide their software maintenance costs in development budgets. Until auditors start to calculate development to maintenance cost ratios for IT organizations this will continue as the perps are laughing all the way to the bank.
It's fundamentally an accounting problem.
Re: (Score:2)
The senior management in a Car Manufacturer do not understand the composition of engine lubricants, nor why one is superior to another.
The senior management in a Dairy do not understand the mechanics of attaching a pump to a cows udder.
I don't understand why, in IT, people seem to think management in THEIR industry should somehow be superior to managers in other industries. Each has a job to do, and does it.
In the case of your Powerpoint to explain why a memory upgrade for a server was needed, I would sugge
Re: (Score:2)
Not always. For example, the Senior VPs at Verizon can discuss the bits and bytes of the GSM protocol with you. They understand how 3G can be subverted, how A5 is weak, signal strength, all that.
And have you ever looked at a hospital's administration board? Notice how many of those people have the word Doctor in their title? If a hospital doesn't promote doctors into executive positions, they make bad decisions and patients suffer and die.
If a company makes an effort to promote engineers to lead up the
Re: (Score:2)
Carly is radioactive. HP paid her millions just to go away.
The dems are dreaming if they think she will be nominated.
Information-poor article (Score:5, Informative)
I read the linked article hoping for insight on how to identify redundant infrastructure, steer divergent IT departments towards common solutions, or at least practical examples of uniformisation / centralization of *something*, anything, but there were none. It's all just a tech-free summary of the guy's accomplishments, as you'd present in a management meeting to tout your success as an IT manager. That's good for him and for the ACS, I guess, but it's pretty pointless to post it here.
Re: (Score:3)
Re: (Score:2)
Translation (Score:3)
increasing spending on strategic projects from 5 percent to 40 percent
Rarely does this sort of change involve spending more money in absolute terms. Most likely people doing stuff like support were retasked, outsourced, or simply cut.
Most of the stuff in the summary sounds like a good move, but I've seen companies that were fairly short-sighted in making these kinds of moves. At work we've been focusing on "strategic projects" for a long time now in belt-tightening mode and it seems like the biggest result is that every department under the sun has sprouted its own mini IT department that does all the stuff that IT stopped doing, usually less efficiently than it would be done if centralized.
When IT cuts a service that really was necessary, the result usually involves a net loss of money.
Re: (Score:2)
increasing spending on strategic projects from 5 percent to 40 percent
Rarely does this sort of change involve spending more money in absolute terms. Most likely people doing stuff like support were retasked, outsourced, or simply cut.
Most of the stuff in the summary sounds like a good move, but I've seen companies that were fairly short-sighted in making these kinds of moves. At work we've been focusing on "strategic projects" for a long time now in belt-tightening mode and it seems like the biggest result is that every department under the sun has sprouted its own mini IT department that does all the stuff that IT stopped doing, usually less efficiently than it would be done if centralized.
When IT cuts a service that really was necessary, the result usually involves a net loss of money.
They spent 8 times more and actually started paying attention.
Stuff worked better.
Imagine that.
All this tells me is to stop wasting money giving it to the cancer society as they are prone to wasting it.
Re: (Score:2)
There is a tendency in big companies to glorify small jobs.
At work there was a need for somebody to do a fairly administrative task. There was quite a bit of work - probably the better part of a full-time equivalent.
Now, if they had posted a job for $15-20/hr they could have hired somebody exceptionally qualified to do the work, and they would have done the work diligently and been happy to have the job.
Instead they used this as an excuse to hire an advanced degree position for probably $70/hr total cost,
So how did it save them? (Score:5, Funny)
one global entity ... consolidating dozens of data centers into three ... increasing spending ... reducing 600 core systems down to fewer than 200
As far as I can tell he created a single point of failure, reduced diversity and thus resilience, and instead of getting a cost advantage, it cost more. Reminds me of a joke:
A shepherd was herding his flock in a remote pasture when suddenly a brand-new BMW advanced out of the dust cloud towards him. The driver, a young man in a Broni suit, Gucci shoes, Ray Ban sunglasses and YSL tie, leaned out the window and asked the shepherd... "If I tell you exactly how many sheep you have in your flock, will you give me one?" The shepherd looked at the man, obviously a yuppie, then looked at his peacefully grazing flock and calmly answered "sure".
The yuppie parked his car, whipped out his IBM ThinkPad and connected it to a cell phone, then he surfed to a NASA page on the internet where he called up a GPS satellite navigation system, scanned the area, and then opened up a database and an Excel spreadsheet with complex formulas. He sent an email on his Blackberry and, after a few minutes, received a response. Finally, he prints out a 130-page report on his miniaturized printer then turns to the shepherd and says, "You have exactly 1586 sheep. "That is correct; take one of the sheep." said the shepherd. He watches the young man select one of the animals and bundle it into his car.
Then the shepherd says: "If I can tell you exactly what your business is, will you give me back my animal?", "OK, why not." answered the young man. "Clearly, you are a consultant." said the shepherd. "That's correct." says the yuppie, "but how did you guess that?" "No guessing required." answers the shepherd. "You turned up here although nobody called you. You want to get paid for an answer I already knew, to a question I never asked, and you don't know crap about my business...... Now give me back my dog."
Re: (Score:2)
Re: (Score:2)
Re:So how did it save them? (Score:5, Funny)
The man in the balloon called to the man in the field and said, "Hello! Can you tell me where I am?"
The man in the field said, "You're in a balloon, floating about 40 feet above a field."
"You must be a developer," said the man in the balloon.
"Yes, I am. How did you know?" said the man in the field.
"Because what you've told me is technically accurate, but it's of no use whatsoever," said the man in the balloon.
"You must be a manager," said the main in the field.
"Yes, I am. How did you know?" said the man in the balloon.
"Because you don't know where you are, you don't know how you got here, and you don't know how to get out of the situation you're in, but somehow it's my fault," said the man in the field.
Re: (Score:3, Insightful)
As far as I can tell he created a single point of failure, reduced diversity and thus resilience, and instead of getting a cost advantage, it cost more.
And how did you determine that? 200 servers, 3 datacenters, if you can consolidate all your stuff onto 66 servers in each datacenter, say 22 clusters of 3 with global routing (3DNS or something similar) so every datacenter is "live" with a portion of the traffic, clustered DBs between datacenters, etc, you could have excellent redundancy. And if you replaced 600 older servers with new big honkin servers you could have even increased the capacity.
Re: (Score:2)
a) Running a datacenter isn't cheap
b) Most likely they went from paid-for, working systems in a refrigerated closet somewhere to a contractor-ran, equipment-rental datacenter with big yearly contracts to IBM, PeopleSoft etc. A well managed local system is a lot cheaper than a borrowed system.
c) Most likely the stuff was local for a reason. Putting it away in a datacenter on the other side of the country not only increases latency, you also need fatter pipes. You go from a 10Mbps/10Mbps line being sufficient
Re: (Score:1)
Umm... he took a sheep, not a dog.
If a shepherd can't tell the difference perhaps he's really in trouble.
Re: (Score:2)
woosh
you must be a consultant
Now the hot question is? (Score:2)
Re: (Score:2)
Will it work?
Who knows? Only Time [youtube.com]!
Some of the aside projects maybe be due to incompetency, lack of coordination and redundancy, but I bet a few are to escape giving accountability to the HQ.
Which is "good" for everyone starting those projects but bad for HQ (*IF* they are against this kind of "flexibility").
Seeing all this problems as a technical project is doomed to failure.
I guess they (HQ) don't see it so much as a "technical" but as a "managment" project - see above about "flexibility".
What are they doing to prevent this situation won't happen again in a few years?
The usual: they do this "overhaul phase" and wait few years before they do the next... (usualy, things get worse if an organization is expanding)
Who says nowadays one needs a local physical datacenter anymore when rogue players can put together a datacenter in the cloud?
HQ?
Seems reasonable... (Score:5, Insightful)
It's a lucky coincidence that that applies to IT systems as well!
Re: (Score:3)
Frees up more money for the CEO (Score:1)
The CEO of the ACS is paid well over a million dollars per year for his services, in addition to having 24/7 use of a private jet, personal limousine and chauffeur, and security guards.
For every $1 the ACS spends on direct patient benefits, it pays $6.40 in salary, compensation, and overhead, according to 2008 financial filings.
BOHICA (Score:1)
It seems they've done a good job pulling all the disparate regions of support into a more compact set of teams, tools and services and have done a fairly good job of it so far.
That being said, the consolidation of IT services always leaves out the most important and most broken aspect of IT - communications. Companies assume business communication improves with the introduction of better phone tools, more robust email services and more efficient instant message systems. Generally, the exact opposite is th
More money to saving lives - heh! (Score:2)
>> while streamlining IT can often be painful upfront for IT managers, the payoff for sticking with it can feed into saving more lives.
This made me giggle. The CEO of the American Cancer Society pulls down more than $2M a year. Any IT savings are much more likely to be plowed back into executive bonuses than charity work. https://www.charitywatch.org/c... [charitywatch.org]
How does this "save lives" (Score:2)
Reading through the summary I can't see a breakthrough in cancer so I'm pretty dubious an IT shift has "saved lives" unless somehow it is directing more funds into research - but I doubt that is the case.
Brings back early memories running at 300 baud (Score:2)
My dad ran the epidemiology department for the American Cancer Society when I was a pre-teen and teenager in the '80s. I grew up dialing into their VAX 11/780 with a 300 baud acoustic coupler modem. At first I used a DECwriter terminal [columbia.edu], which didn't have a screen—all the output was noisily printed to 132-column tractor feed. Eventually my folks brought home a VT180 [vintage-computer.com], around the same time that we upgraded to a 1200 baud modem. I'll never forget playing Crystal Caverns, and creating ASCII "animations" as