Anatomy of the VA's IT Meltdown 137
Lucas123 writes "According to a Computerworld story, a relatively simple breakdown in communications led to a day-long systems outage within the VA's medical centers. The ultimate result of the outage: the cancellation of a project to centralize IT systems at more than 150 medical facilities into four regional data processing centers. The shutdown 'left months of work to recover data to update the medical records of thousands of veterans. The procedural failure also exposed a common problem in IT transformation efforts: Fault lines appear when management reporting shifts from local to regional.'"
awesome! (Score:4, Informative)
Instantly, technicians present began to troubleshoot the problem. "There was a lot of attention on the signs and symptoms of the problem and very little attention on what is very often the first step you have in triaging an IT incident, which is, 'What was the last thing that got changed in this environment?'" Raffin said.
p.s. I am shocked at how many junior cowboy IT people remain employed, given the supposed glut of hire-able and knowledgeable folks.
Re:Assumption junction, what's your function? (Score:5, Informative)
DOH! Looks like someone was making assumptions without reading the article. They considered switching to the backup, but since they didn't know whether the problem was on their end or the server's end, they were afraid that switching to the backup data center would destroy that one as well.
Re:Good thing they weren't running Ubuntu (Score:2, Informative)
Re:In other words.... (Score:5, Informative)
At the time, there seemed to be a lot of waste (think $10,000 CD burner in 1993ish, optical cards with images and data impressed on them, etc). But they really were trying to be ahead of the game - a friend of mine showed me his green card and it was almost identical to a design I was working with when I was at the DVA. They also had mechanisms for charging back to private insurance companies in the event a veteran was only partially covered for a visit.
Oh, and just about all the software that was written and in use by those hospitals are in the public domain and downloadable [worldvista.org] for free - many other hospitals use VistA as their base.
Re:They messed up everything they could mess up. (Score:3, Informative)
The article also states that the cause was a network configuration error. While in a perfect world you would have test and QA systems that are identical to production, it is rarely feasible. The cost to set up large parallel networks with exactly the same configuration in addition to software with the same configuration is generally cost prohibitive. By "same configuration" I mean same IP addresses, port assignments, routing rules, nationwide WAN links, etc.
Re:Why always centralizing? (Score:3, Informative)
You also need to realize that these system use a hierarchical database for speed, so "joins" are much more complicated after the fact, than simply centralizing onto regional servers. Also regionalization is how the VA works as well with local integrated delivery networks (sort of hub-and-spoke with clinics, small and large medical centers) where the patients are often going between institutions in the region.
I recently left the VA system, and have to say their system is better than almost anyone else's, and has a similar failure rate to those at other institutions; as someone pointed out, it is just that this makes bigger press. In general there are very few health systems which have a single pervasive medical record of this scale (heck most places don't have electronic medical records at all) such as Kaiser Permanente here in the US, and the NHS in the UK, so these stories seem all the more spectacular since they are so rare.
Re:In other words.... (Score:3, Informative)
Re:Cache is part of the problem (Score:3, Informative)
M is what T-SQL/stored procs wants to be when it grows up. I'm pretty sure getting help from Intersystems isn't an issue at the VA.
This is a Management/Change Management issue. Not a technical issue.
Re:In other words.... (Score:4, Informative)
http://www.washingtonmonthly.com/features/2005/0501.longman.html [washingtonmonthly.com]
If you think the VA is bad, you can always go to your favorite HMO and have a higher chance of death.
Did I mention that the VA is a leader in hospital IT infrastructure and is decades ahead of other hospitals?
http://en.wikipedia.org/wiki/Veterans_Health_Information_Systems_and_Technology_Architecture [wikipedia.org]
The VA is the largest hospital system in the US and its budget is decreased most years after adjusted for inflation. Given the predicament that Congress puts them in, they've done pretty well.
However, every single mistake they make is a public headline. Private hospitals have the luxury of being sued and quietly settle for $$$. Instead, the VA has to endure lots of bad publicity.
If the VA was a corporation, costs would skyrocket and even more corners would be cut. If you want to make it better, how about you ask Congress to provide adequate funding for the avalanche of people they are getting?
Re:In other words.... (Score:3, Informative)
HOWEVER, it is a brilliant example in which a public outrage was sparked, and the government was forced to do its job, and did indeed clean things up after the horrible conditions were brought to light.
If it were a private hospital, I fear that things would have been kept hush-hush for far longer through lawsuits and settlements. Even then, the worst that the government could do to the place would be to either impose fines, or shut them down entirely. Neither of these scenarios would benefit the patient.
90% of the time, patients at hospitals don't have an active choice in which hospital they arrive at. If you're severely injured or sick, you're not going to drive 2 extra hours to the "better" hospital. You're going to want to get immediate medical attention. In this regards, hospitals are prime candidates for nationalization, as they are hardly ever in competition with each other, which in turn results in all sorts of nasty side effects like inefficiency, negligence, etc.... It really does make sense to hold every hospital to the same exact standards across the nation.
You're also conveniently ignoring the fact that Walter Reed was a single entity in a very large system. There are always going to be a few outliers. Considering that the VA isn't funded nearly as well as it should be (we cut funding to the VA, and used the cash to fund an illegal war) it's still a pretty darn nice health system. An outrage was sparked because a government-run institution that was part of system in which all of the hospitals should have been more or less equally, had fallen behind. This simply would not have occurred in a private system.
To use these reasons to argue against a nationalized healthcare system in the US is to be completely ignorant. We're one of the only industrialized nations *without* a nationalized healthcare system open to all. The idea works. Reconciling the general incompetence of the US government is another issue entirely.