Big HMO Jolted By Email, System Failures 171
JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."
Smoke, meet fire... (Score:5, Informative)
I don't know what to say about the first e-mail until I see more but the CEO sounds like a real jack ass in the second one. And if that 99.2% number is right then they got bigger problems than some email - that's all I can say!
ac
Re:Smoke, meet fire... (Score:4, Informative)
Re: (Score:3, Insightful)
Re: (Score:3, Informative)
Looks like things are getting better now; they have brought in several IT and facilities
Re: (Score:3, Informative)
For instance, we stated having failures of refrigeration units in a new facility I helped build. Everything was going fine, until we stated getting calls regarding multiple equipment failures and risk to the product ins
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
Re: (Score:2, Interesting)
From the vendor perspective (on-site maintenance manager for a multi-national bank), we insisted that certain downtime be scheduled (IBM 3800's are *very* picky about when they are fed parts). As a result, during the 18 months I was in charge of one of the 3 data centers, we only dropped below 7/9's one week. Neither the vendor (if ethical) nor the customer wants to see any less.
Re: (Score:2)
The last 3 jobs I applied for did! (Scary: 2 were government, and one was a company that designed databases for container shipping.)
Re: (Score:2)
Manager's seem to love (or did) that five 9's lingo... i'm trying to point out that by itself means nothing. It's like saying my OS is rock solid.. without mentioning it's also not running anything at all.
Re:Smoke, meet fire... (Score:5, Informative)
In all reality they are not the great evil. They certainly could do better, but by no means are they horrid (this assumes that you, as a patient, know what you need, and press for a second opinion &&|| a specialist when the initial diagnosis doesn't seem quite right).
-nB
Re: (Score:2, Insightful)
Re: (Score:2)
Now I realize that this is true to _some_ extent everywhere, just because of reality, I really hope that this isn't part of the culture or procedure at Kaiser facilities.
Many people recieving health care do not have the capability to 'own' their profesional care. Its great that some people (and/or thier families) can do this, but it would be a shame if anyone who can't is simply thrown to the wolves.
Re: (Score:2)
Owning your own care, while not a culture or expectation at kaiser, is a fact of life. The staff is overworked, and the number one gripe I hear from staff is about the computer system. They find out where I work, and I get unloaded on...
It's simply that the staff has more to do than they can realistically cover. My kids' pediatrician is great, but if you don't get a morning appointment you will b
Typical of medical and insurance businesses. (Score:2, Interesting)
Re: (Score:2)
While the use of old technology at a physicians office may be true, I wouldn't generalize hospitals in that category. I have consulted at two different hospital systems in the past six years. At each location, the hospital was pushing to the forefront of technology. My last hospital was in the early stages of virtualizing all of their hardware environment and was using the most recent UNIX hardware
Re: (Score:3, Insightful)
Re: (Score:3, Informative)
Re: (Score:3, Insightful)
There are several legacy, "green screen" apps in the insurance industry still kicking around out there. Some of these were initially developed as far back as the late 1970s. They are capable of high transaction rates in comparison to java-driven, web-based, "thin client" applications. Many of today's OLTP applictions would be better suited for ASCI-based interf
Re: (Score:3, Interesting)
I was involved in a project a few years back where a hospital's IT system ran on a bunch of OpenVMS systems, because the main application only ran on OpenVMS. (Which I concede is a capable operating system (logicals are even more powerful than symlinks, etc. etc.) but was outdated even in 2001.) They were also - by a factor of four or so - the largest deployment of the software, so they ran into all
My dentist... (Score:2)
Re: (Score:2, Insightful)
Re: (Score:3, Insightful)
That's not being cheap. That's being a very conservative organization who can't really gamble on things suddenly not working.
I have seen numerous projects where a company with some older machines (say, mainframes) is being sold a piece of software to replace their 20 year old system -- and replace it with a Windows bo
Re: (Score:2)
Eventually the printer died. The app (dos, running on a 486) was hard coded for only three possible printers.
Guess how many I was able to find? none. not on ebay, not supported with emulation. none. I contacted the person who wrote the app and asked if I could have a copy of the source, to update the printer driver. I offered to sign an NDA and a non-redistribution agreement, as well as giving the source c
Re: (Score:2)
I had personally, in 1999, had to go in and call Texas Instruments specifically to special-order a logic board for an ailing TI-931 terminal (early 80's model). I'm betting it's still there, right next to the old COMBAT 300-baud modem they used to call in their accounting sheets to Ford Motor Company every night.
(Then again, I was living in my hometown in Northwest Arkansas at the time, so draw your conclusions from that too...)
Re: (Score:2)
Re: (Score:3, Interesting)
> hospital, insurance company or doctors offices. These places
> hate to spend money on IT and let old gear sit in place for
> almost ever.
Except that this article shows why that might be: true critical systems need to be reliable and understandable. Systems that have been in production for many many years often meet those requirements. That is why you often see 3270 green screen applications in large medical offices, and DOS or (old,
Re: (Score:2)
Or option 3, has no choice but to be in an HMO because that's all their employer offers.
I'm in that situation and almost all medical expenses are taken care of. Then again, I'm a basic user. Twice a year dentist check is all I need.
Granted, I wish I could opt out of paying for any medical and get the money in hand and pay my own, rare, expenses but I don't have that option. It would be cheaper for me to get the money and
Young people are cheap to insure (Score:2)
Granted, I wish I could opt out of paying for any medical and get the money in hand and pay my own, rare, expenses but I don't have that option. It would be cheaper for me to get the money and pay when I go to a visit than it is to piss away money every paycheck and not use it.
Your message sounds to me as though you're fairly young. Have you ever checked insurance premium charts? The cost to cover a 20-something is about half the cost of someone in their late 40s and above. The whole idea of HMOs and group insurance was to spread the risk among huge patient pools so no one patient would ever face large medical bills for the unexpected. So while 20-somethings fuss over $400 monthly premiums deducted from their paycheck, keep in mind that the level amount deducted from all employe
Re: (Score:2)
While the concept of pooling resources sounds good, in the end it only ends up hurting more than it helps. You
Re: (Score:2)
Re: (Score:2)
That is a problem with healthcare in this country. If you let companies make a profit off of anyone's illness, everyone will be screwed (except the CEO's of those healthcare companies). I lived in the Soviet Union and I can confirm that as bad as the Soviets were they had very good basic healthcare. It didn't matter what you had who you were, you could get treatment. You just had to walk into a hospital tell the
Re: (Score:2)
In the end, we'll have only sick/old people with insurance, costing $10k/month, and everyone else with no coverage and an unpayable $1m hospital bill if they fall down the stairs.
That seems pretty dysfunction
Epic Systems? No wonder... (Score:5, Funny)
C'mon, it really can't be that bad, can it?
Oh, Epic Systems? No wonder. Dude, you're f****ed.
Re: (Score:2)
Re:Epic Systems? No wonder... (Score:4, Informative)
*sigh* - too true. Epic requires a monumental effort just to get the off-the-shelf product to work properly. Kaiser is doing a massive amount of internal development in addition to trying to implement as Epics biggest customer.
Part of me wants to think that Justin is just fresh meat who hasn't put in the time to become as jaded as the rest of us in HIS. The other part of me thinks he might be right and that Kaiser has bitten off way more than it can chew. Failure to understand the effort involved is just as pervasive as the massive waste he's critiquing, and just as harmful.
Re: (Score:3, Interesting)
Re: (Score:2)
Anybody got some useful references on how much Epic sucks and how screwed we are? I'd like to know ahead of time so I can start looking for a new job.
Re: (Score:2)
Re: (Score:2)
Then I asked the recruiter what database systems they use and they mentioned CACHE and the MUMPS language. MUMPS is some archaic pre-C
Did they perhaps build it in MUMPS? (Score:2, Funny)
Re: (Score:2)
$4billion for a failing healthcare system? (Score:3, Funny)
As someone going to a Kaiser appt. today... (Score:2, Interesting)
Re: (Score:2)
I experience poor service whenever I deal with any organization in Health Care. Of course, where I live, Kaiser isn't a significant presence, so I can't blame them or their new software. I just blame it on the shitty system of Health Care we have here in the US.
Re: (Score:3, Funny)
Re: (Score:2)
Ever since the '40s we've been trading in zebra finches. Much easier to carry, and you can swap 2 finches for a goat and 10 finches for a Cow at any local farmer's market.
What's an HMO? (Score:2)
Re: (Score:2)
Re:What's an HMO? (Score:4, Funny)
If only there was a way for someone to find information on the internet. Like a centralized tool or website that indexes other websites and allows people to search for it when they don't understand.
Or wait, even better - how about an encyclopedia like website that could contain vast amounts of knowledge.
Ah, that would be grand.
Re: (Score:2)
Re: (Score:2)
Slashdot is read by people outside the USA.
Firstly, Slashdot is intentionally US-centric [slashdot.org]. If people from France or Japan or wherever don't like that, they should look for other news websites, because this one is apparently not for them. Secondly, the GP's point is that Google and Wikipedia are also read by people outside the USA, and in this case should have been. Here are the results from a Google search on HMO [google.com], the first of which is the Wikipedia page on it, with "Health Maintenance Organization" spelled out right on the search results page.
Re: (Score:2)
Re: (Score:2)
So what? (Score:3, Insightful)
Private incompetence pales in comparison to government incompetence.
Re: (Score:2)
Re: (Score:3, Insightful)
Actually, the oposite is true. For example, the US Medicare system spends about 98% of its funding on patient care. Even the largest and most efficient private insurers only can manage about 90%. The radical right-wing wants to preserve the profits of private insurers and so they spend a great deal of their propaganda efforts highlighting mistakes in the public sector so it appears that government is less efficient than it actually is.
Re: (Score:3, Interesting)
For example, the US Medicare system spends about 98% of its funding on patient care.
Administrative overhead is just about the worst possible measure of efficiency. What you want to measure is amount of needed services delivered for the cost. Medicare's low administrative overhead means that it doesn't have the administrative capacity to avoid paying for unnecessary procedures or to catch providers upcoding visits or procedures or outright inventing them. Even ignoring the question of how much of the services Medicare pays for are actually necessary, Medicare has a huge rate of outrigh
Long-term Kaiser patient disputes wild claims (Score:5, Informative)
Let me tell you of my experience visiting my doctor yesterday (Thursday). During my visit, my doctor pulled up recent lab results on the exam room console. He was able to prescribe new medication and schedule follow-up lab test through the system (no paper). I went down the hall to get a tetanus booster, then walk downstairs to pick up my prescription. All with no paperwork. I believe their system is phenomenally successful. I won't dispute the cost of this project.
I've undergone a number of procedures and consumed considerable medical resources during the past year. All of my records are computerized and information is easily shared among their medical professionals. The doctors, nurses, lab technicians have access to information required to deliver quality medical care to me.
Re: (Score:3, Interesting)
Does the "pain" I feel really matter when there are actual sick people up in that building relying on providers who need up-to-date and accurate informati
Re: (Score:3, Insightful)
It really pains me every-time I see one of these large organiz
Re: (Score:2)
I seem to remember that Kaiser had an internal division which was working on a electronic-medical records project back in 2000/2002. They shut down the project, laid off a bunch of staff, and started over with a whole new system.
I know that not everyone has had a good experience. I have used Kaiser in the past, and it was horrible. For us, this ne
Re: (Score:2)
Re: (Score:2)
A previous article... (Score:5, Interesting)
As someone who has been frustrated by a variety of Electronic Medical Records systems in different medical settings, I must say that my "favorite" has been VistA [vistasoftware.org] (the medical records software used by the Veteran's Administration, and no relation to Microsoft Vista). Currently, I'm using GE's Centricity [gehealthcare.com] at my work site and have had some minor problems that have resulted in delays in entering my data. (Problems with VistA were more related to the entire network being down - problems with Centricty have been with database connectivity... I wish I could say more about it, but I'm not an IT person, I'm just a lowly end-user).
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
This all could have been avoided... (Score:4, Funny)
Hyperion: If it's good enough for Google, it's good enough for you.
Percentages and Marketing speak (Score:4, Informative)
It's also why many companies can say 99% uptime as that's close to 88 hours of downtime out of a year.
Hours in a year 8760
99% = 87.6 hours
99.2% = 70.08 hours
99.7% = 26.28 hours
If they are at 99.2% right now, I wonder how the heck they are going to get that extra 0.5% percentage points with all the problems they have now.
Re: (Score:3, Insightful)
Re: (Score:2)
Sure there are computer with 5 9 availability, my first job was programming on a Stratus, an expensive but very reliable computer, but
1) even with a Stratus those 5 9 are mostly theoretical, in one year, I've seen:
- power outage: the maintenance guy cut the wrong cable.
- several software failures.
2) a complex system is usually less reliable than the parts used to build it.
Re: (Score:2)
I think though that if you start with hardware that is only 99.9% reliable, it will be impossible t
Re: (Score:2)
In theory, 99.7% uptime is a system outage of 6 hours once a quarter for maintenance. It might take them a coup
Re: (Score:2)
I cannot say that is a High Availability system, such as this, that they use the same standards.
And to say that they can't achieve 99.9% availability for power is entirely false. Rackspace, who hosts my companies website, has far better than 99.9% uptime on their infrastructure. That's what redundancy and backup generators are for
The worst? (Score:2)
'the worst [technology] project I have seen in my 25 years in the business.'
Yikes. If his 25 years of experience are anything like my 10 years, then that's saying a LOT.
The State of Maine fails: (Score:2)
http://www.cio.com/article/20133/Maine_s_Medicaid_ Mistakes [cio.com]
Windows system doesn't scale. What a shock. (Score:5, Insightful)
From the ComputerWorld article: Deal and an IT employee, who spoke to Computerworld on the condition of anonymity, said part of the problem with the HealthConnect system is that the Citrix Application Delivery infrastructure implemented by Kasier just can't handle the load of the Epic Systems.
"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network. For every user who connects to HealthConnect, they connect via Citrix, and we're running into monumental problems in scaling the Citrix servers."
So instead of deploying the app on N thousands Windows desktop, they deciced to use Citrix to remotely connect to a pool of servers. The Citrix server and the Windows machine at the other end could not stand the load. Big surprise.
The way normal people would do it is use an X11 graphic application (X11 is available for Win32), or use a Java webstart client, or even do everything within a browser, or... But there are many, many way to architect a distributed app these days.
The ONE thing you shouldn't do is deploy lots of Windows servers, use the half-baked ICA protocol, and expect everything to be peachy.
Remember, CIO boys and girls: Uncle Bill's broken OS just cost lil' Cliff Dodd his job. Don't be the next one. Keep Win32 where it belongs, outside the server room.
Re: (Score:2)
"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it.
Considering the performance and flexibility problems observed even on small, normal Citrix installations, I'd have to say the above sounds like a bad move -- probably one among many in this project.
Heck, what could be sillier than basing a critical system on a Citrix deployment of untested scale and unconventional topology?
The way normal people would do
Re: (Score:2)
Re: (Score:2)
Kaiser Permanente has significant problems (Score:3, Informative)
Add to this the reports of KP dumping homeless patients on the streets. There was the kidney transplant scandal. The patient information becoming available online scandal. And all of this in just the past 2 years. Kaiser is clearly flawed from top to bottom. They are the poster-child for real health care reform in this country.
Re: (Score:2)
Epic Systems (Score:3, Funny)
Interesting website .. (Score:2)
Kaiser sub-standard care. (Score:3, Informative)
For me the biggest issue came when it was time for discharge. The handed me a hand written sheet with about 12 different medications scribbled in "Doctorese" cursive writing. I couldn't read half of the sheet, and only the briefest of notes for each medication was listed. I was shocked that I didn't receive a computer printout with the medication instructions, especially for medications needed to be taken in the case of quadruple bypass heart surgery.
After many hours of sorting through the medications, trying to figure out if he had already been given a dose at the hospital or not, etc... I realized that when I had visited a non-Kaiser emergency room for a sprained ankle I got a multi-page printout with very thorough instructions for the medication and aftercare. On another visit when I had 107 temperature it was the same thing. I was very certain of the doctors orders for post-visit care.
With Kaiser though I was left confused with short hand written notes about a dozen medications. Confusion is not a state I wanted to be in when I'm charged with the care of one of the most important people in the world (to me) who has a life threatening condition!
Re: (Score:2)
If so, did you get satisfactory results? If not, why?
I really hope you didn't walk away saying "ok, so this is what they give me, I guess I have to figure it out on my own." As a patient or care-giver, IMHO if you don't understand or can't read what you've been given, you need to get it straightened out before you leave the hospital/doctor. And the hos
Re: (Score:2)
Re: (Score:2)
Re: (Score:2)
If you had a temperature of 107F (Score:2)
Re: (Score:2)
Can you say "will never work in IT again" ? (Score:2, Flamebait)
Repeat after me: sir, can you spare some change?
That'll be a useful skill for you.
Re: (Score:2, Insightful)
The world needs more people with spines like this kid, good for him.
Repeat article and get some perspective (Score:2)
Second, it is true that the medical health system they're implementing is super-complicated but name one that isn't that does what everyone wants. The FBI spent $100m+ [washingtonpost.com]
Puny Americans and your tiny incompetence (Score:2, Funny)
Ha! Witness the failure of American-style capitalism to produce REAL waste! Why, here in the UK our glorious NHS have just recently thrown away 12bn of public money on a computer system that has ZERO percent uptime -- because nobody could decide what it was for and nobody wanted it! Compare that to the paltry few billions your private enterprise was able to throw away!
And that's not all! The graft, corruption, bribery and crime surrounding the NHS system was such that Accenture *refused to work on it* -
99.7% is a GOAL? (Score:2)
Dilbert... (Score:2)
Reminds me of a PHB speech in a Dilbert book... Typed from fuzzy memory:
"This year our goal is 0 serious injuries! In hindsight, last year's goal of just 20 serious injuries was a mistake. We had to seriously injure 6 people to meet that goal."