Big HMO Jolted By Email, System Failures

JoanofAlaska writes "The Wall Street Journal is running a front page story about the internal mass e-mail that exposed the failing $4 billion dollar electronic medical record system at Kaiser Permanente, the biggest non-profit HMO in the country. When word of the system's meltdown quickly spread back in November, one reporter obtained a 722 page internal document that showed patient safety lapses as a result of the system's problems. Then in February, the Los Angeles Times had a front page story in which a systems analyst who worked on the project called it 'the worst [technology] project I have seen in my 25 years in the business.' They've created a website to try to rebuild confidence in the project, and they say their goal for system availability is 99.7% (they're currently at 99.2%)."

Smoke, meet fire...

Here's the first e-mail [kaiserthrive.org]. Their CEO sent out an e-mail response [blog-city.com] to all their employees the next day (the same day it looks like the CIO suddenly resigned effective immediately).

I don't know what to say about the first e-mail until I see more but the CEO sounds like a real jack ass in the second one. And if that 99.2% number is right then they got bigger problems than some email - that's all I can say!

ac

Re:Smoke, meet fire...

To put the 99.2% figure in perspective, it amounts to almost 3 full 24-hour days of downtime a year.

Re:

I'd like to see more places quote availability numbers as a percentage AND the longest downtime interval in the previous year. 12 minutes of downtime every night at midnight sucks, but 3 solid days of downtime is a disaster.

Re:

I think in 2006 they had 2-3 major incidents in each of their two data centers. Power might have only been out for around 15 minutes each, but system recovery took well over 8 hours for many of the problems. There were also several planned shutdowns for

Re:

What's wrong with N+1 redundancy? While I agree it requires an attentive response from Facilities to replace the failed unit quickly, my experience with facilities departments is that, if there is greater redundancy, they tend to rely on it to stretch out

Re:

The "five 9's" and all that is a joke anyways. We have that with our vendor... so that means we can have an office down for a solid week and they still met their goals. Just because a contract is "xyz 9's" doesn't mean jake without more info.

Re:Smoke, meet fire...

but say you can't get antibiotics for a day to treat an infection due to system downtime

That would not happen. As a kaiser member I can tell you three things:

• The patient owns their care. If you don't keep track of things then you will not get the care you need. (this is not malice, but rather I suspect kaiser is a victim of their size)
• The doctors and nursing staff want to do their best for the most part, but are overworked, again leading to "you own your own care"
• even when there is an outage, if an ER doc wants meds for a patient, they can get them. The computer system does all the record keeping, the humans still do the medicine, they have paper forms for everything, and when the computer is being an ass, they simply fill out the form like the "old days"

In all reality they are not the great evil. They certainly could do better, but by no means are they horrid (this assumes that you, as a patient, know what you need, and press for a second opinion &&|| a specialist when the initial diagnosis doesn't seem quite right).
-nB

Typical of medical and insurance businesses.

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices. These places hate to spend money on IT and let old gear sit in place for almost ever. A local hospital here has several Windows 98 pc's still in us

Re:

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices

While the use of old technology at a physicians office may be true, I wouldn't generalize hospitals in that category. I have consulted at two diffe

Re:

Considering how maintaining accuracy of patient history, treatment records, billing records, allergies, lab reports, and others is paramount I'd say if it ain't broke don't fix it. Could you imagine the change controls in an environment like that? "Potenti

Re:

What parent said. WYSE terminals aren't that bad, given that a lot of the medical data industry got it start from what is now known as VistA [sourceforge.net]. By some strange coincidence, the name of its predecessor was DHCP. Anyway, it worked on dirt-old VAX systems an

Re:

when my wife was in Insurance billing before she got her CPA she worked on old wyse 75 terminals and this was for a HUGE rich insurance company.

There are several legacy, "green screen" apps in the insurance industry still kicking around out there. Some

Re:

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices.

I was involved in a project a few years back where a hospital's IT system ran on a bunch of OpenVMS systems, because the main application only ran

My dentist...

My dentist office still uses DOS/text mode on a PC (no idea what's the setup -- maybe a 386?).

Re:

Hospitals generally don't keep old computer equipment because they are cheap bastards. Because of the patient safety issues, and related federal/state regulations, most medical related software have *extremely* narrow specs. They keep the old hardware arou

Re:

If you want to see the oldest computer gear simply go to a hospital, insurance company or doctors offices. These places hate to spend money on IT and let old gear sit in place for almost ever.

That's not being cheap. That's being a very conservative organi

Re:

> If you want to see the oldest computer gear simply go to a
> hospital, insurance company or doctors offices. These places
> hate to spend money on IT and let old gear sit in place for
> almost ever.

Except that this article shows why that might

Re:

Anyone in an HMO is either dirt poor and has no choice or a total idiot

Or option 3, has no choice but to be in an HMO because that's all their employer offers.

I'm in that situation and almost all medical expenses are taken care of. Then again, I'

Young people are cheap to insure

Granted, I wish I could opt out of paying for any medical and get the money in hand and pay my own, rare, expenses but I don't have that option. It would be cheaper for me to get the money and pay when I go to a visit than it is to piss away money every paycheck and not use it.

Your message sounds to me as though you're fairly young. Have you ever checked insurance premium charts? The cost to cover a 20-something is about half the cost of someone in their late 40s and above. The whole idea of HMOs and group insurance was to spre

Epic Systems? No wonder...

In a blistering 2,000-word treatise, Mr. Deal wrote: "We're spending recklessly, to the tune of over $1.5 billion in waste every year, primarily on HealthConnect, but also on other inefficient and ineffective information technology projects."

C'mon, it really can't be that bad, can it?

Kaiser refutes Mr. Deal's assessment of its custom software system, developed by Epic Systems Corp.

Oh, Epic Systems? No wonder. Dude, you're f****ed.

Re:

Man. And you thought it was just DNF.

Re:Epic Systems? No wonder...

Oh, Epic Systems? No wonder. Dude, you're f****ed.

*sigh* - too true. Epic requires a monumental effort just to get the off-the-shelf product to work properly. Kaiser is doing a massive amount of internal development in addition to trying to implement as Epics biggest customer.

Part of me wants to think that Justin is just fresh meat who hasn't put in the time to become as jaded as the rest of us in HIS. The other part of me thinks he might be right and that Kaiser has bitten off way more than it can chew. Failure to understand the effort involved is just as pervasive as the massive waste he's critiquing, and just as harmful.

Re:

Exactly. Just do a search for Epic Systems employment and you'll find plenty info Epic. I am sure your "sigh" means that you already do know. I wouldn't wish even my enemy to work for Epic Systems. That place is so messed up, it is scary. Their CEO and fou

Did they perhaps build it in MUMPS?

More horrors [worsethanfailure.com] than you want to imagine.

$4billion for a failing healthcare system?

They're just not trying hard enough [theregister.co.uk]

As someone going to a Kaiser appt. today...

The problems with the new Kaiser software are obvious to anyone who's been to Kaiser recently or spoken to a doctor or nurses who work there. Test results disappear, appointments disappear.... sometimes the people on the phone can't schedule appointments

Re:

Test results disappear, appointments disappear.... sometimes the people on the phone can't schedule appointments at all and tell people to call back later.

I experience poor service whenever I deal with any organization in Health Care. Of course, where

Re:

You actually have healthcare in the US ? I didn't realise that, the impression I got is that when you get ill in the States you have to go off to the market rather than a clinic or hospital like you do elsewhere. I guess once you get to market you can swap

What's an HMO?

For the benefit of those of us who aren't Americans, why not say what an HMO is?

Re:

Health Maintenance Organization [wikipedia.org]

Re:What's an HMO?

For the benefit of those of us who aren't Americans, why not say what an HMO is?

If only there was a way for someone to find information on the internet. Like a centralized tool or website that indexes other websites and allows people to search for it when they don't understand.

Or wait, even better - how about an encyclopedia like website that could contain vast amounts of knowledge.

Ah, that would be grand.

So what?

From what I understand the British government have blown about $25,000,000,000 on a computer system for the National Health Service which doesn't work worth a damn and which the doctors didn't want in the first place.

Private incompetence pales in comparison to government incompetence.

Re:

I think there's a whole load of incredibly expensive and functionally useless computer systems in the NHS, this latest one is in good company.

Re:

Private incompetence pales in comparison to government incompetence.

Actually, the oposite is true. For example, the US Medicare system spends about 98% of its funding on patient care. Even the largest and most efficient private insurers only can manage abo

Re:

For example, the US Medicare system spends about 98% of its funding on patient care.

Administrative overhead is just about the worst possible measure of efficiency. What you want to measure is amount of needed services delivered for the cost. Medicare's low administrative overhead means that it doesn't have the administrative capacity

Long-term Kaiser patient disputes wild claims

This 45-year old lifetime Kaiser patient believes the media frenzy has blown this out of proportion. Kaiser Permanente undertook the monumental task of converting its patient records system into an electronic information system. Converting the hundreds of paper records for each of millions of patients is truly a monumental task and some problems will occur.

Let me tell you of my experience visiting my doctor yesterday (Thursday). During my visit, my doctor pulled up recent lab results on the exam room console. He was able to prescribe new medication and schedule follow-up lab test through the system (no paper). I went down the hall to get a tetanus booster, then walk downstairs to pick up my prescription. All with no paperwork. I believe their system is phenomenally successful. I won't dispute the cost of this project.

I've undergone a number of procedures and consumed considerable medical resources during the past year. All of my records are computerized and information is easily shared among their medical professionals. The doctors, nurses, lab technicians have access to information required to deliver quality medical care to me.

Re:

As someone who was up until 1:30 this morning frantically loading RAs into Chronicles in preparation for my Sunday go-live, I really needed to read your comment today. Dealing with Epics goat cluster of poor design decisions can easily lead an IS shop int

Re:

As a 45 year Kaiser member you should. What they have done and are still tuning is nothing special. Other HMO/MMOs have already done such and many did this years ago for a far cheaper price. Yes, what you saw was convenience yesterday and that is an awesom

Re:

You're plain wrong. They have switched to "e-prescribing" as I did not take a traditional 4x6 prescription not to the pharmacy. It was all by computer!

A previous article...

There is a Computerworld article [computerworld.com] from the previous slashdot story that seems pretty helpful in understanding the meltdown of their electronic medical records systems. They say that they are running the world's largest Citrix server system, and it does not scale well for their purposes.

As someone who has been frustrated by a variety of Electronic Medical Records systems in different medical settings, I must say that my "favorite" has been VistA [vistasoftware.org] (the medical records software used by the Veteran's Administration, and no relation to Microsoft Vista). Currently, I'm using GE's Centricity [gehealthcare.com] at my work site and have had some minor problems that have resulted in delays in entering my data. (Problems with VistA were more related to the entire network being down - problems with Centricty have been with database connectivity... I wish I could say more about it, but I'm not an IT person, I'm just a lowly end-user).

This all could have been avoided...

If only these guys had used Hyperion to manage their data, they wouldn't be having this issue. Hyperion can effortlessly manage up to 12 GIGABYTES of data, and all you have to do is partition it into 3 different pieces.

Hyperion: If it's good enough for Google, it's good enough for you.

Percentages and Marketing speak

Dell offers contracts for 99.9% system availability. This means that if you pay the service contract, your within 30 miles of a depot, they guarantee the system won't go down because of hardware for 8.76 hours out of the year. In training we were calculating all that. Technically, it calculates to two "no post" service calls out of the year. So it looks really good when you market 99.9% reliability.

It's also why many companies can say 99% uptime as that's close to 88 hours of downtime out of a year.

Hours in a year 8760
99% = 87.6 hours
99.2% = 70.08 hours
99.7% = 26.28 hours

If they are at 99.2% right now, I wonder how the heck they are going to get that extra 0.5% percentage points with all the problems they have now.

Re:

You know what's sad? I started my IT career with Compaq at it's (nee DEC) non-stop computing department. 99.99999% uptime for these suckers. Redundant everything, hotswapping of HDs, all in 1999. Can't believe people are touting 99.9% hardware availability

The worst?

'the worst [technology] project I have seen in my 25 years in the business.'

Yikes. If his 25 years of experience are anything like my 10 years, then that's saying a LOT.

The State of Maine fails:

Maine does a system cutover over 2 years ago, and its still broken. Doctors go out of business because they can't get paid, and everything's a mess. I think this one wins :-/

http://www.cio.com/article/20133/Maine_s_Medicaid _ Mistakes [cio.com]

Windows system doesn't scale. What a shock.

From the ComputerWorld article: Deal and an IT employee, who spoke to Computerworld on the condition of anonymity, said part of the problem with the HealthConnect system is that the Citrix Application Delivery infrastructure implemented by Kasier just can't handle the load of the Epic Systems.

"We're the largest Citrix deployment in the world," Deal said. "We're using it in a way that's quite different from the way most organizations are using it. A lot of users use it to allow remote users to connect to the network. But we actually use it from inside the network. For every user who connects to HealthConnect, they connect via Citrix, and we're running into monumental problems in scaling the Citrix servers."

So instead of deploying the app on N thousands Windows desktop, they deciced to use Citrix to remotely connect to a pool of servers. The Citrix server and the Windows machine at the other end could not stand the load. Big surprise.

The way normal people would do it is use an X11 graphic application (X11 is available for Win32), or use a Java webstart client, or even do everything within a browser, or... But there are many, many way to architect a distributed app these days.

The ONE thing you shouldn't do is deploy lots of Windows servers, use the half-baked ICA protocol, and expect everything to be peachy.

Remember, CIO boys and girls: Uncle Bill's broken OS just cost lil' Cliff Dodd his job. Don't be the next one. Keep Win32 where it belongs, outside the server room.

Kaiser Permanente has significant problems

Having been, unfortunately, a former user of Kaiser-Permanente's services, I can say with confidence that their problems span way beyond software problems. 7 years ago, I was misdiagnosed with asthma by 3 Kaiser doctors. A 4th finally agreed that I didn't have asthma (and I don't). My actual problem was panic attacks caused by excessive adrenaline production, which no Kaiser doctor was able to determine. I had to go outside of Kaiser to find a doctor competent enough to determine the actual problem and treat me appropriately (with beta-blockers for the adrenaline instead of mind-numbing drugs like Paxil and Valium).

Add to this the reports of KP dumping homeless patients on the streets. There was the kidney transplant scandal. The patient information becoming available online scandal. And all of this in just the past 2 years. Kaiser is clearly flawed from top to bottom. They are the poster-child for real health care reform in this country.

Epic Systems

Epic Systems-- selling $4b systems based on clunky, non-relational databases you've never heard of.

Interesting website ..

Interesting website .. [livejournal.com]

Kaiser sub-standard care.

My father had a heart attack about a month ago. He is a member of Kaiser. Kaiser has it's own hospitals, doctors, and entire network that you must use. His heart attack resulted in him needing to undergo heart surgery. (The advantage to this sort of system is that my father's heart surgeon performed over 1000 bypasses every year and as a result is in the top tier of experts.)

For me the biggest issue came when it was time for discharge. The handed me a hand written sheet with about 12 different medications scribbled in "Doctorese" cursive writing. I couldn't read half of the sheet, and only the briefest of notes for each medication was listed. I was shocked that I didn't receive a computer printout with the medication instructions, especially for medications needed to be taken in the case of quadruple bypass heart surgery.

After many hours of sorting through the medications, trying to figure out if he had already been given a dose at the hospital or not, etc... I realized that when I had visited a non-Kaiser emergency room for a sprained ankle I got a multi-page printout with very thorough instructions for the medication and aftercare. On another visit when I had 107 temperature it was the same thing. I was very certain of the doctors orders for post-visit care.

With Kaiser though I was left confused with short hand written notes about a dozen medications. Confusion is not a state I wanted to be in when I'm charged with the care of one of the most important people in the world (to me) who has a life threatening condition!