California Cancels $208 Million IT Overhaul Halfway Through

g01d4 writes "According to the LA Times, 'California's computer problems, which have already cost taxpayers hundreds of millions of dollars, have mounted as state officials cut short work on a $208-million DMV technology overhaul that is only half done. The state has spent $135 million total on the overhaul so far. The state's contractor, HP Enterprise Services, has received nearly $50 million of the money spent on the project. Botello said the company will not receive the remaining $26 million in its contract. ... Last week, the controller's office fired the contractor responsible for a $371-million upgrade to the state's payroll system, citing a trial run filled with mishaps. More than $254 million has already been spent.' It's hard not to feel like the Tokyo man in the street watching the latest round of Godzilla the state vs. Rodan the big contractor."

Sadly

[mu51c10rd]

And this is the state that has Silicon Valley...you would think there would be a lot of good expertise in the computing arena for the state to tap in to. However, in their defense, this happens constantly in the federal government too. So much money wasted...

Re:Sadly

[Teresita]

Same thing will happen to the high speed rail, and California will have a $50 billion dollar track between Hooterville and Jerkwater.

Fallacy of Sunk Costs

[Galaga88]

I'm glad to see that they didn't fall prey too badly to the fallacy of sunk costs [wikipedia.org]. Too many places wouldn't realize they've already lost the money they threw at the project, and no amount of extra spending in the hopes that it will eventually succeed will get that back.

Re:Sadly

[TFAFalcon]

Why finish a project someone is paying you for? Do enough (cheaply) to get 50% of the payout, get fired, form a new company and get hired again.

Re:Sadly

[doctor woot]

you would think there would be a lot of good expertise in the computing arena for the state to tap in to.

Ahahaha, with our government? If they even had the slightest idea of how important it was to stay out of the fucking 1980's with IT equipment that serves critical functions for the state and its citizens, they wouldn't have waited for the problem to "cost taxpayers hundreds of millions of dollars" to do anything about it.

If they can't get that much straight, how can they possibly hope to know what technical criterion to look for when hiring contractors?

Re:Typical of the Federal Government too

[Atrox Canis]

Umm, I thought the AC post was at least on topic since he used the word "too" in the title. Just sayin'.

Re:Typical of the Federal Government too

[tibit]

It's not about the government. It's all about the useless IT consulting companies. Pretty much every single flashy consulting company billboard/AD that you see at an airport is just a way to milk the gullible and not deliver. This is an across-the-board problem. Nobody wants to fucking do their jobs. The government thinks they don't need the right people to do it, so they hire a contractor. The contractor doesn't want to do the job either, it's not their core competency (nobody knows what it is anyway), so they hire subcontractors. Subcontractors have very little vested interest in anything, and they maybe deliver, maybe not, but due to multiple layers of clueless management, it's of no use anyway. So there you go.

Re:Typical of the Federal Government too

[ogar572]

Pretty much sums it up.

Re:Sadly

[Nerdfest]

Doesn't that make it IBMs turn next?

Um, math?

[DigitalSorceress]

I was going to use my mod points to mod up the first person who questioned the new math behind how a $208 million dollar project cancelled halfway through already cost $254 million dollars.

Alas, nobody had yet... and it's just about beer-o-clock here.

Re:Typical of the Federal Government too

[MightyYar]

I thought it was an "emoticon" of a large breasted woman giving the finger.

Why $208 million?

[maroberts]

Okay, the system presumably has to handle about 30 million drivers and vehicle statistics, as well as other information such as traffic citations. I assume it's only accessed through a few hundred offices plus allow access to authorised systems (police etc) at any one time. Obviously it's got to be reasonably secure and perhaps operate at more than one site to cater for disaster recovery and redundancy. This is not beyond the capabilities of a few large servers to handle (I presume that cloud storage may be out due to security issues). Such a system could supply the information to Windows/Unix or even phone app clients. I assume driving licenses and vehicle ownership records have to be printed and sent from an office somewhere.

What else is in the scope of the project? Why does it cost several hundred million bucks to develop a new system? I can understand perhaps 10 million to develop and install. The biggest problem I can see is porting the data from the "40 year old antiquated system" to the new one. Someone must be able to explain where the extra £198 million has to go, apart from the contractors pockets.

Government + Consultants = Failure

[Tridus]

This is entirely normal when you take a government that chronically under-staffs on IT and relies on consultants. They go and try to do something big, and they don't have the expertise in house to deal with it. Enter more consultants, particularly of the variety that like to write a lot of powerpoint presentations and bill a lot of hours but never actually deliver a bloody thing. Of course, since the government doesn't have enough IT expertise to actually figure that out, the high level senior managers that love powerpoint and high-level mumbo jumbo MBA talk think everything is going well.

And then, scope creep happens. It follows one of three lines:
1. Election happens. New government comes in, with new priorities and a new way they want to do things. This is obviously bad for a huge project in progress.
2. The existing project has a new department join in, which means new managers and thus a new set of demands. Instead of starting up a new project, they try to shoehorn those into the current project to satisfy management's desire for design by a giant committee of managers.
3. Someone realizes that the project didn't actually have all the requirements properly captured in the first place, which is pretty much inevitable in my experience.

You'd think at some point the government would learn that they can't manage projects in this way and rely on consultants to sort it out, but they never do. Of course, in the case of #1 or #2 even in house IT doesn't really save you, but in my experience they tend to be more flexible than a giant Enterprise consulting outfit (mostly because there's no contract they can hide behind to deliver X, even if X doesn't actually solve the problem that prompted the project in the first place).

The whole process is a giant mess.

Re:Typical of the Federal Government too

[Nadaka]

>95% of failed, past due or overbudget IT projects are a result of insufficient, incorrect or everchanging requirements from the customer organization and the people on our side who interface with them. It is a result of people thinking that computers are magic, large software projects are easy to change completely after they have been nearly completed, and people will understand what they mean instead of what they say.

Re:Nothing to see here...

[TechyImmigrant]

>"The DMV project began in 2006, according to the California Technology Agency. Instead of using 40-year-old, "dangerously antiquated technology," DMV staffers were supposed to get a modern, user-friendly system that minimized the risk of "catastrophic failure," according to a DMV report on the project."

This encapsulates solving multiple problems at the same time. This cannot be done. You update large systems by plotting a path through incremental improvements that get deployed, tested and fixed before the next increment, so that get you to where you need to be. It might not seem like the optimal path, but anything involving a switch over of technology, UI, back end, infastructure and buckets of code all at the same time is simply never going to work.

In the case of the DMV, it might involve unifying disjoint databases pair by pair until you have only one, while maintaining the same interface to the heterogeneous clients. Then one by one converting the heterogeneous clients to a standard back end interface. Then one by one adding the features of a client to the grand unified client and switching over that system, until the GUC has all the features for all the clients are new client. Then one by one, updating the organizational procedures to make them better, and updating the GUC while doing so. You can make these changes one by one. You can roll back one step if it doesn't work right the first time. You can measure progress by the number of working updates, not in how much less non-working the global-replace-systems is today.

Re:Typical of the Federal Government too

[Virtucon]

Oh good grief what a bunch of bunk. Do you know why big contracting companies exist? It's because companies have lost faith in their internal organizations to deliver. Plain and simple. I've seen it in dozens of organizations in my career where an entrenched group builds a castle of "can't." So, the execs hire an outside firm not only to get what they want but to also force these little castles to actually deliver something. Lazy contractors? Please, how about lazy employees who feel that they hold the keys and as long as they keep pushing back and feel empowered so nobody is going to mess with their careers even though they may be doing such innovative things like writing System 360 Assembly Language. It's everywhere and it's not just in IT, people in this country have become lazy and foolish relying on attitude rather than customer service and trying to do a good job.
Every large organization has this problem and IT isn't just one of the areas where dead space can occur. So, the big companies come in, push change, make big promises that sometimes are overblown. In the case of California I could probably guess that the specifications of what were required were done by bureaucrats who have no clue on how to spec out requirements or were based on something that wasn't possible to build. Are there bad firms? yes, but are there bad customers? hell yes and they can make it absolutely impossible to deliver anything because the same people who have to approve or test anything are usually overworked, or not committed to the project leaving the contractor and subsequently the whole project in limbo. That usually leaves to failure despite the best efforts of all parties involved.

So before you blame consulting companies for this failure, remember they wouldn't exist if people were doing their job or came to the realization that their skills and abilities are out of alignment with what their management expects.

Re:Sounds familiar but on a smaller scale...

[PRMan]

Or that HP sucks as a software company.

Could it be the government agency's fault?

[cyberidian]

Could it be that the way the government contracts are structured and micromanaged by government agencies is the problem and not the contractor or their programmers? I work for a company that provides government services under contract to the State of California and the government agency that oversees us micromanages us so much that it is often impossible to to develop systems properly. The 4 biggest problems I see are 1)constantly changing requirements that are written by government employees with little or no IT/web knowledge 2) contracts secured by being the lowest bidder which do not allow us to have the resources to properly design or test the system we are building 3) forcing us to work with other contractors including non-profit ones that are "donating" their services (very strange to me really) and that provide inferior IT systems we must use or integrate. 4) Requirements, features and design being dictated by government agencies or advocacy groups with little knoweldge of system design & development. For example, we are currently forced to support an application written by one of these "non-profits" that uses ASP classic and violates every current IT standard. My company has the IT staff & talent to completely rewrite the application but we are not allowed to and must instead support and integrate the badly written one that was donated to the state. It is unclear why this non-profit is allowed to force the agency & us to use their product, but it seems they have political connections that make it so. I believe also that government contracts almost always go to the lowest bidder and not the company with the most expertise. Often a contractor is the lowest bidder because they plan to cut corners and not follow good IT practices, or have not estimated costs correctly. Also as a web developer for a company that works under government contracts, I cannot count the number of times we have received requirements for a website from people that have little or no computer skills, let alone web skills or experience. You would think in this day and age that the government employees providing requirements for government IT systems would have at least basic IT knowledge, but this is often not the case. I am not exaggerating that I have received requirements from people that have no Excel, Word or even email skills and have obviously barely even used the Internet. Many people in the top levels of government management are older (baby boomers) or were promoted for reasons other than great IT skills. They often have no professional experience with developing IT Systems, ADA or other required standards and yet they are the one writing the criteria for the contracts and the system requirements. State agencies also often demand that large amounts of money be spent on "usability studies" or other commitees where a lot of people discuss and dictate what the IT contractor should do in building the new system. The people running these studies often have very poor IT skills themselves and have little experience designing IT systems, but they often have an enormous say in how the system is designed. By the time the IT contractor's development staff is involved in the project, everything is already specified by non-IT government people and between that and the contractor management trying to save every dime (therefore not providing resource for testing), it is not really possible to build a quality system. I say all of this inspite of the fact that the State of California actually has a good Department of Technology Services that provides great ADA compliant web templates. The California State government is so large that even with a good DTS department, the management and staff at specific agencies providing the requirements for a new system may have no knoweldge or interaction with that department and never involve them in creating the contract or project requirements. I think the solution to this is the state should be involving its DTS department in creating all contracts and requirements for new systems projects and ind

Re:Typical of the Federal Government too

[sakshale]

The problem is that you are both right!

Just look at the new California high speed rail system and how it went from a simple concept to a bloated monster.

Both teams are screwing up and they will both blame the other team for the failure.

Re:Typical of the Federal Government too

[Lije Baley]

There are some cases also where this misalignment of skills and management expectations is more of a management deficiency. Many organizations have technical people who are quite willing and capable, but they have been pigeon-holed and beaten down by policies which incentivize apathy. I have worked with long-time developers in quasi-government jobs who have skills only on legacy systems, and I have had the pleasure of helping them participate in the development of modern SOA interfaces. Most of these people just need an opportunity to learn, grow, and feel like their contributions will be meaningful. And it is not expensive, in fact if it looks expensive you are doing it wrong! You don't need or want conventional "training" for them, and if it is done right, it can cost little or nothing in extra time. These people have a goldmine of lost productivity to tap into -- productivity that poor management has beaten out of them and that good management can cash in on.

Re:Sounds familiar but on a smaller scale...

[roc97007]

Or that HP sucks as a software company.

Or maybe it turned out exactly the way they intended. I mean hey, HP made $50M on the deal.

Re:Typical of the Federal Government too

[erp_consultant]

"If the customers don't know what they want, you teach them" - Umm...if the customer doesn't know what they want then the project is in deep and immediate trouble. It is the customers business to run and if they don't know what they want they had damn well better figure it out before they spend a boatload of money on a team of consultants.

I have worked on many large projects, some with government agencies some with large private companies, and the most successful ones have a common thread. The common thread is that the critical project decisions are a collaborative effort, not simply punted to the consultants to say "here, you figure it out". If a company has management that is unwilling or unable to make decisions that company will fail. I don't care how many consultants you bring in, if the management cannot articulate how the business is currently run and what the future goals are the project will fail.

Are there shitty consultants out there? Sure. Just like there are company executives that have no business running a lemonade stand, never mind a multi-billion dollar company or large government office. To succeed you need talent on both sides.

Some of the government IT systems that I have been exposed to are arcane, to say the least. Often they have little or no documentation and are extremely complex. Many of the agencies are under funded and under staffed. Talent and motivation, frankly, are sometimes lacking. Basically it's a difficult environment to work in.

Really big projects like this have a high failure rate. There are a multitude of reasons, some of them above. I suspect that both sides share in the blame.