Don't Shoot Me, I'm Only the Software 392
ctwxman writes "How often have you heard about some massive crash and then the blame was placed on the software? "Disasters are often blamed on bad software, but the cause is rarely bad programming." If you've been looking to blame your boss, this article from MSNBC says your ship has come in! Poor planning, poor execution and poor leadership are more likely to blame than bad code when it comes to systems that fail. "
Also... (Score:2, Insightful)
As I recall, it also came from a tech who didn't do his job right in rebooting the machine that handled the software.
You can't always blame software; you have to blame the end-users too.
Summary of article (Score:3, Insightful)
Big projects require organization or shit happens.
Uh, that's it. Thrilled?
Re:Irony (Score:2, Insightful)
Management vs. Intelligence (Score:4, Insightful)
Management skated, two programmers fired.
That's why I raise cattle and write hunting articles these days.
Constraints (Score:5, Insightful)
Software projects seem to be primarily constrained by time/money which is usually controlled by management (read: boss)
If one wants to test software properly then you will need lots of the constraints (i.e. time and/or money). Just before a coder is testing his block, he/she will generally say something like:
"I'm finished the block, just need to test it a bit more"
Generally that is not what management will hear, they hear:
"I'm finished"
So they think "its ready". I've seen several first generation projects get hit by this problem (in commercial environments). In the IC design world (where its not generally possible to just flash the firmware to fix a bug) its accepted that at this point - i.e. primary design is finished you are only 50% of the way through. We spend at least half the time verifying the blocks. Management in IC design have accepted that this just as important as the implementation and so don't go off making wild assumptions.
So rather than just pawn off the blame onto your boss, it really is (partially anyway) your fault as well for not highlighting the fact that your block is not as tested as you would like it to be.
The philosophy of open source seems to limit the "its ready" effect to a good degree and hence the better code quality perception. When main stream commercial coding picks up the slack, it should get better as well. But generally a lot of these messes can be attributed to communication (person to person) failure rather than coder/boss failures.
Hmm.... (Score:1, Insightful)
Re:Also... (Score:1, Insightful)
M$NBC says $oftware is Good! Blame the user. (Score:5, Insightful)
Sorry Microsoft, it's the software. When I go to the local airport and see a kiosk displaying a Windoze 2000 screen saver instead of information, something is wrong with the software running the kiosk. I'm sure that the kiosk owner followed all of the directions given and the stupid thing did not work anyway. A box that has to be restarted once a month and crashes when it's not has a software problem. Having two of them will simply multiply the problem by a factor of two.
How am I so sure that software not people are to blame? It's easy, I started using non Microsoft software and most of my problems went away. I've got the same old hardware, it just works better under Linux. It does more for me too.
Why is that? It might be that there's no nasty registry that's designed to keep me from "stealing" software. It might be that sane networking models really do eliminate most problems with worms and viruses. It might be that free software really works to make better code. Who cares?
The bottom line is obvious. No amount of blame shifting will change it.
Re:Also... (Score:2, Insightful)
So in other words the life of airplane passengers is depending on the fact if a computer is rebooted manually or not. Thank god nothing really bad happened during this radio outage, otherwise some smartass would have blamed it on the tech that forgot to reboot.
The main problem is obviously we're relying on systems and procedures that never have been tested under emergency conditions.
So far I was never scared to board a plane, but now I am. Especially after learning that air traffic communciation relies on something that I abandoned at home because of security reasons.
If someone is to blame, then the authority that gave permisson to run such systems without proper testing. The question still arising is if this will have consequences. AFAIK there were 5 incidents where the safety distance between planes was violated... shouldn't the FAA invstigate this and enforce procedures to avoid those sort of incidents in the future?
One of the biggest problems (Score:4, Insightful)
And coders cannot be designers, DBAs, or possess much business knowledge. Interaction with the end user is done with a 'business designer'.
As with the childhood game of post office, some of the information gets lost for every node in the SLCD (sftwr life-cycle design) chain.
One of the best fixes is to allow direct interaction of coder/end-user, and merge the designer/developer roles for a better industry understanding.
Dilbert remains a documentary (Score:5, Insightful)
I used to think this. Then I realized that at least the developers knew one end of it -- they knew what the software can do. The other end, what the customer wants out of the system, is usually not known by anyone. Not management, certainly not sales, and not the customer either.
A customer with an existing system will often try to write requirements which amount to "do exactly what the existing system does in exactly the way it does it", which is not what they want or they wouldn't be replacing the system. Or, whoever is providing the business requirements will be so out of touch with their own business that the requirements will be incomplete or wrong. Or on the flip side they'll be so familiar with the system that they'll leave out things which are obvious to them -- but so obscure outside their field that no one on the software side will even notice the omission.
Of course, these problems will be discovered very late in the development cycle, resulting in a scramble to redesign and redevelop, a bunch of fingerpointing, mandatory overtime, and a host of other ills all of which lead to bad and buggy software.
Re:Irony (Score:5, Insightful)
good software engineering is required for large systems. when you are developing hundred thousand lines of code to million lines of code. no amount of good programming will guarrante a good system without solid software engineering processes.
Re:Bullshit! (Score:4, Insightful)
I don't think so.
Its not the software's fault, but lack of teamwork (Score:1, Insightful)
No
You get good software from good teams of business people and developers. Every successful project I've been on has been successful because of a good team comprised of users, "business" people, and developers. Add to a good team a good software development process and you have a better chance of success.
Snippet on blaming the developers (Score:5, Insightful)
Not surprsing that a CEO would make this remark. I can't count the times I've asked the business community I'm working with for clarification of a business rule or requirement, and then get a 'sigh' or other look that says - "I'm too busy to worry about this".
And on the contract I'm working on now, they consider a 30 min phone meeting enough information to build a full blown app - trying to get documentation is like pulling teeth. And of course we know where the finger will be pointed if there's any issues.
To say we're nerds who don't "get it" is just an asinine, condescending remark; a) I'm perfectable able to learn about the business involved, b) If you explain the rules properly most developers I know have no problem at all coding the solution. I find most of the developers I work with brighter than the business community they're working with. The CEOs remark has a dilbert-like quality to it imo, and this guy's one of the 'experts' on the problem in the article... ha!
Re:Buck Passers (Score:5, Insightful)
MS employees (Score:5, Insightful)
Amen to that. I don't know where this idea that MS doesn't hire skilled people to design and develop software came from, but it's wrong.
It has always appeared to me that MS hires top students from the very best schools.
bhj
Nevertheless... (Score:3, Insightful)
Nevertheless, it's those poor planners, poor executors, and poor leaders who are in charge. You really think they are going to take the blame? No, of course not! It's so much easier, more fun, and better for your career to tell upper management that it was just the programmers who couldn't follow their instructions correctly.
Programmers will then get blamed, the poor managers will get a bonus for "correctly" identifying the problem, and corporate America will sail on as it always has: giving the big bucks to the managers and sales folks, while ignoring the programmers.
Who me, bitter?
Re:Irony (Score:5, Insightful)
Welcome to every single software project ever.
Speaking as a senior software test goon . . . . (Score:5, Insightful)
Last month, a system that controls communications between commercial jets and air traffic controllers in southern California shut off because some maintenance had not been performed.
Yeah. That maintenance they failed to perform? It was their mandated once-a-month reboot of their windows system, because it locks up after 43 days.
This was the result of bad programming.
Anyway, as a QA guy, I can assure you that bad programming abounds. It's my job to make sure you never see it. Part of that job is trying to drill into programmer's heads the concept that performing to spec when used as directed is not sufficient.
Re:Irony (Score:5, Insightful)
If the builders get given a plan where the roof is placed underneath the house, they should question it, not just build it blindly without asking.
Daniel
Re:Buck Passers (Score:4, Insightful)
This is not something unique to IT -- it's something fundamental to any command structure which relies on communication between unequals.
It's only common name is the SNAFU Principle [jargon.net], which was coined by Robert Anton Wilson (there's a very good discussion of it in his book Prometheus Rising).
In Illuminatus!, a satirical study of social pathologies, Robert Anton Wilson and Robert Shea brought out an important principal that causes trouble in hierarchies: the Snafu Principle. People tend to say what they think the boss wants to hear, especially if they have noticed that the practice of ``shooting the messenger'' is common. This means that the information passed up the pyramid is distorted at each level. Thus, each higher layer of managers tends to have less and less contact with reality, and near the top they are often completely out of touch.
Considering you deal with users who... (Score:2, Insightful)
Re:Irony (Score:2, Insightful)
Will you people ever get off this silly juvenile crap? MSNBC and Slate have never demonstrated anything but complete editorial independence from Microsoft, and they (and MS) deserve credit for it, instead of constant sneering from the audience at VA Linux's propaganda mill.
Article is basicly right (Score:4, Insightful)
Hmm... I wonder if the execs running nuclear power plants have finished installing windows to run them....
Better yet, we can put windows in charge of the ICBM fire control systems. We'll be *so* state of the industry.
Re:Also... (Score:5, Insightful)
Re:Irony (Score:5, Insightful)
No, not every software project. The typical deadline for open source software is "when it's ready". Which often isn't an unrealistic deadline. However, the shitload-of-features problem can happen there, too (and is usually the main reason if "when it's ready" gets unrealistic).
Re:Buck Passers (Score:5, Insightful)
Programers, at least the good ones, know how to say this, and say it loud and often. Likewise, the true professional salesmen who have actually studied their craft know how to say "This promotional technique of yours has actually been proven in study after study to be a pointless waste of time." The cabinet builder knows how to say, "That joint will fail." The day laborer knows how to say "If you really insist I do it that way you'll get less real work per day out of me and I'll be out on comp in six months."
Managment knows how to say, "I'm sorry, but our policy is. . . "
They're often very good at saying it, because they get a lot of practice saying it, instead of practicing how to listen to the people they've hired because they possess certain special skills and knowledge.
Engineers know how to say, "The Space Shuttle will blow up if you launch below a certain temperature."
It turns out they were right, but managment followed policy.
Managment's solution? Fire the engineers for speaking out.
So long as you work for managment that views its role as telling you what to do, and your role to just shut up and do it, actually doing your job the way you percieve it as an expert is simply a short walk to the unemployment line.
Once upon a time my mother was the manager of vehcile registration renewal for the NYS DMV. Her superior walked into her division one day and found her cleaning a microwave oven.
"Do you think we pay people at your grade level to clean ovens?" he asked her.
"Everyone of my people is working on production, in the only profit making division of the state government. Do you want me to stop one of my people from making money for the state just to clean an oven when I've really got nothing better to do right now myself?"
Fortunately for her, after a moment of reflection, he actually got it and replied, "You know, I never thought of it that way before. I guess sometimes we do pay people at your grade level to clean ovens."
Both he and my mom are fairly rare examples of managment. Managment serves the purpose of making sure the secretary has paper clips when he needs them, the assembly line workers have bolts when they need them, and to gently nudge people back on course if they should happen to stray a bit from the path, not "tell them what to do."
But somebody has negelected to tell most of the managers that.
KFG
Re:Irony (Score:3, Insightful)
Mod parent up (Score:5, Insightful)
Re:MS employees (Score:4, Insightful)
Re:M$NBC says $oftware is Good! Blame the user. (Score:4, Insightful)
If you build an entire campaign around "you know who to blame when things go wrong" then you can't whine when people take you up on it.
Re:Snippet on blaming the developers (Score:3, Insightful)
I agree with you than in an ideal world you could go to management of some sort and get the situation fixed. But in my experience, esp as a contractor, it's usually an uphill battle not worth fighting if it's problem with the company's culture.
If the culture is screwed up, I politely decline a contract extension when the time comes. My job is to design and code, not fixed f*cked up management. I had one 3-month contract where I had 4 different managers in that timeframe, due to political power plays and all kinds of crap, there's no way to as a contractor to correct that kind of situation in that timeframe.
When I was wet behind the ears, I used to try, but it was an exercise in futility, so now I don't bother. I certainly will ask for what I need, but if I can't get it - I just document my requests, cover my a*s, and move on when the time comes.
Re:Buck Passers - Works Both Ways (Score:3, Insightful)
Sorry boss, you're getting paid to know. Spend some time (gasp! outside of work if you need to) and read up. While you're not expected to know every last implementation detail, you should understand the capabilities of your chosen platforms completely.
Re:Irony (Score:2, Insightful)
Well, we don't necessarily need to blame it on their affiliation with MS, poor journalism could also be the problem.
If I remember correctly, this "maintenance" was a monthly reboot which was necessary because the software (in this case Windows) would fail without it. You can blame that on poor management, poor training, lack of diligence, or faulty software.
The fact is, if the software wasn't faulty, at least in this case the rest would not have mattered.
On the other hand, the best managed company with the best training and most diligent employees could not possibly have fixed the problem with Windows. I suppose if your going to blame management for anything, it'd be deploying Windows for mission critical tasks.
Mission-critical? (Score:5, Insightful)
Saying you need regular reboots is the same as saying you need a firewall to protect against viruses: both show flaws in the design of OS.
And as far as "fscking their disks every day" goes, that's more sloppy design. You shouldn't have to do that. Fsck fixes file system errors resulting from poor application behavior, environmental problems, and (sometimes) hardware troubles. You shouldn't have those every day in mission-critical systems, but even if you do then putting in place a system of daily fsck is not the way to fix it.
I've had a production application server running for the last 288 days. It's due to come down for OS updates, but it will do so on my schedule, not because its operating system is poorly designed.
Re:Buck Passers (Score:3, Insightful)
The other problem is is that this type of 'message modification' can be done unconciously, from choice of words and adjectives, 'forgetting' to mention bad news, and even body language. Even subtle moderations can have an impact in a large heirachy.
It *is* possible to fight this, but what you then have to do is have a manager and the people they manage feel and act like equals. If there is no fear of reprisals for bad news, and no fear of reprisals for honest mistakes, then the quality of the communication within the organization will rise.
Really, the solution is to try and structure your organization as something other than a pyramid. You don't need to run an organization with a single "alpha male" at the head, and everyone reporting indirectly to that person or board. But that seems to be the cultural norm in western society.
Re:MS employees (Score:5, Insightful)
It has always appeared to me that MS hires top students from the very best schools.
That's not a Good Thing. Very few 21-year-olds, even those who got the best grades at the best schools, understand software design or business process well enough for a major company to be able to rely on them.
Real-world experience is an important factors in successful design, and it's something that can't be taught in a school.
As smart as each new class of new direct-from-school hires might be (and I've known several, and they were all brilliant), Microsoft would probably generate higher-quality software if they hired 35-year-olds with a dozen years of experience at other successful software companies instead. Of course, it's going to be harder to find 35-year-olds willing to work 60-hour weeks in return for $45 grand, a free bike, and all the soda they can drink...
Re:Bullshit! (Score:3, Insightful)
Re:D*mnd if you do and D*mnd if you dont (Score:3, Insightful)
I have done CMMI and I am not a fan of heavy process done in the heavy handed way CMMI is done. Its a great way for extra levels of management to justify themselvs. That much said, CMMI does ask developers to do important things (and don't quote me, this is just what it means to me):
1. Coding standards
2. Unit testing (automate it!)
3. Peer reviews
4. System testing
CMMI makes you do all of that and document it. The paperwork is over the top but the result is better software.
The big winners are having unit test code that aims for 100% coverage and real peer reviews (the many eyes approach). The peer reviews can be initially painful but everyone learns their bad habits quickly and soon gets out of them.
The downside is that it comes with a whole crew of managers, inspectors, auditors, validators, finger pointers, and beurocrats who are working hard to justiy their existance by beating up the developers and their code as much as possible.
In the end, it all comes down to common sense: plan what you are going to do. Do it carefully. Test it. Ask your peers to review it and learn from their comments. Put it all together and test it again. Remember your problems so you don't repeat them.
Re:MS employees (Score:1, Insightful)
They make them.
Give the brightest graduate a few months in MS's insane corporate culture and they'll be explaining the advantages of
Too many suits who haven't read F.P. Brooks (Score:5, Insightful)
Re:Irony (Score:4, Insightful)
Re:Irony (Score:2, Insightful)
Never ever?
Perhaps you haven't been paying attention. Even Diogenes, living in his barrel, paid more attention to the world than that.
Open your eyes (Score:5, Insightful)
It has always appeared to me that MS hires top students from the very best schools.
That is true--unfortunately they have been known to hire them AWAY from the best schools too (ie. before they graduate). It doesn't matter if they are top five percentile students--if they have zero practical experience and are thrown into a situation beyond their capabilities the result can be less than ideal. Nonetheless, I think that by now MS has figured out how to select and place recent grads and students hired before graduation. I think the problem is now deeper than that.
Microsoft triumphed over other tech companies that were prominent in its early days because BillG learned it had to become a marketing company (the same reason Apple still exists today--Jobs knew that from the start and Gates is a very quick study). Other tech companies remained software companies--they toiled away to make their next killer app the best it could be and marketing was an afterthought.
At Microsoft, from 1980 on at least, has been a marketing comapny first, with software development second. The most important technology it markets was invented elsewhere and merely extended by Microsoft. Only in the company's latter life have they been truly serious about research. The long time "thinkers" are brilliant but historically little has come out of Microsoft's research that has been commercially successful given the potential funding power MS has had.
Therein lies the problem. The article is right--software isn't the root cause of the vast majority of failures (even when the failure is the direct result of a software bug). At Microsoft, software design is driven by marketing--time deadlines, customer requests for features, backwards compatibility/legacy support etc. The result is the house of cards we build our systems upon today.
That result is unavoidable without EXTREMELY skilled planning and throttling the pace of change. Unfortunately, The MS Ship sails where the winds take it, and the pace of change has been rapid and relentless until now. I once thought the problems with MS products were because too many drop-outs were running the show. After seeing this blog [asp.net] I can see what the development teams have had to cope with. They have to do the impossible and try to get it done before the deadline slips yet again and MS market cap slips a few million and BillG comes down to yell at them. In some cases you have to be brilliant just to survive at MS.
So anyways, I think software bus are the immediate cause of a lot of disasters, but the ROOT cause definitely is poor planning and project management that leads to unstable system development.
Re:Buck Passers (Score:2, Insightful)
While building a show, you have a director who basically controls everything and everyone. He says 'stand here, put that there, turn that light down'. (He hopefully delegates some of that.) This is how some bosses seem to act...but they're managers, not directors.
When the show is ready to open, he's built everything exactly how he wants it, and he then turns it over to the stage manager, who then proceeds to, hopefully, do nothing except call out 'Okay, do the next thing on my mark. Mark.' repeatedly.
Of course, as we're talking about something that, no matter what anyone does, seems to repeatedly involve actors, said stage manager will spend a good deal of time locating them. Or something equally silly, like lighting system tripping a breaker, the curtain failing to operate, or a fly that won't fly.
So, by analogy, the board of directors of a company should 'build the show', and then hire managers to make sure things keep running.
Re:Mod parent up (Score:3, Insightful)
Nowadays there's no revenue from quick fixes; the only revenue is from the initial sale. So theres a relentless pressure to ship something, ANYTHING, to keep the sales going. If you don't crank out the versions often enough, you could run out of money because the market's full of what you have today and you have no other source of income.
Some vendors offer per-incident paid support, but my experience is that that never gets used, because of the hassle involved in getting approval for unplanned expenditures. Figuring out a workaround in-house may cost more, but the cost isn't so easy to see and there's no delay while someone ponders whether to spend it.
Re:MS employees (Score:2, Insightful)
How is that still not a Good Thing? You start with someone who is smart and you teach them how to be a developer. MS is all about employee retention, so over time, they have not only smart employees, but ones with solid experience.
And by the way, they don't toss newbs right out of college right into major development roles.
Let me guess, who weren't one of those who got the best grades at the best schools.
Re:Dilbert remains a documentary (Score:3, Insightful)
One approach to minimizing the tendency to state soltuions that I've seen promulgated in a few places is to tackle requirements definition from a different perspective. Rather than expressing requirements in terms of "things the system must do", write them in terms of "problem symptoms that must be suppressed" (i.e. "what do I want to chnage about the current system). This isn't a panacaea, but it does seem to help get people thinking in the right direction.
The fundamental problem, of course, is that problem-definition is just one component of the larger skill of problem-solving, which just doesn't seem to be taught at all any more. Most schools seem to focus on "training" and cookbook solutions (which is what most students seem to want too), rather than on "education" and problem-solving skills. It's particularly annoying that problem-solving skills aren't taught to management types, since (to me at least) a large portion of their job is supposed to about identifying problems (in resource allocation, scheduling, market strategy, inter-personal dynamics, etc), and solving those problems.
Your analogy is crap (Score:3, Insightful)
This isn't about a standard maintenance procedure, since a server should not have to be rebooted constantly in order to maintain. stability/functionality. That's like saying it's ok to swap the oil every second flight because it's cheaper than fixing the actual problem (that there's a leak in the first place).
And actually, considering that many earlier windows problems were caused by memory leaks... not such a bad analogy now...
Re:nobody yet knows how to design software (Score:4, Insightful)
No they don't. It's been studied for decades, but in all that time we've still not settled on anything that's actually demonstrated over long periods of time to be good. Hardware, materials and other available resources are continuously evolving and changing, meaning that software design research has nothing to reliably settle on before things change again.
We don't even have consistent and proven programming languages. Today it's Java, C#, VB and a variety of imperative scripting languages. Yesterday it was C and C++, before that it was Fortran, and before that there have been variants of assembler. And as we use these languages, we're constantly discovering more and more about language design and developing new languages.
HCI is still in very early stages of development, and that's a major part of software engineering. (If people can't use software then what's the point?) The vast majority of software development shops -- particularly smaller ones -- don't even employ HCI experts, and substantial proportions of developers still don't respect them or understand what the point is.
Something like bridge building, for instance, has been studied for centuries (if not millenia). It relies on consistent physics, consistent tools and well understood environments. Organisations that build bridges have well established experience, procedures and regulations that are put in place throughout their organisation. Software development's been studied for a few decades with the existing materials, resources and expectations constantly changed from underneath it.
Organisations that build software still don't have any reasonable idea of how to arrange themselves, or what procedures they should be using. There have certainly been some pretty good ideas from relatively recent ongoing studies, but the fact that managers and developers and marketers and whoever else frequently don't gel together very well with usually bad results is just an ongoing consequence of the fact that it's a very new field.
Just because software engineering has been studied for a few decades doesn't mean we know what we're talking about, or even that we know what we're studying.
Re:Irony (Score:2, Insightful)