What Makes Software Development So Hard?

lizzyben writes to mention that CIO Insight is running a short piece that takes a look at why the rocky culture of software development continues to exist despite all of the missed deadlines, blown budgets, and broken promises. From the article: "I was not really looking or thinking about big software projects. I was just coming out of my experiences at Salon, where we built a content management system in 2000, which was painful. I was one of the people in charge of it, and when the dust cleared, I thought, I don't really know that much about software development. Other people must have figured it out better than I have; I must go and learn. So I started reading, and talking to people, and realized it's a big subject and an unsolved problem. And the bigger the project, the harder the problem."

Re:good question

It isn't. Haven't you seen them, all they do it wave their hands around and look silly. That's the point. Everyone thinks it's a breeze when it isn't, so everyone underestimates everything.

The real point is...

...that software development needs to have a team of programmers who are talented and competent, and a project leader who knows his people and can effectively lead and manage both the team and the project... and a corporate culture which doesn't treat its people distrustfully like a bunch of indentured servants, then they can write good software and get it done in reasonable amount of time. Getting and retaining a team of very good talented and experienced programmers is very difficult and expensive however since most development firms are cheapskates and not only do not pay enough, but treat their people like dirt, binding them under all kinds of bullshit NDA and non-compete contracts, poor work environment with noisy cubicles or "bullpen" office areas, not empowering them with latest technology workstations and tools, etc. The software development company that teats its programmers golden is the rare exception... the exact opposite "sweatshop" like I described above is generally the norm, as I've seen it in my 20+ years in the biz, hence you usually get kids right out of school as entry-level positions doing the bulk of the development. The best cream of the crop programmers have mostly all quit programming years ago and now are all sitting in management positions and programming no longer.

A good orchestra conductor who is in front of a bunch of rank beginner inexperienced musicians will not be able to make very good sounding music. You get what you pay for.

Re:The real point is...

The week before a concert is absolutely brutal, especially if you're rehearsing with a band/orchestra (but even if you're playing solo piano and rehearsing on your own). You're treated fairly well at other times (most importantly, after the performance), but you can sure feel like dirt during all of the rehearsals.

I guess it's like being treated with respect as a programmer, except that you still have deathmarches.

Your analogy is good, though - hire amateur musicians, and you're not going to get a good outcome. If the conductor knows next to nothing about music, you're not going to get a good outcome. If the instruments the orchestra is using are not tuned well, you're not going to get a good outcome. If you rehearse a piano part for weeks and the conductor suddenly asks you to play the oboe instead five minutes before the concert, you're not going to get a good outcome...

Re:good question

Well, if you wanted to get really technical about it, the conductor's work is mostly done before the performance, so that any reasonably proficient orchestra would be able to do a reasonably good run sans the conductor.

Conducting, however, is a lot harder than it looks, especially during rehearsal. It requires the simultaneous processing of the score (possibly twenty plus lines going on simultaneously), time (not so easy once you get into funky metric changes), expression (precisely what the conductor is all about) while keeping track of mistakes that happened during the piece even though not stopping immediately. It's extremely easy to see when a conductor is inexperienced, boring, or hasn't looked over their music properly.

For conductors of student groups, they also have to keep the members of the ensemble engaged through tasteful storytelling while not completely going off tangent, they must be extremely creative in figuring out how to get their point across, they must be careful not to over-rehearse a section, etc etc etc.

Ask any marching band student turned drum major. Being a good musician does not mean you'll be a good conductor, and a more generalized notion is the core meaning of most of the other analogies offered by Slashdotters around here.

Re:good question

Wow,sounds a lot like running a project!

Simultaneous management of multiple requirements (possibly 5,000 competing business requirements competing simultaneously); time (not so easy when the scope of the project is creaping, but the deadline isn't, or the estimating process was off, or the requirements have changed.....) while keeping track of 1,000 action items, issues, defects and delivery dates at the same time. It's extremely easy to see when a Project Manager is inexperienced, or undertrained, or bored, or being overworked into an early grave!

For project Managers there is the ongoing need to maintain team cohesion, and team spirit, and still get the code out the door - tested and working perfectly, on time and under budget.

Like musicians, being a good programmer doesn't mean that you'll make a good Project Manager. Some of the best programmers out there would be dead in the water if put in charge of a real project, for no fault of their own, other than being totally unequipped to run a project.

As for delivering on time and on budget, it is a well known truism that the first 90% of the job takes 90% of the budget, and unless you are REALLY lucky, it's likely that the other 10% of the job will also take 90% of the budget....but I digress.

Re:good question

Why is herding cats so hard?

Re:Cats and penguins...

It makes one wonder why the Linux mascot is a Penguin?

Indeed - it should have been this [emory.edu] - it so appropos in so many ways.

Captain obvious

Herding cats is hard because you are using the wrong management technique. You herd cattle (and sheep and goats and pigs etc.), you do *not* herd cats. Cats, you put them in the general area of the mice and let them do what they are good at. Micromanagement of cats is a losing proposition.

Software development is hard because....

... of all the hot babes running around the offices in miniskirts giving massages that make it tricky to type. That, and the martinis that keep spilling in the keyboard. Combine that with the constant parties, sailing trips, ski trips, etc and it's a wonder anything gets done.

What's that you say? It's not the .com bubble anymore? Glad I left software.

Re:good question

You are close, but the real piece you are missing is that most of a conductors work goes on long before you see them up on stage conducting. The real work is in music selection (you have to know your orchestra and what its strengths and weaknesses are), properly managing rehearsal schedules (what groups should meet, and what they should work on), choosing the right players and the right section leaders (the latter is both a delegation and a political art), and the actual concert conducting is much more show than anything else (unless things start to go wrong).

I know a player in the Philadelphia Orchestra and he tells me that they largely ignore the conducting that happens during a performance (granted that conductor's stick work is impossible to precisely read). They know how he wants the piece played because of what they did in rehearsal.

Oh... and I do have the experience to comment on this, I was in orchestras for 12 years.

Re:good question

The thing that orchestras have that most programmers don't is a program design (the score) that they're working with their peers to develop (play) from. The composer knew what he wanted. Most users don't.

Re:It's not hard

Spoken like someone who hasn't ever been involved in the development of a large software project.

Large projects bring problems with them that aren't noticeable on small projects. The working set of my project is around 10 gigs, most of which is code and text files. The tree changes quite frequently, and syncing to that tree is painful.

What makes it more painful is when the tree is broken. So we had to develop tools to help ensure that the tree isn't broken, and that we have a way to tell what the last known good submission was.

There's performance issues related to the source repository, because no matter what repository you're using, they all have issues when you have 200 people working in the same place at the same time. (This is true of virtually any database application).

Re:It's not hard

Exactly. Anybody who thinks it's easy to complete a large software project without problems has obviously never worked on real software. Secondly, anybody who asks why it's so hard to develop good software has never worked on a large software project. You can say a lot of stuff about following processes, and doing testing, but until you're working on a large project, with lots of different developers, with time and money constraints, you don't know what you're talking about.

Re:It's not hard

In my experience the only time it becomes difficult is when people are NOT doing their jobs.

In that case, you have essentially proven that you have no experience with large software projects. Even if everybody is qualified for the job they are hired to do, and are enthusiastic about the project, and doing the things they are supposed to do, things just don't work out right at first try. Or second try. Or third try. Or... you get the picture. And then, when that problem's finally done with, you have 7 new ones, where at least 3 of them are total show stoppers.

It's of course easy to start pointing fingers at people. That guy is an idiot. The management has no idea what we are doing. Writing documentation is killing my productivity. I had to rewrite this assholes code since he didn't follow my favourite bracing style. And so on... The point is, the guy you call an idiot probably knows three hundred times more than you about fluid dynamics, which is what all the project is about. Management isn't supposed to know what you are doing, they are supposed to handle budgets, equipment, hiring, firing, etc. And the last two are kind of obvious...

Software development is hard, because it's not a solved problem. You don't build software the same way you build buildings. There are no rigid rules to follow, no best practices that can be universally agreed upon. The purpose of each new software project is to solve (a) problem(s) that has never been solved before. And because of that, there are great uncertainties involved. You can guesstimate a lot of parameters, but eventually, some of the unknowns are going to bite you in the ass. (As Rumsfeld said: There are known knowns...)

The only other problem I've encountered with software development are when people have unrealistic expectations, despite me telling them exactly what to expect and delivering on exactly that.

Not exactly an easy person to work with, are you?

Re:It's not hard

I have no idea, but according to his blog, he can't adjust his alarm clock. So, my guess is he's in management somehow. ;-)

( http://www.makesitgood.net/ [makesitgood.net] )

Re:It's complicated

There are a number of problems:

1. Clients rarely know what they want. Most software projects are designed and written for someone else's requirements. These requirements often come in the form of "I don't like that, but I have no idea what I actually want."

2. Coders are systematizers, who tend towards the asocial end of the spectrum, which is why silicon valley produces so many autistic and Auspergers's kids. Large projects require communication, and there are a lot of coders out there who may be good with machines but are lousy with people.

3. As a result, lead programmers are often tantrum throwing prima-donnas, intellectual bullies who use their position to undercut those under them and make themselves look superior. You may have to promote and fire half a dozen programmers before you fill the lead programmer position with someone who is actually good at the job. In the meanwhile, you've just lost 5 good programmers. And if you actually manage to find a good one, you'll have to pay him in blood, because he's probably worth more than your house.

3. Salesmen, who deal with the client, are professional bullshitters, who are generally wafer thin on the technical details. That's actually not a slam--this is literally their job--to sing whatever lullaby the client wants to hear. Programmers will generally learn their lesson after being burned a couple of times and will not promise the impossible. The sales people never get burned--it was the programmers who screwed up, right? Calls for heroics on the part of programmers invariably begin with someone in marketing.

So, a few tips:

1. Get someone to hammer out the details of the specifications before signing the contract.

2. Get someone with some technical knowledge to study those specs, and the deadlines, before signing the product.

3. Get a lead programmer who fights for his team, not for the himself, and who is more interested in getting things done the right way than in getting them done his way.

4. Get a project manager (preferrably female--tends to calm the more socially inept coders, less testosterone poisoning) who handles communication between the coding team and the client. Do not, repeat DO NOT allow the coder to meet the client. They will eat the client! You will have a hard time getting paid, and hiding the bones is a bitch.

It's design not development

Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.

Re:It's design not development

Well no one really teaches software design do they.

You can go to school to learn to be a bridge builder and come out of it knowing all the exact specifications to build a bridge and probably design a fairly good bridge, or with a bit of creativity and some extra architectural skills a really cool bridge. Software design isn't really taught in this manor, sure your taught how all the bridge building tools work, and even a lot of the engineering specifications. But I have yet to see the software design school that covered more than a class or two into truly how to design software. Then again, we've been building bridges for thousands of years, and designing software systems for a few decades. It does take time for these things to really get figured out.

Re:It's design not development

That and software is more like art than engineering, but no one wants to admit that.
Of all the Renaissance painters that ever painted, only a few became forever famous.

Same thing with coders.

I personally would use the term "Craft" rather than art because I have always believed that art is about "saying" something through your craft which I don't think is really the goal of software development. The truth is that anyone can be taught to be a reasonably good muscian, artist, or blacksmith and with proper motivation anyone can become great. I would say that there are two problems currently, schools focus on the wrong things (how often did you gather requirements, properly design, implement to given specifications, effectively test, or maintain a program in your school years?) and the industry seems to focus more on technology rather than skillset (does it really make that big of a difference to program in C# as compared to Java?); the result is you have someone who is trained to use a hammer designing your house, and you ask someone if they know how to drive a "Ford" when you are interviewing a truck driver.

Re:It's design not development

We've had a couple of thousand years of practice building bridges, and bridges solve a known problem: connect points A and B when there's a gap or a river inbetween.

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

Re:It's design not development

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

That was yesterday....We'll be sending you the updated specs soon.

Re:It's design not development

That was yesterday....We'll be sending you the updated specs soon.

There are lies, damned lies, and software specs.

Re:It's design not development

Software engineering is "Do X, Y, and Z except when A, B, but not C except when A and B and when user isnt E except when user is E and A and B and X and Y are not Z but Z is B when F is ..."

Close, but not quite. software engineering is "I want it to do blassie. Make it do blassie by yesterday." Where 'blassie' is a loose concept and the developer has to determine and define the underlying logic.

-Rick

Re:It's design not development

For the love of God! If you want us to troubleshoot your code, you're going to have to use some meaningful variable names and stop putting multiple operations on a single line!

Re:It's design not development

> you're making it sound easy and simple

The execution isn't simple, but the problem is pretty well defined. You don't generally get people asking for exit and entry lanes in the middle of a bridge, or asking it to accomodate aeroplanes, or asking you to build them out of styrofoam just because some magazine said styrofoam was the hottest new material. Or deciding that each piece of the bridge should have a special connector that requires new tools, or finding that your welders suddenly don't work when you choose a different brand of paint.

Software engineering generally sucks because it's surrounded by suckage all around. Programmers are trained and treated as wiz-kid hackers who can do everything ad hoc, projects and tools are designed that way, and it's no surprise that the products come out that way too.

When it's all said and done, it's probably harder to build a good large bridge than a good payroll program. The thing is, most bridges are eventually finished someday.

Re:It's design not development

Writing the code from a good design is easy. It is creating a good accurate design, capturing all the requirements accurately and ensuring the end user's expectations are correct.

The problem is:

• There is no such thing as "all the requirements." The only thing you can count on in software development is that the requirements will change
• The end users have no idea what to expect or what they want. They may think they want X, but they really want Y after seeing what X looks like.

Re:It's design not development

But people aren't willing to wait or pay for engineered software. Windows already costs $199, and a real OS that followed engineering principles wouldn't cost $199. It would cost probably around $10,000 per copy, if it were being sold on every desktop. And it wouldn't look as nice, and wouldn't do all the whiz-bang things that windows does. You can get engineered software, just ask NASA, but don't expect that grandma will be running engineered software on her $299 Dell box any time soon.

Re:It's design not development

The last project I was on which had requirements that didn't change at all during development was 7 years ago. And it was built right on schedule. For me it's been the ever-changing requirements which delay projects the most. The other half of the problem is usually bad estimates, but that's almost irrelevant after the requirements change.

What makes software development so hard......

Viagara, Cialis, red bull, two brazilian hookers and a swiss midget.

One word: Skill, Talent and Knowledge

Anyone can write a book. But, writing a book that's *GOOD* and that people will want to read is an entirely different matter.
Computer software is no different.

People expect too much too soon.

Good software takes time to develop. There is sometimes a tendancy to set unrealistic goals and when they aren't met the people in charge feel let down.

What large projects DON'T have problems?

In related news, humans still can't seem to bridges [sfgate.com] with any reliable schedule or budget. Despite the fact that bridges have been built probably since the dawn of man, and we've been building suspension bridges for at least 500 years [wikipedia.org].

Two things make software "hard"

First, the people who make the technology decisions often don't have the required technical know-how, and have a terrible tendency not to listen to people who do. OTOH, the people who have the technical know-how have absolutely no idea how to write a business case. Thus, there's usually a disconnect between the people who understand the business requirements, and the people who understand the technical requirements. Vendor loyalty has been known to be a sticky issue as a result.

Second, there's always a problem getting a bunch of talented, egotistical (ok, so not all software developers have ego problems...) quirky, eccentric and generally difficult people to work toward a common goal. The common analogy to being a successful director/manager of a software project is to that of "herding cats". My experience has been that business types don't react well to the often-emotional developer types, hearing the emotional outburst, but ignoring the content of it. Developers would do well to learn some more social skills, and director/manager types would do well to listen better.

mandelbr0t

Development fails because developers are ignored

Well said.

But while stereotypes persist that programmers have no people skills, you forget that many business people don't either.

Just ask yourself: how many effective, people-oriented bosses have I ever had? If your answer is "not many", you're not alone.

I've been software engineering for over a decade. These are a few observations I'd like to share with managers:

1. Hire good people. If you don't, you're hosed before you start. If you can't tell the good people from the bad, you're in the wrong business.
2. Software engineering often fails because software engineers are ignored. In a badly run company, nobody even asks developers which is the best approach. Imagine if civil engineering were like this...
3. "Software is too expensive to do cheaply". Sure, you can get 20 engineers with 2 years experience apiece. But it's more effective to get 4 engineers with 10 years experience.
4. Let the team (largely) manage themselves. Open source projects work just fine without management - or, at least they do no worse than managed software...
5. Don't do things to keep morale high; instead, cease doing things that keep morale low. We engineers just want to do good work in a nice environment. It goes back to hiring good people and then leaving them well alone.
6. Give the team a hand in decision making. The team will be more incentivised to hit targets they set than the ones you set. Afterall, the only incentive they have for hitting your impossible deadlines is furthering your career.
7. Automated regression tests! There are still managers who don't know what these are/how important they are. Along with all other good processes, automated regression tests reduce stress.

Theory is great

practical application isn't so. Requirements change, clients see something and want something different. You work a month on something and find out it isn't working anywhere close to the way it should. Tasks are more complex then people thought at first, the list goes on. It is a complex thing with infinite ways to do things. It isn't any easy process.

Where software developers sell themselves short

When you've contracted an engineer to design a bridge, and you want to make several large-scale changes to that bridge, the engineer will come back and say "If you want these changes done, this is how long it will take. If I don't have that much time, I can't make the changes".

In fact, I'd go a step further; software developers tend to say "This is how long it will take to make the change, and this is how long it will take for me to hack something together." Bridge engineers don't say things like that. They don't put that "hack something unsafe together" option out there on the table, and neither should we.

I think one of the biggest problems in our industry is accountability. The engineer would never put the unsafe option on the table, because the engineer knows he'll loose his license and go to prison if the bridge collapses. With software, on the other hand, we just expect our customers to deal with the fact that it fails, and we behave accordingly - and unprofessionally.