Comair Done In by 16-Bit Counter 441
Gogo Dodo writes "According to the Cincinnati Post, the Comair system crash was caused by an overflowed 16-bit counter. Perhaps Comair should have paid for the software upgrade to MaestroCrew." You heard it here first...
Common problem (Score:4, Insightful)
Re:Comair? (Score:2, Insightful)
Re:Maybe it had "worked just fine" for them? (Score:5, Insightful)
Re:Maybe it had "worked just fine" for them? (Score:5, Insightful)
Re:Signed or unsigned (Score:2, Insightful)
Since 2^16 = 65536, I'm guessing signed.
Re:Comair? (Score:5, Insightful)
Now my question would be, since they're owned by Delta, why wouldn't Comair flights be handled within Delta's own reservation/flight tracking system?
p.s. I've traveled through CVG, on Delta, during the holidays. Not anymore... One weather-delayed flight and the whole system falls apart.
Re:Common problem (Score:4, Insightful)
Even if a system is outsourced it doesn't provide a company with 100% stable system. Frequently businesses define the type of system they want hardware/software and the amount they're willing to pay for it.
I work in a company that provides outsourced solutions. Monthly we provide info to businesses about their system. Also, we frequently make recommendations to augment the systems to improve performance. Businesses often choose to ignore our reports and recommendations.
Nothing's more frustrating then a meeting with a business having them tell us we mucked it up and in return we drop off the last 6 months of recommendations on upgrades to provide them additional hardware for their growing requirements and question why they choose to ignore it.
Now I'm not saying the provider didn't muck up. But, what I am saying is your statement that it's all the provider's fault may not be the case as the airlines probably choose to stay on that system as it 'met' their needs as they saw them.
Damn you 2s Complement! (Score:3, Insightful)
Error checking is the real culprit (Score:5, Insightful)
But the real problem is a lack of error checking. It sounds like the code had something like:
int num_crew_changes;
crew_change_list[++num_crew_changes] = blah;
And the counter wrapped and the system crashed.
The code should have said:
if (num_crew_changes == MAXINT)
{
ERROR(E1234, "too many crew changes");
}
The system is still degraded after 32767 crew changes. It might be so degraded as to be unusable. But at least the company would know the extent of the degradation and could pull out the appropriate "Plan B". It's much safer and better to work around a known problem of known scope than to work around a system crash when you don't know the exact problem.
Re:98 (Score:1, Insightful)
I'm not sure how the open source movement can pride itself with the quality of imbecile fanboys it attracts.
Re:Maybe it had "worked just fine" for them? (Score:3, Insightful)
That said, it's very true that many businesses get by "just fine" with existing, antiquated systems. Justifying system upgrades can be difficult from a conventional cost-benefit standpoint, when a large part of the benefit is based on preventing theoretical problems like this one.
Playing with fire (Score:2, Insightful)
Re:Bugtraq covered this as well.. (Score:5, Insightful)
Rubbish. Don't judge yesteryear's programs by today's standards. Back then 4MB RAM cost more than $200. That's how important memory conservation was. In 1989 using an int was a perfectly acceptable choice. If you were programming back then you'd know how loathe programmers were to use longs when they didn't have to. (Granted an unsigned int would've worked better here, but that 64K limit could've also been reached.)
The software spec probably says something to the effect of "Don't attempt to schedule more than 32,767 crew changes." If you're running software that's more than a decade old you need to know what the limits of your software are.
Re:It's times like this... (Score:4, Insightful)
A sells tickets for $0 loss.
B sells tickets for $75 loss.
B gains many customers. However, the more customers the more loss they incur. Recall EVERY SEAT costs them $75. Eventually B just runs out of money and ups the costs.
Now A and B sell at the same cost. Customers notice the price hike and get upset [because for some reason people think air travel is a god given right so they get insanely upset at everything].
Sure some won-over customers will stay with B but many will spread out [many are also not particularly loyal they just use whatever cheaptickets.com tells them to].
Tell me I'm wrong. Tell me that most airlines haven't been filing for protection. Come on, tell me
Tom
Re:Comair? (Score:2, Insightful)
Thank you. So if the article just said something like...
...then it would have been enough. That one amendment -- "airline shutdown in the midwest last weekend" -- would have been enough to make the article perfectly clear to anyone who wasn't up to date with the story so far.
Is this really so much to expect? I wasn't trolling, honest, I'm just increasingly frustrated with how addled the editorial review of articles is around here. They've had years to figure this stuff out, but it's just as bad now as it was when Slashdot got started. Are they ever going to start taking their jobs as editors seriously & professionally? (And this isn't meant to be a blanket complaint -- Pudge for one seems to be aware of these basics and tries to do right with his articles, but other editors consistently muck this up...)
Re:65535+2 post (Score:3, Insightful)
Re:How old was this software? (Score:2, Insightful)
Re:Maybe it had "worked just fine" for them? (Score:2, Insightful)
Re:65535+2 post (Score:3, Insightful)
The article says they had 1100 flights on one day. That's 34100 per month. So basicly they seem to have had on average about one crew schedule change per flight.
Now, that is a very bad failure rate, but given that any one crew change probably causes a mass of knock-on changes (Fred misses this flight, so you have to substitute John, and then someone has to take over what John should be doing for the rest of the day, and Fred won't be on the right place to do what he was supposed to do this afternoon, and John has reached his flight-hours limit so you have to get Harry in early and...), a good flu epedemic going around the fleet plus some bad weather delays and technical faults all coming in the same month would probably do it.
Re:unsigned (Score:4, Insightful)
fixes the problem.
You do realize that you've just fallen into the same trap, right? That doesn't fix the problem worth a damn. I mean, sure it doubles the amount of changes. And yes, 64,000 should be enough. But, hey, 32,000 should have been enough too, right?
Programs have internal limits. That's kosher. What's not appropriate is allowing the user base to exceed them or - for something like this - come close to exceeding them, without giving some kind of warning that notifies people of an impending problem and provides possible solutions (purge data, etc). Now you may point out that adding that kind of security increases the cost and complexity of software. Yup. That's why true enterprise software is expensive. Because that's what you're paying for.
Another alternative would have been for the software wrap and start purging existing records to make room for new ones. Either way, there should have been some defined strategy for the boundary condition, and there wasn't.
The other thing that the software vendor should have done when pushing their upgrade is point out that the previous version wouldn't allow flights to continue in that situation, but the new version expanded it to (some large number). Instead, they probably said, "We're 32 bit!" or something totally meaningless to the people evaluating the business case for the upgrade.
Re:Hmmmm.... Maybe I should edit my software (Score:4, Insightful)
It always fits perfectly in the context as well, as this example proves.
Re:It shows what unions are all about (Score:1, Insightful)
As for the US Airways workers being greedy, maybe you should read up on what US Airways has done to their pensions, benefits packages and wages over the last couple of years. Some US Airways workers are surviving below the poverty line, relying on welfare and food stamps to feed their families. So which would you prefer; that the workers continue to live on Government handouts (Paid for by you!) or that they try and get their employer to pay a fair wage which they can use to support themselves and their families?
Re:Common problem (Score:4, Insightful)
Only if it's done wrong.
The "value" of outsourcing in that particular example is that it forces the company to completely spec out the system requirements. No changes without documentation. With poorly controlled internal development, changes happens in the hallways or the cafeteria: "Hey, Rick, did you add the code to handle the offline situation?" "Oh, right, I'll just put in a return value for you." This leads to code that doesn't match its spec, making it harder to maintain. Outsourcing tends to enforce a good interface between spec and code, (which is what your claim seems to be.)
Internally developed programs don't necessarily receive the same amount of attention to detail because the programmers typically have an idea about the business domain of the problem, and can work more with less documentation. In some organizations, this leads to "fast and loose" -- great for response time, not so great for maintainability.
I think the "value" of outsourcing in a case like ComAir's is one of liability: ComAir will probably try to play "let's blame the vendor." Or, maybe they'll offer up for sacrifice only the one guy who signed the contract with the vendor, and not an entire division. But, when a failure reaches this magnitude, I don't think they'll get off that easy.
Re:Maybe it had "worked just fine" for them? (Score:3, Insightful)
Very rarely does anyone have the luxury of designing for something with a hundred year life expectancy and a budget to match.
Re:Maybe it had "worked just fine" for them? (Score:4, Insightful)
Business decides to buy a software package. After a while, upgrades come out, and the old version keeps getting pushed to the limits. IT adivses business of this, and says that an upgrade/replacement will resolve the problem, but business refuses to authorize said upgrade/replacement.
How do you propose IT "make it work" when their hands are tied? Even worse, IT will take the blame when it wasn't even their decision to make.
Re:ComAir Now Hiring IT People (Score:3, Insightful)
Read to the bottom. They're also hiring a "Staff Scheduler". Only a high school diploma required and 1 year of experience. Maybe they should raise their qualification requirements for this one given recent difficulties
Re:Bugtraq covered this as well.. (Score:4, Insightful)
The only place where shaving bits made sense for us was on data records: we had a hash file with 2.1 million records, each 29 bytes long and it they all had to fit on a single 80MB hard drive. We squeezed every single bit out of those records (including developing a 3-byte integer to handle amounts that we told them could never exceed $99,999.99 (among other things, larger amounts would not have printed correctly.) But they were read-only records to us: we never wrote more than a few thousand rows of data, and we had plenty of space for the day's processing. And when they did have the odd line item that exceeded $100,000.00, they figured out to break it up into multiple smaller items.
And we got bit more than once by overflows. It took like three separate f-ups to get this guy to acknowledge that he needed to stop being stingy with the bytes. Even then, he'd still try to sneak in some memory "savings", but at least he stopped arguing when we called him on them.
Re:From Another article... (Score:4, Insightful)
I would suspect the attitude of debating a limit without knowing the business context your design choice exists in is probably what created this error to begin with.
Re:It's times like this... (Score:3, Insightful)
My point is the **real** solution is to
get this, this is a doozy
====> **** NOT FLY THE FUCKING PLANE IN THE FIRST PLACE **** <====
If you're over supplying the true demand then you're always going to waste money. Don't make 90 million gizmos when there is only demand for 1 million gizmos.
The demand for air travel only surged when discount rates appeared. Discount rates only appeared to fill seats [re: artificial demand].
Essentially if you're flight is an emergency or longer than say 3-4 hours than it's a "need". People used to go by bus and train remember? You could take a one hour flight to the next city or spend a day on a bus. When flights were 400-500 dollars a seat people used to take the bus. Now that the same flight may cost 150 or so it's more attractive to fly.
But just because you CAN do something doesn't mean you SHOULD.
I mean they COULD sell the tickets for a buck each. That would fill up planes quick too. By your logic is that a good idea?
Tom
You've missed one dimension (Score:3, Insightful)
You want to fly to Los Angeles in a month and purchase the ticket then. The price you pay reflects the value of being able to make that choice then and assuring the airline of a seat being filled in a month. The value of the seat changes as time goes on such that 1 day before the plane leaves the seat is now worth a lot more to someone that has to get to Los Angeles the next day, no matter what the cost. Of course there is the other aspect as well - the seat has no value once the plane leaves.
Managing this changing value is what makes airline ticket prices incredibly complicated.
Re:From Another article... (Score:2, Insightful)
Well, figure 3 crewmembers per flight, something like 1500 flights per day, cancel most flights for one snow and ice day near the end of the month.
Maybe not so crazy.
Re:It shows what unions are all about (Score:5, Insightful)
The smaller carriers all have one thing in common - no unions. They do not pay their pilots as much, and their pilots do not get paid if they don't fly. The number one expense for an airline is fuel, but the number two expense are the pilots, stewardesses, mechanics, and baggage handlers. There was no way for older airlines to meet the new market conditions (fly more for less profit per flight) without paying people less. The problem is that no one wants to be paid less, so instead they get rid of the least powerful people (who also happen to be the least paid). This is also specified in the union contract... all laying off actions must be FILO.
Essentially, the major carriers are hamstrung by the unions, and they will not survive long term. Unions work by artificially limiting labor supply - but that doesn't work if there is not enough work.
The unions say how evil it is that they are getting pay cuts, but where exactly do they expect the money to come from? The government really should not prop up certain providers when others are eager to take there place. Competition works for the most part. Air travel is becoming a commodity market, like cars. Market transitions cause upheavals, and change the market leaders - especially if the current leaders cannot change their bussiness structure.
I have to say that I totally disagree about management being incompetant - the current management (at least the upper level ones I deal with) are extremely good. They may even get the airline to survive and change to the current market conditions. But what has really destroyed the airlines is the changing markets, and the unions preventing the old airlines to change with the times. The only thing management could have done would be to have rejected the union contracts earlier. But I doubt if that was possible.
Unions seem to believe that society owes them a living. The problem is that society (except in the form of government) is not a person, and so recognizes no debts. Fighting that is totally ineffective because there is no one to fight.
Coding practice (Score:3, Insightful)
Back in the 80s when I was C programmer (K&R, thank you, the one true C), C integer types were not standardized. "Integers" were defind to be the most natural size for a machine (typically a data word), "shorts" were defined to be no larger than ints, but possibly smaller (and thus possibly more space efficient). This reflected the philosophy of C-as-portable-assembler: if you were indexing an array of character representations of digits, for example, there was no reason not to use a short. It was conceivable that, since arrays were essentially immutable pointers and array indices were merely offsets against those pointers, you might want to refernce a negative offset from an array base in some kind of clever trick.
Various C implementations used short/integer sizes like 8/16 (for microprocessors like the 8080), 16/16, 16/32. These days, there are some mininal assumptions we can count on. Ansi-C specifies the following as minimal data sizes for char/short/int/long: 8/16/16/32. In practices IIRC, most modern compilers use 8/16/32/32, in other words a 32 bit int. GCC, I think uses 8/16/32/64.
The problem with this airline scenario I would expect is a kind primitive cousin of cut-and-paste coding. This is where the the programmer is pasting something like this from his mental scrapbook:
int i;
TRANSACTION trArray[];
It's very easy to do something like this. A really conscientious programmer asks himself whether the index value is indexing something that doesn't have a prescribed limit in the specifications (in this case I'm guessing it was probably indexing a file position). If there is no prescribed limit he uses an unsigned long. If there is a prescribed limit that would allow an integer index, he still uses an unsigned long unless he indexing something which logically can't grow larger, or until the profiler forces him.
Which brings me to what I find curious about this. Either: the programmer chose to index the value by an signed short (which would be almost inconceivably stupid as opposed to unforgiveably negligent), or he was using a C compiler with a 16 bit integer, which while possible under ANSI IIRC, seems terribly archaic.
Java, of course, uses 32 bit ints. But you aren't completely safe from this sort of thing. For example FileInputStream has two methods of interest here:
this is very safe, since it uses a long, which in java is 64 bits; even unsigned, there is little chance of overflow.
However consider this:
What happens when a programmer decides to skip around in a LARGE file using this API? If he decides to skip forward by more than 2,147,483,647 bytes the signed int will silently be converted to a negative offset, at least as of java 1.4. Granted the possibility is slim in most applications.
Re:Maybe it had "worked just fine" for them? (Score:3, Insightful)
THIS INCLUDES COMPUTING.
Part of serving the business is being built to capacity. This is no different than a correctly tooled factory or having enough warm bodies.
Your mentality simply falls from the illusion that IT isn't an integral part of the business. You can always choose to be "penny wise". However, that always comes with inherent risk.
They question you need to ask your CIO is: Do you feel lucky?