Software Bug Halts F-22 Flight

mgh02114 writes "The new US stealth fighter, the F-22 Raptor, was deployed for the first time to Asia earlier this month. On Feb. 11, twelve Raptors flying from Hawaii to Japan were forced to turn back when a software glitch crashed all of the F-22s' on-board computers as they crossed the international date line. The delay in arrival in Japan was previously reported, with rumors of problems with the software. CNN television, however, this morning reported that every fighter completely lost all navigation and communications when they crossed the international date line. They reportedly had to turn around and follow their tankers by visual contact back to Hawaii. According to the CNN story, if they had not been with their tankers, or the weather had been bad, this would have been serious. CNN has not put up anything on their website yet." The Peoples Daily of China reported on Feb. 17 that two Raptors had landed on Okinawa.

Re:I've been wondering...

[Anonymous Coward]

Is it because nobody has the time or patience to put up with Windows/Linux except for friendless, sexless nerds like you?

You obviously aren't to far from the crowd that you've unfairly and wrongly stereotyped if you've got time to post to Slashdot in the first five minutes of a new post and felt the need to take a precious 2 minutes away all your sex-orgies and circle jerks with your friends to point this out.

Chris

Re:Overflow

[Anonymous Coward]

The problem probably isn't with the time change. Airplanes use GMT so the local time doesn't matter. The problem is probably related to the longitude going from W179.99 degrees to E180 degrees.

Re:Real redundancy

[Anonymous Coward]

the raptor is a fighter. the most is can kill is a coupla dozen. its a non critical platform. lose a raptor and you lose what...2 2000lb bombs and a coupla air to air missiles ?
now if it was a B2 carrying nukes it might be a cause for concern. the shuttle is hugely expensive compared to the raptor and they spend nearly $10mil every year in ensuring the software is perfect. fighters dont get much software development time.

UTC

[Excelcia]

The answer to all these problems is very simple. For any mission critical application, use UTC and only UTC. No time zones, no date line, no converting. If the software isn't even aware of the concept of date/time localization, then it's not going to run into problems.

Oh, and while they're at it, standardize on metric too. Maybe we can save our interstellar probes at the same time we are saving our warplanes.

Design? Lack of foresight?

[the_skywise]

Assuming it WAS a time issue upon crossing the International Dateline...

Design problem? Why should navigation software require "local time"? They knew they were crossing the international dateline, so they must be linked to GPS timing systems... why not just use GPS' universal time? (Sure, you want local time eventually for your displays but that's a "view" calculation, not one intrinsic to the navigation software)

Bug tracking problem? Did the testers not think of testing about a time zone change? Did they assume the above that everything would be on a universal time and therefore didn't see the need for crossing time zones?

Why wasn't this a stock reusable code module in Lockheed Martin's labs?!?

(And for a media look at this issue, check out the anime Geneshaft or the movie The Pentagon Wars)

Er what?

[chanrobi]

Are you telling me that the F-22 has no analog backup flight system? For gosh sakes even the F-16 has a similar system. A cursory google search that the F-22 is equipped with an "LN-100G Inertial Navigation System with Embedded GPS". It sounds incredible that the summary implies that the only way they would've made it home was via formation flying with a tanker? Can anyone with more detailed information on the F-22 clarify?

Re:Don't worry

[MSFanBoi2]

Actually not really. The Eurofighters have very limited air to ground functionality at least until Block 5 hits sometime in late 2007. It was expected in 2005 but per usual, was very behind. The F-22 can carry two 1000 lb JDAM internally (or 8 GBU-39s) for a total of 2000 lbs of internal weapons and up to 5000 lbs of external weapons on four (two per wing) removable hard points (two of which are plumbed for fuel).

We won't even go into the fact that the F-22 is faster with a full weapons load and much faster at both high and low altitudes when fitted with a typical combat load, has a much longer range (up to 2x with combat load), the F-22 also has a superior thrust to weight ratio, has a higher reliablity rate (97% to 86%),

Re:I doubt they lost communication...

[theEteam]

In most modern aircraft, control for all avionics equipment is done through a central mission computer. If that computer crashes(usually there are two but they have identical software), all avionics will be unavailable. This includes radar, navigation, most radios, etc. Usually there is a backup RCU(remote control unit) for one of the radios and of course you can still steer, but that is about it.

Moderation?

[Anonymous Coward]

Why do you guys give +5 to someone who doesn't know for sure how the date line works, and who merely looked up which SI prefix was small enough to cause a 64-bit overflow? Most likely the bug has to do with overflow in position, not time. Even assuming this has to do with time overflow, modern GPS electronics can only measure signals to within 10 nanosecond. Using femtoseconds (10,000,000x smaller) is complete BS to make his argument work.

Bullshit

[Anonymous Coward]

a software glitch crashed all of the F-22s' on-board computers

Bullshit. The F-22 cannot be piloted manually, if all the computers crashed, then so would the planes.

Given this inconsistency, I am disinclined to believe the rest of the story.

Re:Cost Efficiency: EuroFighter vs. F-22

[Rakishi]

The F-22 is a full stealth fighter, the EuroFighter is not in any way. If you cannot understand why you cannot base a stealth plane design in any major way on something that is not a stealth plane (hell, no stealth fighter has existed before the F-22) then why are you eve talking about this as you clearly have no idea about anything involved.

And yes the F-22 is likely worth 84% more than the Eurofighter in terms of performance due to stealth alone.

Incidentally since the F-22 is what the F-35 is based on that $70billion has technically led to the creation of two planes, the later of which is being sold quite widely.

I have worked on Commecial and DoD avionics

[EMB Numbers]

I have worked on Commercial and DoD avionics, and this type of thing is inexcusable.

Commercial avionics software of the sort described is governed by a standard called DO-178B level A or level B. The process is so rigorous that the slogan is "no-one has ever died from software failure in a commercial airliner, yet." DO-178B level A is expensive. It is virtually impossible that a software error of the nature described could get into a certified aircraft.

Having said that, the military is not obliged to follow commercial standards, but there is a trend toward using DO 178-B in military systems in part because the Europeans are starting to require commercial JAA/FAA certification for all aircraft that enter their air space. But even in the more lax military world, every line of code is typically formally reviewed and there are independent testers. The type of error described should have shown up in simulators before the first flight of the aircraft. Test flights should have stimulated the error long before a squadron ever attempted a transpacific flight.

Even worse still, avionics systems are supposed to be isolated from each other. Navigation radios typically share nothing but power with GPS or with engine instruments etc. Great effort prevents one system from disturbing the power of another too. Aircraft typically have two or more separate primary navigation systems plus inertial guidance and old fashion compass + baring/vector navigation. Military aircraft need to survive both normal equipment failures and battle damage. Military radios (including navigation) need to be isolated from other systems for security reasons too. Those NSA guarded encryption systems can not be contaminated by software that has lower security classification (like navigation)without somebody going to federal prison for a long time.

The bottom line is that something very very wrong, negligent, and illegal needed to happen for the described error mode to manifest. That makes me doubt the story.

Re:Cost Efficiency: EuroFighter vs. F-22

[LibertineR]

You might consider that the F-22 makes the Eurofighter look silly in 1v1 dogfights, is faster though both aircraft have supercruise, flies higher, and has that little thing called stealth that allows it to attack heavily defended targets, and close to attack range and shoot other aircraft before being seen. It has greater range, higher operating altitude, better situational awareness for the pilot, monster engines, and did I mention STEALTH?

The return on investment is HEAVILY in favor of the F-22. There is no aircraft anywhere even close. The Eurofighter is the second best fighter aircraft ever built, but it is miles from being in the same class as the F-22 Raptor.

Position problems more likely

[EmbeddedJanitor]

It is amazing how much navigation software does not handle the East/West rollover properly. Having international development/testing teams scattered over the whole globe sure helps.

If you're going to write software like this, then test it or simulate it at all the wierd places in the world: date line [East/West rollover], equator [north/south chnange], GMT+13 hours [NZ daylight saving time].

Re:Real redundancy

[SnowZero]

The F22 does not normally have external mounts, but there are hardpoints where they can be added. Of course that would defeat the stealth, but if you're about to drop two nukes, at that point you're probably beyond being sneaky.

Re:Overflow

[im_thatoneguy]

That could be the dumbest idea I've heard all day.

So... a program that's in danger of being cut back intentionally causes a significant failure! Why not just submit a proposal to cancel the program? These are not the headlines LM wants right now. When lots of money has been spent, people irrationally expect perfection. Flying to Japan participating in exercises and kicking ass would have gone much further to proving the program viability than creating false doubts of reliability!

Re:Fixed

[rbanffy]

It is the fact of life. In the modern computer age.

It sure doesn't need to be like that.

Our desktop computers crash because we can tolerate crashes. There is some redundancy - if my notebook crashes, I reboot it and, in a couple minutes, I am back to work. If it breaks, I grab another computer and continue.

A plane, on the other hand, should work at all times. When lives depend on some equipment, one should enforce much higher standards than we do on desktop or even mission-critical busines software. Nobody dies if your sales people have a 5 minute outage. Nobody dies if you can't create a patient record. People die when the computers a plane relies upon fail.

It's completely unacceptable - and quite alarming - to see a plane malfunction like that on its first deployment.

Things like that should have been exercised years ago. By now, the code should be rock-stable. Whant kind of quality assurance they did?

Re:Cost Efficiency: EuroFighter vs. F-22

[Anonymous Coward]

And yes the F-22 is likely worth 84% more than the Eurofighter in terms of performance due to stealth alone.

Only if stealth is a requirement. In a real dogfight, the Eurofighter likely wins because maneuverability was foremost in its design, whereas the F-22 has stealth as the foremost design priority. The thought is that engagements are likely to be fought a distance with missles, and the low observability tech will allow the American aircraft to engage long before the enemy can return fire. This does not jive entirely with engagements of the past, which often involve close range encounters to verify enemy, or orders to wait until fired upon to return fire.

Compare this to the ability to put twice as many aircraft in the sky, carrying more munitions (while the F-22 has some stealty weapons bays, maxed out with a full bomb load involves external mounts with has a huge impact on radar visibility). Point is, whether stealth is worth 84% more has more to do with your mission profile and expected enemy/target,

Re:I have worked on Commecial and DoD avionics

[nonsequitor]

I too have worked as a contractor doing avionics work. What really surprises me in all of this is that there was no hardware watchdog, or way to reset the box on the way back. I used to work on multi-function displays, ADIs, HSIs, TCAS, etc... The adage goes that no information is better than old information so after going blank, it should have come back up in less than a minute. The fact that the failure state entered by crossing the dateline was persistent after a reboot is criminal negligence, these are people's lives here. Pilots have breakers for everything, they would have cut power and restored it after exhausting all other options, the fact it still was not operational says a lot, none of it good.

Re:Don't worry

[Dunbal]

Not to mention that the only decent fighters Europe have ever created were all WWII era...

      I dunno, the Americans seemed to quite like the idea of the AV-8A Harrier [wikipedia.org], a British creation.

Re:Er what?

[Jerf]

Before you go ballistic, bear in mind that unless you've got data sources beyond those cited in the Slashdot blurb, the most technical details come from CNN, which is about one step from priding itself on its ignorance of military matters, and has a less-than-distinguished history on the technical details front as well. Put the two together and the odds are low that you've got anything like an accurate view, let alone a complete one.

You can trust the what and the when; I wouldn't trust their how or why any further than I could spit.

(This isn't anti-CNN; this is anti-almost-everything news media. Journalists aren't required to learn squat about science or technology for their degree and it tends to show up in every last article they write with even a passing connection to science or technology. Any even cursory overview of stories on any technical subject you know about will reveal this. Remember that "multi-gear rocket" atrocity from a day or two ago?)

Re:Fixed

[0100010001010011]

And even more people die when all systems work perfectly.

Re:UTC

[Dun Malg]

As others have noted, this is less likely to have been an issue of the time/date than it is of transitioning from 179.9degW longitude to 180E longitude. You're just assuming it's a date/time issue because we call it the International Date Line. Note how none of the sources have details.

Re:Don't worry

[ralphdaugherty]

As shown in Vietnam and the current Iraq situation, America has great difficulty in fighting a loosely-organized resistance.

      It's because we care about killing innocent civilians, and they are indistinguishable from innocent civilians.

      If we can't identify the enemy, it's a good sign we shouldn't be there.

  rd

Re:Don't worry

[wirelessbuzzers]

I sense someone who still hasn't got over the US getting defeated by a bunch of little Asian peasants in black pyjamas. "We could have won if we'd really wanted to, that does it, I'm taking my ball and going home..."

We didn't just lose the Vietnam war to the Vietnamese; we lost the war largely to public opinion. When we pulled out (right after the Tet offensive) we were winning the war: most VC operatives had compromised themselves to participate in the Tet offensive, and the North Vietnamese army had taken very heavy casualties.

Certainly, the Vietnamese (and the Iraqis after them) demonstrated that conventional military might fares poorly against a resistance, but the reality of the Vietnam was not so simple as American folklore suggests.

Re:Overflow

[Littleman_TAMU]

Disclaimer: I work for DoD, but not on the Raptor program. All I know about the Raptor is gleaned from public sources.

Yeah, because a glitch capable of dropping a missile could never arm it, aim it, and drop it.

No, it couldn't. The pilot would have to have already armed the weapons with a physical switch whose state would be read in a completely different routine and you'd better believe that routine has the hell tested out of it. He'd also have to hit a button to fire the weapon, again, different routine, lots of testing. Even if all that happened, a target would have to be acquired which, once again, requires pilot intervention, many routines, lots more testing. You're deluding yourself if you think the testing for weapons systems isn't more stringent than a date checking routine.

Sure, nav systems are separate from flight systems. Except that they're not really because as the in-flight-entertainment crash showed, even systems that aren't connected are.

You're comparing a commercial jetliner with a military jet. Also, it doesn't seem like you read the in-flight entertainment article since even a cursory glance lets you know that the entertainment system is completely separated from the flight and nav systems, it's even in the slashdot summary.

This is exactly what you get if you hire a bunch of hacks who live in their parents' basements - critical software that obviously doesn't have a test plan.

Obvious troll.

I'm a little surprised that they've never simulated this, or their simulator isn't. Either way it's laughable.

You never finished your second thought. I'm surprised they didn't test something like this either. I have a feeling CNN is wrong about this.

You're placing a lot of trust in design principles that, if they really were followed, would have presented the failure just witnessed. Surprisingly, this doesn't seem to shake your faith.

Except that design and software development practices will almost assuredly never catch every flaw. That said, I find it hard to believe that the nav computers weren't designed for a situation like this and reset themselves and bring critical systems online first following the reset since that's a standard failsafe design practice. Since no one has actually confirmed that this is the case, and given the media's horrible grasp of technology related matters, I'd bet that what really happened is that the nav computers reset once and it was decided that it was more important to find out what caused the reset than to continue on. Even more likely is that the computer didn't even reset, but the pilots saw a big enough display problem that it needed to be corrected. The military is pretty careful when it comes to new systems and would prefer to scratch a mission than risk a fancy new jet and its pilot.