Software Bug Halts F-22 Flight

mgh02114 writes "The new US stealth fighter, the F-22 Raptor, was deployed for the first time to Asia earlier this month. On Feb. 11, twelve Raptors flying from Hawaii to Japan were forced to turn back when a software glitch crashed all of the F-22s' on-board computers as they crossed the international date line. The delay in arrival in Japan was previously reported, with rumors of problems with the software. CNN television, however, this morning reported that every fighter completely lost all navigation and communications when they crossed the international date line. They reportedly had to turn around and follow their tankers by visual contact back to Hawaii. According to the CNN story, if they had not been with their tankers, or the weather had been bad, this would have been serious. CNN has not put up anything on their website yet." The Peoples Daily of China reported on Feb. 17 that two Raptors had landed on Okinawa.

Re:crash narrowly averted

[ColaMan]

I've heard of a software glitch causing a crash before, but this is ridiculous.

Not really - read the Risks-Forum Digest [ncl.ac.uk], especially the earlier years, and you'll find that software quite often causes physical harm.

Re:Real redundancy

[Richard_at_work]

The F-22 can carry the standard USAF air delivered nuclear weapon as maintained within the US military arsenal today, either one internal or two external. The radiation from the weapons has no effect on the stealth, either before or after detonation (the stealth capability involved is an advanced form of that used on the B-2 and B-1B bombers, both of which were at their inception designed to be purely nuclear armed bombers).

Gotta know your limitations...

[TigerNut]

When I worked at a high end civilian GPS equipment manufacturer, we had a test department where, among other things, a complete list of "special" dates and locations were kept on file. Any new position solution software release was regression tested against all previously known and guessed potential date/time rollovers, as well as making sure that motion across geographic coordinate boundaries didn't cause erratic behavior. Obviously whoever supplied the inertial navigation solution for the F22 hasn't quite gotten there yet... Testing in the lab is cheap. Burning a couple of tons of Jet-A and putting a bunch of people at risk is not.

Some exaggeration in the story, I suspect

[CardinalPilot]

The F-22 has a fly-by-wire control system. If there really were a crash of ALL on-board computer systems, communication and navigation would not have been the most immediate concerns!

Re:UTC

[Chmcginn]

They probably already do... When I was spending time in uniform, all our (non-workstation) computers did all their work in GMT, anyway. And considering it was the navigation systems that crashed, I think the "international date line" thing is spurious - the problem was more likely going from W to E, not today to yesterday.

Not the first time this bug has shown up.

[sbaker]

You'd think they'd have learned from this one:

        http://www.f20a.com/f20ins.htm [f20a.com]

Fixed

[daveschroeder]

Well, whatever the issue - which is probably something similar to what you suspect - it's now fixed. Here's the transcript from CNN this morning. Since the F-22 is fly-by-wire, it's also worth pointing out that all systems didn't crash, else these F-22s would be sitting in the Pacific. I've no doubt it affected navigation, communications, and similar subsystems, and was probably related to physical location in terms of time, position over the Earth, or both, given the nature of the issue.

>> 25 Years from development to deployment, the F-22 Raptor is the most advanced fighting machine in the air. It was no match for a computer glitch that left six of them high above the pacific ocean, deaf, dumb, and blind as they headed to their first deployment. So what happened? We turn to a man that's at home in the cockpit. Retired Air Force General Don Shepperd. Let me set the scene, Don. These F-22s, headed from the Air Force base in Hawaii to an Air Force base in Japan. They were approaching the international date line, pick it up from there.

>> You got it right. You want everything to go right with the frontline fighter. $125, 135 Million a copy. The F-22 raptor is our frontline fighter, air defense, air superiority, and it can drop bombs. It is stealthy and fast. You want it to go right. On the international deployment to the pacific, it didn't. At the international date line, whoops. All systems dumped. When i say all systems I mean all systems, navigation, part of the communications, fuel systems, and they were -- they could have been in real trouble. They were with their tankers. The tankers -- tried to reset their systems. Couldn't get them reset. Tankers brought them back to Hawaii. This could have been real serious. Certainly could have been real serious if the weather had been bad. Turned out okay. Fixed in 48 hours. It was a computer glitch in the millions of lines of code; somebody made an error in a couple lines of the code and everything goes.

>> This is almost like the feared Y2K problem that happened to these aircraft. We should point out, the computer problems in 2000. The computers absolutely went absolutely haywire and became useless?

>> Absolutely. When you think of airplanes from the old days, with cables and that type of thing and connects between the sticks and the yokes and the controls -- not that way anymore. Everything is by computer. When your computers go the airplanes go. You have multiple systems. When they all dump at the same time, you can be in real trouble. Luckily this turned out okay.

>> What would have happened if these brand-new $120 million F-22s had been going into battle?

>> You would have been in real trouble in the middle of combat. The good thing is we found this out. Any time -- before, you know, before we get into combat with an airplane like this. Any time you introduce a new airplane, you are going to find glitches, and you are going to find things that go wrong. It happens in our civilian airliners. You don't hear much about it. These things absolutely happen. And luckily had time we found out about it before combat. We got it fixed with tiger teams in about 48 hours and the airplanes were flying again, and completed the deployment. This could have been real serious in combat.

>> You had these advanced air -- not just superiority but air supremacy fighters in there, up there in the air, above the Pacific Ocean, not much more sophisticated than a Cessna 152 with a jet engine?

>> You got it. They are on a 15-hour flight from Hawaii to Okinawa. When all their systems dumped, they needed help. Had they gotten separated from their tankers or weather gotten bad they had no reference and no communications or navigation. They would have turned around and could have found the Hawaiian Islands. If the weather had been bad on approach there could have been real trouble. You get refueling from your tankers and you don't run -- you don't get yourself where you run out of fuel. You

Correct Story?

[shields020]

Last time I checked, the F-22 is not a new plane...are we referring to the new Joint-Strike Fighter, or are we actually speaking about the F-22 that's been publicly known about since the mid-nineties?

Re:Real redundancy

[Anonymous Coward]

I worked as a software engineer on the Space Station for 5 years. I did not specifically work on Shuttle, but you get to know the systems and people, and I never heard anything about shuttle having two sets of redundantly implemented software. More, being very familiar with the test and verification procedures that NASA uses it is hard for me to imagine that a system with 2 operational software packages would ever get through the all of the DCMA and QA approvals needed for flight verification (imagine how many permutations of failure->recovery situations would be possible). Bottom line, I don't think it is true that shuttle has redundant software packages.

Website

[linux pickle]

For those who find it fishy that there's no article link for this story, here is one: http://www.flightglobal.com/articles/2007/02/14/21 2102/pictures-navigational-software-glitch-forces- lockheed-martin-f-22-raptors-back-to-hawaii.html [flightglobal.com]

Re:Overflow

[Sinical]

Please stop. No one is using femtoseconds for uptime.

Something more reasonable is that the nav system (presumably GPS) didn't like having the date change after aquisition. You'd think that'd be a fairly normal thing to have happen, but after the horrible crap I've seen happen with Rockwell Collins' receivers (they SUCK), it wouldn't be too surprising.

To expand on the Rockwell Collins (they SUCK) theme, we eventually got them to admit to us how to retrieve their diagnostic info, including a register that counting up floating point exceptions (yay, divide by zero!). It had well and truly saturated. On a test flight of an, in part, GPS-guided missile, it once croaked right at launch. Since we never understood that we were moving, we never turned on the autopilot. However, rocket motors don't have much in the way of an off switch, so away we went without autopilot. Boink!

So there are plenty of ways for nav systems to suck (especially if they are made by Rockwell Collins (they SUCK)) without needing something completely stupid like measuring data in femtoseconds.

Hold up, I got a few more of these:

Rockwell Collins (they SUCK)
Rockwell Collins (they SUCK)
Rockwell Collins (they SUCK)

That is all.

Re:Real redundancy

[GeffDE]

Nuclear bombs generally use plutonium-239, which emits either alpha or beta radiation. The number of particles emitted by such material is several orders of magnitude less than the number of photons given off by an incandescent lightbulb. At 10 km, the number of alpha or beta particles that would hit a detector (unless the detector were very large) would hardly be above background. Additionally, because Pu-239 is an alpha emitter, the metal encasing it is enough to block (most) of the radiation.

Re:F16 Software had similar problems

[patio11]

Its important to note that that bug was present *in simulation only*. My unit tests catch all sorts of nasty edge cases, including some which cause the system to drop huge chunks of the database -- that is what testing is for!

Re:Millions of lines of code?

[encoderer]

Are you suggesting that flight systems are not as complicated as Windows 3.1?

Besides, it's probably no different than every other real-world software application. We all stand on the shoulders of giants. The technology stack is probably pretty mature and stable, with all the conventions of modern programming: Layers of APIs and abstractions, shared libraries, etc.

It's probably, I'd say, that the systems have many millions of lines of code. It's probably unlikely that the specific subsystems affected had that many lines.

Re:2 EuroFighters 1 F-22

[im_thatoneguy]

Actually they're significantly better than the Eurofighters.

Let's look at a few simple theoretical examples.

You're flying into heavily armed enemy space at night:

      - You fly in 100 Eurofighters. Your enemy has 1000 missiles. You lose 100 Eurofighters
          and hit no targets.
      - You fly in 1 F-22. Your enemy has 1000 missiles, they never detect you. You hit your
          target and leave enemy airspace.

        In this case the F-22 was better than 100 Eurofighters.

        -You're flying alone into enemy territory. You spot a flight of 3 Eurofighters flying in
        formation. You fall into a following position on their tail. You fire 3 missiles
        simultaneously and before the enemy pilots can react. They're dead.

In the Alaskan trials the F-22s ammased 144 kills to 0 losses. That's a pretty good investment. And while they weren't flying against Eurofighters, I'm not sure it would have helped. It doesn't come down to who can turn twice as fast. It's who can fight twice as smart. During this same combat exercise Raptors engaged enemy forces out numbered 4-1 and stil came out victorious.

In previous exercises a single pilot was able to engage 9 enemy fighters, and then ran out of targets, but still had some ammunition remaining. What's most impressive is the ability for the F-22 to multiply the effectiveness of the existing airforce. In the same engagement that F-22 enabled a supporting flight of older aircraft to achieve a kill/loss ratio of 83-1.

Re:Real redundancy

[Indigo]

The Shuttle does indeed have two sets of flight software, Primary Avionics Systems Software (PASS) [nasa.gov], and Backup Flight System (BFS) [nasa.gov]. During critical phases of flight, PASS is loaded on four of the GPCs and BFS is loaded on the fifth. BFS doesn't have all the capabilities of PASS - it is intended to take over in case of an emergency.

Re:Bullshit

[TimSSG]

Both of you can be correct. Each group of models of the F-16 had more digital components and less analog ones. Tim S (retired F16 Radar Repair Technician, F-16 C/D models)

BBC Report on EuroFighter & EuroFighter Beats

[reporter]

The BBC reported on the status of the EuroFighter in a report in 2006 August [bbc.co.uk]. "This [F-22] is very stealthy but costs twice the price of the Eurofighter, and reports suggest that RAF's Eurofighters have flown highly successful missions against the F-22 during recent exercises in the US."

A Scottish report [scotsman.com] describes a dogfight of 1 EuroFighter against 2 F-15s. The EuroFighter reduced both F-15s to smoking rubble.

Based on these reports, we can surmise that the EuroFighter substantially outclasses an F-15 but does not quite beat an F-22. However, the cost of one F-22 enables the purchaser to buy 2 EuroFighters. The 2 EuroFighters could demolish the the one F-22.

Re:Cost Efficiency: EuroFighter vs. F-22

[Anonymous Coward]

"(hell, no stealth fighter has existed before the F-22)".

Immediately, I think of the F-117A. Darn, it's classified as a 'strike aircraft', with armament of an 'internal weapons carriage'. And missles are part of the known munitions stored in the weapons bays.

Sadly, no mention of air-to-air missles being hung, and not a peep that anyone other than the BBC (as if they are authoritative in this area) and a Wikipedia article (semi-ditto) saying it could carry AIM-9's. The A/F-117X would hang AIM-9 and AIM-120 AMRAAMs, but that's not going to be built. The 117 is being removed from service in 2008, priamrily because of cost of spares - F-15 landing gear, F-16 flight controls, even environmental controls from the C-130... Some of this stuff is becoming hard to maintain since the original types ar near the end of their service life. Darn.

But back to topic, the F-117 could have been the first stealth fighter, biut technically it ain't.

And I was all worked up for it to be so... Grrr...

-rick

Re:Cost Efficiency: EuroFighter vs. F-22

[voidptr]

If you'd actually read the article you linked to.. The F- designation on the F-117 is a curious bit of aviation history and Air Force infighting, but the F-117 is a ground attack aircraft, not a fighter, and should really have an A- or B- designation, while the F-22 is an air to air combat plane with limited ground attack capabilities. The 117's internal payload capacity is huge compared to the F-22's ground attack loads (some of which have to be carried outside, destroying the stealth capability) and it's therefore unlikely the F-22 is going to completely replace the F-117 completely anytime soon.

Re:Bullshit

[dmm79]

It's not bullshit. Each of these systems is developed independently which means that computer that assists with controls is totally different than the one that controls NAV or weapons systems. It's NAV and all other systems could have been completely F****D and it wouldn't affect ECU or Flight Assist software at all. But what this does say is that the company that developes NAV software needs to fire all their test and software engineers because this kind of test is one of the most basic ones to do. Aircraft software is tested to DO-178B standards because it's Safety Critical and should have gone through full code coverage. Somebody forgot to do their job.

Re:2 EuroFighters 1 F-22

[PPGMD]

You fire 3 missiles simultaneously and before the enemy pilots can react. They're dead.

Not quite the Eurofighters likely have RWRs so if you are using radar guided missiles they will likely detect your search, and targeting radars. So even with the newer harder to detect radars installed on the F-22 there is still a chance that they detect you from your radar emissions.

The F-22 is a fantastic aircraft, and is the best aircraft flying, but it isn't a perfect aircraft, and it doesn't have the capabilities that some people exaggerate it having. The Alaskan trails were set up by the fighter mafia at the Pentagon trying to justify their decisions in trying to keep the F-22 orders as high as possible.

It's not the first time that they have done this, during the training maneuvers against against the Indian Air Force they sent outdated aircraft and crippled the ROE and engagement envelopes of the AIM-120s. While the IAF didn't have such restrictions, at least none that we know of.

Most modern fighters are intrinsically unstable

[igb]

You can't fly planes like that manually, becuase they are inherently unstable. Even non-stealth aircraft have this property, in order to make them more sensitive in roll. A civilian plane will self-centre from small roll inputs, and you have to overcome that effect to actually roll. The stealth aircraft are such weird shapes, for which aerodynamics come second to radar cross-section, that the designers don't even have the choice.

ian

The post-incident report

[DingerX]

His comments are based on a post-incident report that's been making the rounds on teh intardnet. I'll just paste it in here, if anybody's still reading. I don't vouch for its authority, other than A) I got it off the net, and B) it came with a note saying it was unclassified. Oh yeah, and it matches what the talking head says -- the navigation system brought down all their avionics. it also states what the QA process was that led to the problem:

Date: 12 Feb 07

To: CC

Info: CV, DS

Narrative:


1. A 1st Fighter Wing AEF 6-ship (Petro 91) departed Hickam AFB enroute to AEF location on 10 Feb. Approximately 4 hours into the mission and coincidental with crossing over the International Date Line, all six aircraft experienced a significant avionics failure including:

Both GINS 1 and 2 Fail

FLCS Degrade

Radar Fail

Fuel Degrade

Loss of all attitude references

Loss of Flight Path marker

Loss of all navigation aides (TACAN, ILS, Computed, etc.)

Loss of all heading indications

2. Aircraft communications were available via backup radio only. Only navigation available was via cockpit airspeed and altitude indications (both deemed accurate). All other aircraft systems, to include engines, electrical system and air refueling, were nominal.

3. Flight Lead, Lt Col Tolliver, initiated via the tanker a CONFERENCE HOTEL (CH) call with LM Aero. All CH team recommended workarounds (avionics restarts, date and time resets, etc.) did not resolve the problem.

4. Lt Col Tolliver assessed pressing to the AEF location but decided to turn back and return to Hickam. He also directed the second deployment cell, a 2-ship approximately one hour behind him, to return to Hickam. NOTE: This 2-ship never crossed the International Date Line.

5. Enroute back to Hickam, after crossing back over the International Date Line, avionics restarts were unsuccessfully attempted.

6. All aircraft successfully recovered at Hickam, shut down (cold iron), restarted engines and all avionics malfunctions cleared.

7. An F-22 Crisis Management Team (CMT) has convened. Two telecoms (1300 and 1700 EST) were conducted on 11 Feb. Participants included F-22 Program Office, LM, Boeing, NG and A8F personnel.

8. The F-22 Program is working 24/7 to resolve this issue. Both F-22 avionics integration labs (RAIL and AIL) have successfully duplicated the problem. The problem resides within the GINS software when the aircraft transitions between East/West Longitude. NOTE: Most RAIL and AIL testing simulate GINS inputs and past testing discovered no issues with over flying the Dateline or Poles. It took testing this weekend using actual GINS hardware and software to duplicate this problem.

9. A fix for this software problem has been developed at NG and currently is being evaluated in the RAIL. We should find out at our 1300 CMT telecom today if this fix works.

10. This fix will require an OFP update to be loaded on the aircraft. Currently no IMIS OFP loading support is on-site at Hickam. 1 FW IMIS was previously deployed to AEF location.

11. F-22 Program currently expects software fix, OFP loading hardware and LM support team in place at Hickam by mid-week. Aircraft possibly will be able to depart Hickam for their AEF location by the end of the week.

12. Updates to this issue will be provided as additional information becomes available.
Translation: The navigational system (Global Positioning Inertial Navigation Systems (GINS)) had never been physically tested crossing the date line, but only on simulated real-world inputs. When it crossed the date line for the first time, it crashed, as did the backup, bringing down with it all navigational systems and much of the aircraft's instrumentation, leaving them with backup systems reminiscent of a Cessna 172 (without the navigational stack).

Re:Ironically

[julesh]

A few days ago reading up on good C++ coding techniques I came across Stroustrup's (creator of C++) page citing the coding rules used when working on the Joint Strike Fighter. Reading through the various rules used, this one caught my attention:

        AV Rule 25 (MISRA Rule 127)
        The time handling functions of library shall not be used.

I got to thinking if we had any decent alternatives (at least in C++). And yes there are alternatives and all of them looked equally bad to me. Looks like the F22 guys might have had the same problem finding and using a robust fault tolerant time library.

Why would you need to use a library? The only format you're likely to need in such software is milliseconds offset from some suitable epoch. As long as your hardware can produce such a time value, you're fine.

I think not

[brunes69]

Modern fighter jets are aerodynamically unstable by design. A human can not fly them alone, the computer has to correct the flight path hundreds of times a second.

The flight control software thus most certainly *does* have to keep the plane "stable".

Re:Gotta know your limitations...

[TigerNut]

No way for me to know exactly what happened in that case. However, the inertial navigation system on an aircraft has to know its orientation with respect to the earth (in terms of roll/pitch/yaw) and also its position (lat/long/altitude). THere are a bunch of different ways that you can do that. One of them, for example, is the "earth centered, earth fixes" coordinate system. In this system you compute your position in a 3D orthogonal system and then do a mathematical transform to get your lat/long/altitude position. The corrections that the INS has to make for gravity are dependent on your position on the earth, and there are other corrections required due to Coriolis forces (you're curving over the surface of the earth, which gives extra centripetal acceleration, but not exactly in the same direction as gravity). When these corrections, and the data you're getting from your GPS receiver, are computed (or natively received) in different coordinate systems, then you end up making a bunch of spherical coordinate transforms, usually back and forth. These transforms are matrices of sine and cosine factors of your coordinates, and it could be that somehow they ran one of these matrices through a singularity at the date line, or else maybe it's the even/odd thing: cos of a negative angle less than 90 degrees is positive, just like cos of a positive angle less than 90. Therefore acos(cos(-60 degrees) = 60 degrees. If you didn't guard against that possibility then it could happen that the nav system got highly confused by a sudden large discrepancy between two of its subsystems' solutions, and threw in the towel.