Slashdot Log In
Software Bug Halts F-22 Flight
Posted by
kdawson
on Sun Feb 25, 2007 06:35 PM
from the dare-you-to-cross-this-line dept.
from the dare-you-to-cross-this-line dept.
mgh02114 writes "The new US stealth fighter, the F-22 Raptor, was deployed for the first time to Asia earlier this month. On Feb. 11, twelve Raptors flying from Hawaii to Japan were forced to turn back when a software glitch crashed all of the F-22s' on-board computers as they crossed the international date line. The delay in arrival in Japan was previously reported, with rumors of problems with the software. CNN television, however, this morning reported that every fighter completely lost all navigation and communications when they crossed the international date line. They reportedly had to turn around and follow their tankers by visual contact back to Hawaii. According to the CNN story, if they had not been with their tankers, or the weather had been bad, this would have been serious. CNN has not put up anything on their website yet." The Peoples Daily of China reported on Feb. 17 that two Raptors had landed on Okinawa.
Related Stories
[+]
Technology: F-22 Raptor Cancelled 829 comments
BayaWeaver writes "Slate reports that the F-22 Raptor has been cancelled by the Senate. At an estimated price tag of $339 million per aircraft, even the powerful military-industrial-congressional complex couldn't keep this Cold War program alive in these hard times. They look very cool though and have appeared in movies like Hulk and Transformers. But not to worry too much about the future of the military-industrial-congressional complex: the F-35 Lightning II begins production next year!
As a side note, in 2007 a squadron of Raptors became deaf, dumb and blind when they flew over the International Date Line."
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Real redundancy (Score:5, Interesting)
Re:Real redundancy (Score:5, Interesting)
NASA do not fly the space shuttle during 31 Dec -> 1 Jan [newscientist.com] as
they are not confident of what would happen. Better just
to avoid the problem.
That was one of the pressures to getting the Dec 2k6 flight off the ground.
Parent
Re:Real redundancy (Score:5, Funny)
But they fly over the international date-line every 90 minutes or so with no problems
Parent
they are very confident (Score:5, Funny)
NASA is extremely careful with its software.
They don't fly from Dec 31 to Jan 1 because they know exactly what would happen.
Parent
Re:Real redundancy (Score:5, Informative)
Parent
Re:Real redundancy (Score:4, Insightful)
Parent
Re:Real redundancy (Score:5, Funny)
Yes. I hate those guys.
Parent
Re:Real redundancy (Score:5, Informative)
Parent
crash narrowly averted (Score:5, Funny)
I've heard of a software glitch causing a crash before, but this is ridiculous.
Don't worry (Score:5, Funny)
Re:Don't worry (Score:5, Interesting)
The 27th Fighter Squadron (8 F-22s) at Langley AFB, Virginia fought against 33 F-15Cs and didn't suffer a single loss. The F-15's again didn't even detect the F-22's until they were all locked and targeted.
Then some months later during Exercise Northern Edge F-22's reached a 144-to-zero kill-to-loss ratio against F-15s, F-16s and F/A-18s. Only 12 of the F-22's accounted for nearly 50% of all kills for the Exercise.
Parent
Winning Tactic (Score:5, Funny)
Parent
Source code... (Score:5, Funny)
Reminds me of the Bismarck (Score:5, Interesting)
Gotta know your limitations... (Score:5, Informative)
Some exaggeration in the story, I suspect (Score:5, Informative)
I think not (Score:5, Informative)
The flight control software thus most certainly *does* have to keep the plane "stable".
Parent
Design? Lack of foresight? (Score:5, Insightful)
Design problem? Why should navigation software require "local time"? They knew they were crossing the international dateline, so they must be linked to GPS timing systems... why not just use GPS' universal time? (Sure, you want local time eventually for your displays but that's a "view" calculation, not one intrinsic to the navigation software)
Bug tracking problem? Did the testers not think of testing about a time zone change? Did they assume the above that everything would be on a universal time and therefore didn't see the need for crossing time zones?
Why wasn't this a stock reusable code module in Lockheed Martin's labs?!?
(And for a media look at this issue, check out the anime Geneshaft or the movie The Pentagon Wars)
Were they running Windows? (Score:5, Funny)
Ironically (Score:5, Funny)
I got to thinking if we had any decent alternatives (at least in C++). And yes there are alternatives and all of them looked equally bad to me. Looks like the F22 guys might have had the same problem finding and using a robust fault tolerant time library.
Er what? (Score:4, Insightful)
So... (Score:5, Funny)
Actual dialog message... (Score:5, Funny)
F16 Software had similar problems (Score:5, Interesting)
http://catless.ncl.ac.uk/Risks/3.44.html [ncl.ac.uk]
Microsoft? (Score:5, Funny)
READ: Get Ready For More (Score:5, Interesting)
I told them... (Score:5, Funny)
not the only problem I read... (Score:5, Interesting)
http://www.atimes.com/atimes/Middle_East/IB10Ak05
"Keys notes, however, that the electronic spectrum around Baghdad is polluted by the myriad jamming devices that coalition forces primarily employed to thwart remote detonations of the improvised explosive devices that have inflicted 70% of all US fatalities in that war."
"The potential problem was discovered when the first F-22s were operating near US Navy ships off the Atlantic coast. Navy radars overwhelmed the F-22's automated sensors. Even now, larger, multi-station, purpose-built electronic-intelligence-gathering airplanes encounter difficulties around the Iraqi capital because of the extreme density of jamming devices."
Carry backup. (Score:4, Funny)
Re:Overflow (Score:5, Insightful)
Parent
Not the first time this bug has shown up. (Score:5, Informative)
http://www.f20a.com/f20ins.htm [f20a.com]
Parent
F-16 had a similar bug (Score:5, Interesting)
Parent
Fixed (Score:5, Informative)
>> 25 Years from development to deployment, the F-22 Raptor is the most advanced fighting machine in the air. It was no match for a computer glitch that left six of them high above the pacific ocean, deaf, dumb, and blind as they headed to their first deployment. So what happened? We turn to a man that's at home in the cockpit. Retired Air Force General Don Shepperd. Let me set the scene, Don. These F-22s, headed from the Air Force base in Hawaii to an Air Force base in Japan. They were approaching the international date line, pick it up from there.
>> You got it right. You want everything to go right with the frontline fighter. $125, 135 Million a copy. The F-22 raptor is our frontline fighter, air defense, air superiority, and it can drop bombs. It is stealthy and fast. You want it to go right. On the international deployment to the pacific, it didn't. At the international date line, whoops. All systems dumped. When i say all systems I mean all systems, navigation, part of the communications, fuel systems, and they were -- they could have been in real trouble. They were with their tankers. The tankers -- tried to reset their systems. Couldn't get them reset. Tankers brought them back to Hawaii. This could have been real serious. Certainly could have been real serious if the weather had been bad. Turned out okay. Fixed in 48 hours. It was a computer glitch in the millions of lines of code; somebody made an error in a couple lines of the code and everything goes.
>> This is almost like the feared Y2K problem that happened to these aircraft. We should point out, the computer problems in 2000. The computers absolutely went absolutely haywire and became useless?
>> Absolutely. When you think of airplanes from the old days, with cables and that type of thing and connects between the sticks and the yokes and the controls -- not that way anymore. Everything is by computer. When your computers go the airplanes go. You have multiple systems. When they all dump at the same time, you can be in real trouble. Luckily this turned out okay.
>> What would have happened if these brand-new $120 million F-22s had been going into battle?
>> You would have been in real trouble in the middle of combat. The good thing is we found this out. Any time -- before, you know, before we get into combat with an airplane like this. Any time you introduce a new airplane, you are going to find glitches, and you are going to find things that go wrong. It happens in our civilian airliners. You don't hear much about it. These things absolutely happen. And luckily had time we found out about it before combat. We got it fixed with tiger teams in about 48 hours and the airplanes were flying again, and completed the deployment. This could have been real serious in combat.
>> You had these advanced air -- not just superiority but air supremacy fighters in there, up there in the air, above the Pacific Ocean, not much more sophisticated than a Cessna 152 with a jet engine?
>> You got it. They are on a 15-hour flight from Hawaii to Okinawa. When all their systems dumped, they needed help. Had they gotten separated from their tankers or weather gotten bad they had no reference and no communications or navigation. They would have turned around and could have found the Hawaiian Islands. If the weather had been bad on approach there could have been real trouble. You get refueling from your tankers and you don't run -- you don't get yourself where you run out of fuel. You
Parent
Re:Fixed (Score:5, Insightful)
It sure doesn't need to be like that.
Our desktop computers crash because we can tolerate crashes. There is some redundancy - if my notebook crashes, I reboot it and, in a couple minutes, I am back to work. If it breaks, I grab another computer and continue.
A plane, on the other hand, should work at all times. When lives depend on some equipment, one should enforce much higher standards than we do on desktop or even mission-critical busines software. Nobody dies if your sales people have a 5 minute outage. Nobody dies if you can't create a patient record. People die when the computers a plane relies upon fail.
It's completely unacceptable - and quite alarming - to see a plane malfunction like that on its first deployment.
Things like that should have been exercised years ago. By now, the code should be rock-stable. Whant kind of quality assurance they did?
Parent
Re:Fixed (Score:5, Insightful)
Parent
The post-incident report (Score:5, Informative)
Date: 12 Feb 07
To: CC
Info: CV, DS
Narrative:
1. A 1st Fighter Wing AEF 6-ship (Petro 91) departed Hickam AFB enroute to AEF location on 10 Feb. Approximately 4 hours into the mission and coincidental with crossing over the International Date Line, all six aircraft experienced a significant avionics failure including:
Both GINS 1 and 2 Fail
FLCS Degrade
Radar Fail
Fuel Degrade
Loss of all attitude references
Loss of Flight Path marker
Loss of all navigation aides (TACAN, ILS, Computed, etc.)
Loss of all heading indications
2. Aircraft communications were available via backup radio only. Only navigation available was via cockpit airspeed and altitude indications (both deemed accurate). All other aircraft systems, to include engines, electrical system and air refueling, were nominal.
3. Flight Lead, Lt Col Tolliver, initiated via the tanker a CONFERENCE HOTEL (CH) call with LM Aero. All CH team recommended workarounds (avionics restarts, date and time resets, etc.) did not resolve the problem.
4. Lt Col Tolliver assessed pressing to the AEF location but decided to turn back and return to Hickam. He also directed the second deployment cell, a 2-ship approximately one hour behind him, to return to Hickam. NOTE: This 2-ship never crossed the International Date Line.
5. Enroute back to Hickam, after crossing back over the International Date Line, avionics restarts were unsuccessfully attempted.
6. All aircraft successfully recovered at Hickam, shut down (cold iron), restarted engines and all avionics malfunctions cleared.
7. An F-22 Crisis Management Team (CMT) has convened. Two telecoms (1300 and 1700 EST) were conducted on 11 Feb. Participants included F-22 Program Office, LM, Boeing, NG and A8F personnel.
8. The F-22 Program is working 24/7 to resolve this issue. Both F-22 avionics integration labs (RAIL and AIL) have successfully duplicated the problem. The problem resides within the GINS software when the aircraft transitions between East/West Longitude. NOTE: Most RAIL and AIL testing simulate GINS inputs and past testing discovered no issues with over flying the Dateline or Poles. It took testing this weekend using actual GINS hardware and software to duplicate this problem.
9. A fix for this software problem has been developed at NG and currently is being evaluated in the RAIL. We should find out at our 1300 CMT telecom today if this fix works.
10. This fix will require an OFP update to be loaded on the aircraft. Currently no IMIS OFP loading support is on-site at Hickam. 1 FW IMIS was previously deployed to AEF location.
11. F-22 Program currently expects software fix, OFP loading hardware and LM support team in place at Hickam by mid-week. Aircraft possibly will be able to depart Hickam for their AEF location by the end of the week.
12. Updates to this issue will be provided as additional information becomes available.
Translation: The navigational system (Global Positioning Inertial Navigation Systems (GINS)) had never been physically tested crossing the date line, but only on simulated real-world inputs. When it crossed the date line for the first time, it crashed, as did the backup, bringing down with it all navigational systems and much of the aircraft's instrumentation, leaving them with backup systems reminiscent of a Cessna 172 (without the navigational stack).
Parent
Moderation? (Score:5, Insightful)
Parent
Re:Overflow (Score:5, Informative)
Something more reasonable is that the nav system (presumably GPS) didn't like having the date change after aquisition. You'd think that'd be a fairly normal thing to have happen, but after the horrible crap I've seen happen with Rockwell Collins' receivers (they SUCK), it wouldn't be too surprising.
To expand on the Rockwell Collins (they SUCK) theme, we eventually got them to admit to us how to retrieve their diagnostic info, including a register that counting up floating point exceptions (yay, divide by zero!). It had well and truly saturated. On a test flight of an, in part, GPS-guided missile, it once croaked right at launch. Since we never understood that we were moving, we never turned on the autopilot. However, rocket motors don't have much in the way of an off switch, so away we went without autopilot. Boink!
So there are plenty of ways for nav systems to suck (especially if they are made by Rockwell Collins (they SUCK)) without needing something completely stupid like measuring data in femtoseconds.
Hold up, I got a few more of these:
Rockwell Collins (they SUCK)
Rockwell Collins (they SUCK)
Rockwell Collins (they SUCK)
That is all.
Parent
Re:Overflow (Score:5, Funny)
Parent
I have worked on Commecial and DoD avionics (Score:5, Insightful)
Commercial avionics software of the sort described is governed by a standard called DO-178B level A or level B. The process is so rigorous that the slogan is "no-one has ever died from software failure in a commercial airliner, yet." DO-178B level A is expensive. It is virtually impossible that a software error of the nature described could get into a certified aircraft.
Having said that, the military is not obliged to follow commercial standards, but there is a trend toward using DO 178-B in military systems in part because the Europeans are starting to require commercial JAA/FAA certification for all aircraft that enter their air space. But even in the more lax military world, every line of code is typically formally reviewed and there are independent testers. The type of error described should have shown up in simulators before the first flight of the aircraft. Test flights should have stimulated the error long before a squadron ever attempted a transpacific flight.
Even worse still, avionics systems are supposed to be isolated from each other. Navigation radios typically share nothing but power with GPS or with engine instruments etc. Great effort prevents one system from disturbing the power of another too. Aircraft typically have two or more separate primary navigation systems plus inertial guidance and old fashion compass + baring/vector navigation. Military aircraft need to survive both normal equipment failures and battle damage. Military radios (including navigation) need to be isolated from other systems for security reasons too. Those NSA guarded encryption systems can not be contaminated by software that has lower security classification (like navigation)without somebody going to federal prison for a long time.
The bottom line is that something very very wrong, negligent, and illegal needed to happen for the described error mode to manifest. That makes me doubt the story.
Parent
Re:I have worked on Commecial and DoD avionics (Score:5, Insightful)
Parent
Re:Cost Efficiency: EuroFighter vs. F-22 (Score:5, Insightful)
And yes the F-22 is likely worth 84% more than the Eurofighter in terms of performance due to stealth alone.
Incidentally since the F-22 is what the F-35 is based on that $70billion has technically led to the creation of two planes, the later of which is being sold quite widely.
Parent
Re:Cost Efficiency: EuroFighter vs. F-22 (Score:5, Insightful)
Only if stealth is a requirement. In a real dogfight, the Eurofighter likely wins because maneuverability was foremost in its design, whereas the F-22 has stealth as the foremost design priority. The thought is that engagements are likely to be fought a distance with missles, and the low observability tech will allow the American aircraft to engage long before the enemy can return fire. This does not jive entirely with engagements of the past, which often involve close range encounters to verify enemy, or orders to wait until fired upon to return fire.
Compare this to the ability to put twice as many aircraft in the sky, carrying more munitions (while the F-22 has some stealty weapons bays, maxed out with a full bomb load involves external mounts with has a huge impact on radar visibility). Point is, whether stealth is worth 84% more has more to do with your mission profile and expected enemy/target,
Parent
Re:Cost Efficiency: EuroFighter vs. F-22 (Score:4, Insightful)
The return on investment is HEAVILY in favor of the F-22. There is no aircraft anywhere even close. The Eurofighter is the second best fighter aircraft ever built, but it is miles from being in the same class as the F-22 Raptor.
Parent
Re:2 EuroFighters 1 F-22 (Score:4, Informative)
Let's look at a few simple theoretical examples.
You're flying into heavily armed enemy space at night:
- You fly in 100 Eurofighters. Your enemy has 1000 missiles. You lose 100 Eurofighters
and hit no targets.
- You fly in 1 F-22. Your enemy has 1000 missiles, they never detect you. You hit your
target and leave enemy airspace.
In this case the F-22 was better than 100 Eurofighters.
-You're flying alone into enemy territory. You spot a flight of 3 Eurofighters flying in
formation. You fall into a following position on their tail. You fire 3 missiles
simultaneously and before the enemy pilots can react. They're dead.
In the Alaskan trials the F-22s ammased 144 kills to 0 losses. That's a pretty good investment. And while they weren't flying against Eurofighters, I'm not sure it would have helped. It doesn't come down to who can turn twice as fast. It's who can fight twice as smart. During this same combat exercise Raptors engaged enemy forces out numbered 4-1 and stil came out victorious.
In previous exercises a single pilot was able to engage 9 enemy fighters, and then ran out of targets, but still had some ammunition remaining. What's most impressive is the ability for the F-22 to multiply the effectiveness of the existing airforce. In the same engagement that F-22 enabled a supporting flight of older aircraft to achieve a kill/loss ratio of 83-1.
Parent
Re:2 EuroFighters 1 F-22 (Score:5, Informative)
Not quite the Eurofighters likely have RWRs so if you are using radar guided missiles they will likely detect your search, and targeting radars. So even with the newer harder to detect radars installed on the F-22 there is still a chance that they detect you from your radar emissions.
The F-22 is a fantastic aircraft, and is the best aircraft flying, but it isn't a perfect aircraft, and it doesn't have the capabilities that some people exaggerate it having. The Alaskan trails were set up by the fighter mafia at the Pentagon trying to justify their decisions in trying to keep the F-22 orders as high as possible.
It's not the first time that they have done this, during the training maneuvers against against the Indian Air Force they sent outdated aircraft and crippled the ROE and engagement envelopes of the AIM-120s. While the IAF didn't have such restrictions, at least none that we know of.
Parent
Re:Overflow (Score:5, Interesting)
As for missiles? First, they fly unarmed on ferry missions because ammo is dead weight that reduces range; and second, even if they were armed, what do you really think would happen if an AMRAAM missile was free launched without being turned on, much less having had targeting info downloaded? Drop like a stone, it would, right into the pacific. Bloop. All gone.
Say it's also a good thing water isn't flammable, otherwise fire trucks would show up to fires and only make the situation worse, right?
Parent
Re:UTC (Score:5, Informative)
Parent
Re:first post to say.. (Score:5, Funny)
Parent
Re:I would not want to be him. (Score:5, Funny)
Welcome to defense contracting, you must be new here.
Parent