Want to read Slashdot from your mobile device? Point it at m.slashdot.org and keep reading!

 


Forgot your password?
Close
typodupeerror
×
Bug Software News

Software Glitch Caused 911 Outage For 11 Million People 115

Posted by Soulskill from the off-by-911-error dept.
HughPickens.com writes: Brian Fung reports at the Washington Post that earlier this year emergency services went dark for over six hours for more than 11 million people across seven states. "The outage may have gone unnoticed by some, but for the more than 6,000 people trying to reach help, April 9 may well have been the scariest time of their lives." In a 40-page report (PDF), the FCC found that an entirely preventable software error was responsible for causing 911 service to drop. "It could have been prevented. But it was not," the FCC's report reads. "The causes of this outage highlight vulnerabilities of networks as they transition from the long-familiar methods of reaching 911 to [Internet Protocol]-supported technologies."

On April 9, the software responsible for assigning the identifying code to each incoming 911 call maxed out at a pre-set limit; the counter literally stopped counting at 40 million calls. As a result, the routing system stopped accepting new calls, leading to a bottleneck and a series of cascading failures elsewhere in the 911 infrastructure. Adm. David Simpson, the FCC's chief of public safety and homeland security, says having a single backup does not provide the kind of reliability that is ideal for 911. "Miami is kind of prone to hurricanes. Had a hurricane come at the same time [as the multi-state outage], we would not have had that failover, perhaps. So I think there needs to be more [distribution of 911 capabilities]."
This discussion has been archived. No new comments can be posted.

Software Glitch Caused 911 Outage For 11 Million People More

Software Glitch Caused 911 Outage For 11 Million People

Comments Filter:

  • backup for 911 (Score:5, Informative)

    by iggymanz ( 596061 ) on Wednesday October 22, 2014 @10:13AM (#48203769)

    have your local police and fire phone numbers in your cell phone and posted next to your land line.

    • Which is kind of ok as long as your problem happens in that area, and there is no actual natural or man-made disaster in play.

      A 911 dispatch center has much better resources than a local police or fire department main desk. If you are lucky, they can operate at 10% of the capacity as a proper dispatch center.

      The system should be more robust. It has improved dramatically since 9/11 with absurd amounts of cash poured into many facilities, but it can't do everything. An alternative solution is to not expect

      • Re: (Score:2)

        by tlhIngan ( 30335 )

        It actually is as part of 911-NG (next generation) which is meant to better handle next-generation telephony systems as well. Right now cellphones and VoIP 911 is more of a hack than anything in the current system, while the next-gen system switches to a completely VoIP (over a private network) system with failover and tagging and all that.

        So the next-gen system will allow 911 to be contacted in many ways, including texts and SMS, VoIP (both private and internet), POTS, cellphones and all sorts of other mec

    • Have you ever actually called those?

      They say "If this is an emergency, please hang up and dial 911".

      • use your brain, you have no point

        • I think I specifically have a point that they're not a substitute for emergency calls.

          • Re: (Score:1)

            by umghhh ( 965931 )
            so if there is no emergency calls what then? we all go suck your dick because we have to follow some rule that assumes emergency calls work? let us say that 30% of those 6k people that could not reach emergency number came to your dick for sucking you will be terribly sorry for expressing silly views on how the rules are superior to reality.

    • whats a landline. also, you can just google the numbers or ask siri.

      • Re: (Score:1)

        by sremick ( 91371 )

        whats a landline.

        If you care enough about 911 and emergency situations to be reading this article, and you don't have a landline, then that's on you for being irresponsible. People spend more on texting than it costs to have a landline. No excuses.

        The monthly cost of a landline is cheap insurance in the event of an emergency. Cell towers go down, fail, become over-congested, and cell phone batteries die.

    • Re: (Score:2)

      by David_W ( 35680 )
      Or failing that, do what we used to do before 911 was implemented in our town: Call the operator, tell them who you want (police, fire, ambulance) and that it is an emergency. Not quite as effective since it lacks the central control 911 has, but it generally works. (And in some areas the operator may have the back-end number for 911 and can transfer you to that, which could also work.)

      • Re: (Score:1)

        by Cramer ( 69040 )

        30 years ago, MAYBE. Today, "0" could be answered by a call center anywhere. Your call a) might not be answered by a human at all, or b) might not be answered by someone on the same continent.

    • What is this land line you speak of?

      • Re: (Score:1)

        by sremick ( 91371 )

        What is this land line you speak of?

        If you care enough about 911 and emergency situations to be reading this article, and you don't have a landline, then that's on you for being irresponsible. People spend more on texting than it costs to have a landline. No excuses.

        The monthly cost of a landline is cheap insurance in the event of an emergency. Cell towers go down, fail, become over-congested, and cell phone batteries die.

        • Re: (Score:2)

          by dgatwood ( 11270 )

          The monthly cost of a landline is cheap insurance in the event of an emergency. Cell towers go down, fail, become over-congested, and cell phone batteries die.

          Not around here. I'm paying about $40 per month for a nearly bare-bones land line (only Caller ID). Even if I were on a $0.35 per text plan, I'd spend more money on that land line every month than I would on texting for ten years. Cheap, it ain't.

        • Re: (Score:1)

          by ShaunC ( 203807 )

          If you care enough about 911 and emergency situations to be reading this article, and you don't have a landline, then that's on you for being irresponsible. People spend more on texting than it costs to have a landline.

          Every line in the US is required by law to be able to dial 911, even if you aren't paying for any service at all. This applies to landlines and cellphones. I often keep the police scanner on as background noise, and I can say at least a quarter of the 911 calls in Memphis originate from disconnected cell phones. If your landline gets a dial tone or your cellphone is charged and has a signal, you can dial 911 whether you have a phone plan or not.

          Of course that doesn't help in the event of a 911 outage but th

    • have your local police and fire phone numbers in your cell phone and posted next to your land line.

      That is a great idea.
      But, I used to handle 911 outages. Most 911 outages are due to cable cuts, which would often leave those facilities unreachable as well.

      I'd say that if your phone works, and you can't call 911 or the local hospital, you should assume the trunk leading to those services (foolishly all usually located next to each other) is cut or damaged. So your next best bet would be to call a NON-LOCAL ER. i.e. Call the next town over. Just because downtown is broken doesn't mean the trunk leading to

    • Re: (Score:2)

      by antdude ( 79039 )

      What if one travels a lot in USA? 911 does local ones. Can 411 go to 911?

  • 40 millions doesn't seem to cross any boundary?

    • Re: (Score:2, Insightful)

      by Anonymous Coward

      Probably crosses a licensing boundary. Beats me!

      "We're sorry, your 911 call centre didn't pay their software licensing fee this month. Please call 1-800-RU-LEGIT and report this instance."

    • I think I found the offending code:

      static unsigned int counter[640] // ought to be enough

      ...

      void countcall() {
          int i;
          for (i = 0; counter[i] = 65535; i++) {} // skip the ones we filled up
          counter[i]++;
      }

  • killbots have a preset kill limit (Score:1)

    by Anonymous Coward

    You see, killbots have a preset kill limit. Knowing their weakness, I sent wave after wave of my own men at them until they reached their limit

  • sounds like some dummy read a best practices and conserve the resources book and made a column an interger data type instead of big interger. or whatever the corresponding names are for Oracle or non-MSSQL. some auto process or identity column creates the keys and it reached the max amount. and it wasn't set up to use negative numbers either

  • Comment removed (Score:5, Informative)

    by account_deleted ( 4530225 ) on Wednesday October 22, 2014 @10:27AM (#48203873)
    Comment removed based on user account deletion

  • possible solution, fallback to voicemail (Score:1)

    by Anonymous Coward

    Thank you for calling Springfield RescuePhone!
    if you know the name of the crime being committed, press one!
    To choose from a list of felonies press two!
    If you are being murdered or are calling from a rotary phone please stay on the line!

  • An entirely preventable software error was responsible for causing 911 service to drop. "It could have been prevented. But it was not,"

    So, let us be clear. The error, was not simply preventable but absolutely and completely preventable in all cases. There was no impediment to prevent it. Its prevention was not only possible but also within the reach of any error prevention effort or action. It could have been prevented.

    The preventability of the error was absolute. No situation, fictive or factual, in this or other world, would allow a situation in which this error was not preventable.

    Finally, it's important to note that the eventual series

    • Re: (Score:2)

      by cdrudge ( 68377 )

      All kidding aside, when I read that line I wondered what type of software error isn't preventable. There's things that are easily preventable and should be thought of, but ANYTHING is "entirely preventable".

      • Re: (Score:2)

        by danlip ( 737336 )

        Bad wording. Perhaps "obvious and preventable" would have been better (and why does the counter not go to at least max int, e.g. 2^31-1 = 2,147,483,647).

  • the FCC found that an entirely preventable software error was responsible for causing 911 service to drop

    As opposed to what? An entirely unpreventable software error? Sounds more like a configuration issue than a software error anyway.

  • Failure started at the Administrative level.... (Score:3)

    by Lumpy ( 12016 ) on Wednesday October 22, 2014 @10:48AM (#48204057) Homepage

    It's the fault of the administrators to begin with. I am friends with one of the technical advisors for the midwest EOC and the problem is that the administrators dont know their ass from a hole in the ground and ignore their tech guys and listen to the vendors.

    He has been screaming for all call centers to have analog failover, but the administrators refuse to hear it.

    So who is to blame for the failures? That top moron of Homeland security. IT would have been in place if he would realize that he is not an expert and to actually LISTEN to the experts in the field.

    • Re: (Score:1)

      by uksv29 ( 167362 )

      I'm not convinced you need an analogue failover but you do need fully duplicated systems right down to the power subsystems and cables which you periodically switch between. There is no point having a backup if you don't use it on a regular schedule to be sure it is working properly.

      The solutions are not all technical, you have to be monitoring them properly with the right people who are motivated and properly trained. You also need the proper organisational processes .

      I've seen NOCs on emergency service n

      • Re: (Score:2)

        by uksv29 ( 167362 )

        I've been specifying emergency service systems for over 10 years, duplication, monitoring, management and processes are always at the top of the list.

  • 11 million people, 24/7/365 etc works out to 96.360 billion citizen-hours per year. Assuming these 40 million calls all happened this year and each call took place one full hour each, it works out to a load factor of 0.04%. In other words citizens are NOT calling 911, 99.96% of their time. Since most 911 calls do not last an hour the load factor is even lower than that. This is how over built and inefficient government services are.

    If a private railroad owns rolling stock that would occupy, say 10 miles of

    • Re: (Score:2)

      by sribe ( 304414 )

      This is how over built and inefficient government services are.

      That was one of the most stupid nonsensical posts I have ever seen here. You calculated the "load factor" based on each of 11,000,000 people instead of on the number of 911 operators.

      And of course that's not even counting the fact that 911 services pretty much need to be provisioned to handle *peak* loads, not average (nor even median).

      • I have failed miserably looks like. Even adding the bit about railway rolling stock did not help. Well, that is the problem when you speak with a tongue in the cheek. You end up chewing your own tongue.

        • Re: (Score:2)

          by sribe ( 304414 )

          I have failed miserably looks like. Even adding the bit about railway rolling stock did not help. Well, that is the problem when you speak with a tongue in the cheek. You end up chewing your own tongue.

          That was meant to be tongue in cheek? Oh, OK then ;-)

          Problem is, it's election season, and what you said there was really not much different than some of the bullshit that we're inundated with nightly on TV commercials, and flyers in the mail. My favorite so far is the one accusing a Democrat of attempting to "replace Medicare with a completely government-run system". Uhhhmmm, excuse me???

          • I don't blame you, politicians, especially the Republicans have gone so far out, it is nearly impossible to parody them by exaggerating anything they say or do. No wonder Jon Stewart and Steven Colbert are reduce playing back their footage unmodified and make faces.

  • If calls are lost then help is delayed. This impacts the outcome of incidents.

    I'm not saying that people died because of this but I'm absolutely certain that there were some who suffered worse injury and losses because of the delays. Loss of 6,000 calls will result in a lot of hurt.

    Like so many other issues, it wasn't a single fault but a chain of events. In this case there was a software failure but the fault monitoring systems and support services failed to immediately note that there were no calls going

  • I used to handle this... (Score:3)

    by Charliemopps ( 1157495 ) on Wednesday October 22, 2014 @11:05AM (#48204237)

    I used to work in the NOC for a large Telco and we'd handle 911 outages. Usually 911 goes down because the entire networks down. Like the switch failed, or the trunk from one area that leads to the area the 911 center is in would get cut. Most of this stuff is in a ring so there's usually an alternate route, but in some areas that's not physically possible. For example a remote mountain town with a single road in, would likely have its only trunk running along that same road and it'd get cut all the time as the road constantly needed repair. Chose where you live wisely.

    We'd handle this in different ways depending on the situation. For example, if we had 4 trunks that could handle 4X number of calls, and 3 got cut so it could only handle 1X, we could actually prioritize certain numbers so 911 and emergency services would get priority. If the trunk leading to the 911 center failed, we could do something like re-route the calls to the local police dispatcher who literally had no warning and would suddenly have their phone ringing off the hook. You may say "you should warn them!" but our policy was "Get it done" because who's dieing while you're arguing with the dispatcher about how her days going to suck?

    The most important skill you can have in any NOC is your ability to triage problems. That term comes from the medical world but it's just networking equipment... until you get into the situation I was in. And you're making triage decisions that could actually result in death. These were real engineers that really cared and did what they could. But when you have an area ravaged by hurricane and you tell the tech to put gas in generator 1 instead of 2, because you've been up for 30hrs strait... and a remote goes down so they can't call 911? I just couldn't detach myself from that. I took a pay cut to leave. A lot of people floated through that job, it wasn't just me. It takes a special kind of person that can detach themselves from the consequences of their decisions.

  • For 911 services in 7 states? Set aside issues about the backup system for the moment (which may be a second server in the same data center): Why do all the 911 calls have to funnel through a single system? Emergency services are largely local. Not many people have to make 911 calls across a large region. So why isn't the energency call routing handled by local systems? And calls routed to local service centers?

    Even if there was a common software glitch in all the handling systems, I doubt everyone would h

  • And the number "40,000,000" doesn't come up on my list of "potential overflows to watch out for". What's special about 40 million?

    • Re: (Score:2)

      by danlip ( 737336 )

      As near as I can tell from the TFA somebody just put an arbitrary limit of 40 million in the code somewhere. Would be nice to have more technical details.

  • At least there are no current / recent worries what might make someone want to call 911 ...

    Actually, it's an interesting question -- just what is the threshold? Suspected ebola vomit? Suspicious bag on suburban street? Seems like it would be a very easy system to game, or even to unintentionally render useless. Takes a lot of goodwill and good behavior, all around.

    I took a CPR class last night, and the instructor (a firefighter in his dayjob) basically encouraged people to use 911 more, even for things abou

Slashdot Top Deals

"And remember: Evil will always prevail, because Good is dumb." -- Spaceballs

Close