Become a fan of Slashdot on Facebook

 


Forgot your password?
Close
typodupeerror
×
Data Storage IT Technology

Loud Sound From Fire Alarm System Shuts Down Nasdaq's Scandinavian Data Center (bleepingcomputer.com) 114

Posted by msmash from the oh-damn dept.
Catalin Cimpanu, writing for BleepingComputer: A loud sound emitted by a fire alarm system has destroyed the hard drives of a Swedish data center, downing Nasdaq operations across Northern Europe. The incident took place in the early hours of Wednesday, April 19, and was caused by a gas-based fire alarm system that are typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment. These systems work by releasing inert gas at high speeds, a mechanism usually accompanied by a loud whistle-like sound. With non-calibrated systems, this sound can get very loud, a big no-no in data centers, where loud sounds are known to affect performance, shut down, or even destroy hard drives.
This discussion has been archived. No new comments can be posted.

Loud Sound From Fire Alarm System Shuts Down Nasdaq's Scandinavian Data Center More

Loud Sound From Fire Alarm System Shuts Down Nasdaq's Scandinavian Data Center

Comments Filter:
  • >> this sound can get very loud, a big no-no

    Jeezzez Crisco. What did we do now to get SlashDot editors talking down to us with one syllable at a time?

    • Re:Monosyllabic Much? (Score:4)

      by Nidi62 ( 1525137 ) on Friday April 20, 2018 @10:57AM (#56471255)

      >> this sound can get very loud, a big no-no Jeezzez Crisco. What did we do now to get SlashDot editors talking down to us with one syllable at a time?

      Would you prefer "the aforementioned audible emanation attained a decibel level detrimental to the proper operation of the installed data storage mechanisms"?

      • Re: (Score:1)

        by Anonymous Coward

        Yes actually. We're not in kindergarten. Ideally the writing level would at least be appropriate for a 12th grade audience.

        • I was told once that most news sources are actually written for a seventh grade reading level.

          • I wrote tech bulletins and advisories at the 8th grade level. on advice of the vendors.

            Troubleshooting and diagnosis guides and general documentation I wrote at the 5th grade level, by request, again by those with long experience writing to technical audiences. Not because they are deficient, or marginally illiterate, but because they pay too little attention to detail, are hurried, and don't really care enough. And to limit the vocabulary to the most commonly shared set. Being clever doesn't serve your a

        • Re: (Score:2)

          by Nidi62 ( 1525137 )

          Yes actually. We're not in kindergarten. Ideally the writing level would at least be appropriate for a 12th grade audience.

          Anyone can usually memorize more complicated synonyms of everyday words in order to try and sound smarter. However, a much more effective demonstration of intelligence would be showing the ability to understand what level of communication is appropriate and necessary for both the subject and target of the communication. In effect: Keep It (as)Simple (as possible) Stupid.

          • Re: (Score:3, Interesting)

            by Richard Stalin ( 5369137 )
            “It's a beautiful thing, the destruction of words. Of course the great wastage is in the verbs and adjectives, but there are hundreds of nouns that can be got rid of as well. It isn't only the synonyms; there are also the antonyms. After all, what justification is there for a word which is simply the opposite of some other words? A word contains its opposite in itself. Take 'good,' for instance. If you have a word like 'good,' what need is there for a word like 'bad'? 'Ungood' will do just as well--be

            • This is wrong, by the way, demonstrably wrong.

              'Ungood' isn't the exact opposite of 'good'. It may be better defined as the absence of good, which could be neutral or 'bad'.

              You started it, not including the sarcasm flag is your fault.

        • Re: (Score:2)

          by sjames ( 1099 )

          Nobody wants subtle nuance in a tech manual, save it for poetry and novels. Otherwise you just read like an 8th grader trying to sound sophisticated.

          • Re: (Score:2)

            by Pascoea ( 968200 )
            Exactly. That's the key to technical writing, use exactly as many words as are necessary to accurately get a point across, and no more.

    • Whelp, there go your mod privileges for life.

      Pointing out the... ah... youthful and inexperienced nature of our current crop of editors, let alone their lack of technical prowess, blatant political agenda, or patronizing manner of address to the folks who have been here just a wee bit longer than they have has consequences beyond being called an insensitive clod.

      How those who themselves are in need of a serious LARTing are now running this place is beyond me.

    • Re: (Score:2)

      by suutar ( 1860506 )

      that's a quote from TFA

    • Actually, it's the opposite (Score:1)

      by Anonymous Coward

      Reporters and bloggers cannot even get the facts right let alone write at a higher level.
      Fire alarms make noise to warn people.
      Fire suppression systems extinguish fires.
      Fire protection systems often incorporate suppression and alarm systems.

  • Total failure (Score:3)

    by Daetrin ( 576516 ) on Friday April 20, 2018 @10:51AM (#56471207)
    "caused by a gas-based fire alarm system that are typically deployed in data centers because of their ability to put out fires without destroying non-burnt equipment. [...] With non-calibrated systems, this sound can get very loud [and destroy-non burnt-equipment]"

    You had one job! (Directed at both the system itself and whoever was supposed to be in charge of calibrating it.)

    ...actually, that person may have zero job now.

    • You had one job! (Directed at both the system itself and whoever was supposed to be in charge of calibrating it.) ...actually, that person may have zero job now.

      They were probably a contractor that offered to calibrate the system but some manager decided he didn't want it coming out of their budget.

    • This is actually a common failure mode for FM200 systems (fire PROTECTION, not fire ALARM). The last I had heard about it, research was suggesting that it was the sudden pressure change, but I hadn't been following it closely since.

      A couple bank data centers were hit by it (HSBC was one) in 2016.

  • Comment removed based on user account deletion

  • Fire SUPPRESSION system not fire ALARM system (Score:5, Informative)

    by jtara ( 133429 ) on Friday April 20, 2018 @11:08AM (#56471311)

    It's a fire SUPPRESSION system, not a fire ALARM system.

    At least this time, it's the article (on Bleeping Computer) that is wrong, not the summary on /.

    It's apparent the author of the article didn't bother to read the article (on Motherboard) that she cites about a similar incident in Romania at ING Bank. It clearly states that incident resulted from a "fire extinguishing test".

    The sound BTW comes from the release of the gas, not some alarm. In both cases, the pressure was set too high. It was basically a - very loud - over 130db - hissing sound!

    The second article cites a study about the effect of this sound done by Siemens. Siemens has a vested interest. I guess neither of these data centers were equipped with the Sinorix Silent Nozzle.

    • Thank you. Moreover, I would bet that it wasn't the sound, but the particulates that did it. Whether this was Halon, or FM-200, or some other "clean agent", if you've ever seen a Data Center after one of those goes off, it's covered in powder and dust. The idea that they're truly "clean" is complete crap. Sure, they won't soak your gear in water, but it's not this lovely, clean, unobtrusive gas like the sales folks tell you it is.

      • Re: (Score:1)

        by snugge ( 229110 )

        Hard disks don't die from dust. They *do* die from too loud noises, though.

        Seen that first hand after an incident just like this one.

        • Perhaps the drives themselves don't, but the servers or DASD boxes they're sitting in are very susceptible to dust. I realize the article says the hard drives failed, but further down they talk about needing to replace the servers. I'm asserting that it was more likely the dirt and debris that ultimately caused the problem, not the sound.

      • Oh sure the GAS is clean and unobtrusive... The problem is the speed it is released and where it blows.... I've seen what ends up under the raised floors or in the rafters in a data center after a few years in operation and it's not pretty. Imagine all this dirt, dust and trash suddenly flying around.

        House keeping! Clean up in the server room!

        Then there is the whole, fire alarm pulled, kill the power to limit the damage! Use case. Modern computers do NOT take kindly to having the power yanked out from

      • Re: (Score:2)

        by Khyber ( 864651 )

        "Moreover, I would bet that it wasn't the sound, but the particulates that did it."

        All spinning rust hard drives are fully-sealed against particulates. SSDs don't need that, and the stuff used in electronics fire control tends to be non-conductive.

      • Re: (Score:2)

        by Megol ( 3135005 )

        Do they still use halon? Thought it was reserved for military and critical civilian uses only.

      • Obligatory story about new "clean agent" fire suppression systems (hint: nitrogen + water):

        Tuesday was a regular day for me. I am working remotely and live almost 2 hours away from work. At 11:02am, this chat began between my boss and me:

        Boss: Please monitor all systems. Alarm system fired in server room

        Me: I can't get on firewall

        Me: I'm off vpn

        Me: You still have internet?

        • There are so many reasons to not use water.

          Unless it's very pure, you will deliver minerals, salts, with predictable consequences, not all immediate.

          Water will also dissolve and/or deliver contaminants throughout equipment. Again, not good, and the impacts not necessarily immediate.

          As a solvent, water is different than many used in manufacturing electronic equipment, so what wasn't a problem before exposure may become later. Even high humidity can cause this.

          Pure water should be safe, but it's all the other

          • It is just regular filtered, distilled water. The high humidity is what caused our WAN router to reboot and freak out. We still haven't had any extra issues due to it, but that's not to say that it WONT happen in the future from this one event. We were not pleased with the alarm company.

    • The sound BTW comes from the release of the gas, not some alarm. In both cases, the pressure was set too high. It was basically a - very loud - over 130db - hissing sound!

      Ya, but if you don't pay for calibration and extra nozzles, you get a fire extinguishing system AND an alarm for less cost!

  • Odd (Score:4, Informative)

    by Virtucon ( 127420 ) on Friday April 20, 2018 @11:13AM (#56471349)

    When my Ex-Wife let out loud, obnoxious sounds my hard drives didn't self-destruct. My mind did, but not my hard drives.

  • These sorts of systems, have been well known for ages now to be a high risk for damaging equipment, most of them release with such force that they can do a lot of harm. As a result, many data centers, and other similar businesses have moved away from them to avoid exactly this sort of incident. A popular alternative right now is water mist, despite what people immediately think, these have actually been proven to be safe around electronics when properly designed and deployed.

    • Re: (Score:1)

      by Anonymous Coward

      These type of systems work by displacing of reacting or otherwise removing the oxygen from the room - starving the fire. I am pretty sure most businesses have moved away from them because of the liability of killing all the humans that don't make it out of the room before the gas releases.

      • Re:Well known for ages now (Score:4, Insightful)

        by Strider- ( 39683 ) on Friday April 20, 2018 @01:33PM (#56472293)

        For most common materials to burn, you need an oxygen content of 12% or higher. On the other hand, a human can maintain consciousness down to something like 5%. As such, inerting systems are designed to drop the oxygen content to something like 8%. Too low for combustion to take place, but high enough not to kill the occupants. Nitrogen type inerting systems will actually often include a small amount of CO2 in the gas mix; this causes any remaining occupants to breath harder, thus allowing them to work better in a low oxygen atmosphere.

        reference: I worked on a small power plant with a two compartment FM-200 fire suppression system. One of the things that the design Engineers needed was an accurate measurement of the room volume so that they could calibrate the amount of agent in the tanks appropriately. However, just in case, before an agent dump there is a 30 second siren and several large "Cancel dump" mushroom switches.

  • Poor Planning? (Score:3, Informative)

    by bestweasel ( 773758 ) on Friday April 20, 2018 @11:22AM (#56471395)

    Another site without replication, redundancy and automatic failover, similar to when British Airways' entire computer system collapsed because of a power surge when someone turned it off and on again.

    It's not possible to test all scenarios ("Right, now we're going to see how our network would cope if someone took an axe to that cabinet. Derek, when you're ready") but someone should have planned for a data centre failure. Is this lack of resilience common in big networks?

    • Is this lack of resilience common in big networks?

      Yes, it is. Resilience is conceptually easy to understand, but historically exceptionally hard to implement because it requires rigorous controls on such mundane things as where you plugged that server and network switch in. Because it's so hard to maintain, supposedly resilient systems often are not really. Somebody moved some power cable, switched a network port to another switch or forgot to update the standby system and their data backup scripts... Then the unthinkable happens and you discover some s

      • Yup, as simple as unplugging a power cord because 'everything's redundant, we can cover this'.

        You can do that once only.

      • And it's somewhat more complicated here since also every member of the exchange have to switch over their systems to use the secondary site, and due to their rules they cannot open unless all members have switched. Also there had already been trades between members right before and right at the moment the servers was destroyed so there where some cleaning up to do as well (all the trades had to be cancelled).

      • Re: (Score:2)

        by sjames ( 1099 )

        One of my favorites was a diesel generator with a small local tank and a larger tank installed well behind the building. Never a problem during the monthly tests where they actually failed over to the generator.

        Actual power failure happens, generator picks up the load just fine. An hour later, the generator shuts down. Turns out the transfer pump that fed the small local tank from the large tank was connected to grid power only.

    • Well they have a secondary site at which they opened trading hours later, the question that is left for Nasdaq to answer is why it took them so long to open trading on the secondary site when they and us the members take part in regularly disaster exercises where everyone switches to use the secondary site within seconds.

  • Didn't we learn not to yell at hard drives? (Score:2, Insightful)

    by Anonymous Coward

    No seriously, I can't find the video atm but many years ago there was a guy from (Sun?) showing how susceptible they are to sound vibrations. His test literally was yelling at them while watching dtrace.

    Update: Found it. Was Brendan Gregg - https://www.youtube.com/watch?v=tDacjrSCeq4

    * (headphone warning)

    • Ah, yes. I remember that video. I was quite surprised that drives could be impacted that easily just by the human voice.

  • Reminds me of an old Usenet legend... (Score:1)

    by Anonymous Coward

    Dunno if it's true or not, but I've always found it amusing nonetheless:

    https://www.hactrn.net/sra/vaxen.html

  • Pro tip: fire alarms make noise, fire protection systems put out fires.... this was the latter. A clean agent release system.
  • There's an interesting Youtube video where a server admin screams directly into a rack of spinning drives, and shows the latency / error rate spiking on the screen as he does that. I forget where it is but it's interesting.

  • If you've ever wondered what it sounded like.. (Score:3)

    by Alpha232 ( 922118 ) on Friday April 20, 2018 @12:22PM (#56471757)

    This sounds like an Inergen type system... the two failings are the particulate matter it stirs up and the vibration caused by the sounds of a freight train rolling through...

    https://youtu.be/yM80eBR_b2w [youtu.be]

  • Something's not quite right about that. For the year 2018, at least.

  • A fire alarm system isn't designed to extinguish a fire, a suppression system is. I hate to be pedantic, but there is a difference. The device in your home that disturbs fajita night is an alarm system...

  • Did anyone bother to check if there was an actual fire or are they too busy kvetching about the noise?

  • Can confirm (Score:3)

    by RabidMonkey ( 30447 ) <canadaboy.gmail@com> on Friday April 20, 2018 @06:28PM (#56474635) Homepage

    At a previous employer, we lost an entire row of servers in a DC after a water leak (somehow) triggered the suppression system. The 'explosion' was strong enough to knock the doors off cabinets, bend 2 cabinets, and cause a couple hundreds drives to be dead. Thankfully our service was spread out far enough to survive the loss of a row for a few week while we waited for all new disks to arrive from IBM.

    The pictures were crazy, it looks like a bomb went off.

Slashdot Top Deals

Two can Live as Cheaply as One for Half as Long. -- Howard Kandel

Close