City's IT Infrastructure Brought To Its Knees By Data Center Outage 102
An anonymous reader writes "On July 11th in Calgary, Canada, a fire and explosion was reported at the Shaw Communications headquarters. This took down a large swath of IT infrastructure, including Shaw's telephone and Internet customers, local radio stations, emergency 911 services, provincial services such Alberta Health Services computers, and Alberta Registries. One news site reports that 'The building was designed with network backups, but the explosion damaged those systems as well.' No doubt this has been a hard lesson on how NOT to host critical public services."
Maybe the city/provinces should skip on redundancy (Score:3, Interesting)
The issue with the city/provincial critical services is that they didn't have geographical redundancy due to the cost. Yes the building had redundant power, and networks but it was the whole building that was affected by this. At the end of the day, Shaw did fuck up, but all the essential servers completely fucked up.
What really happened... (Score:4, Interesting)
Shaw had a generator overheat and literally blow up which damaged their other 2 generators and caused an electrical arc fire. This fire set off the sprinklers and in turn, the water shut down the backup systems.
Yes, it was stupid that Shaw housed all their critical systems, including backups, in one building but even more stupid was the fact that they used a water based sprinkler system in a bloody telecom room.
Also, Alberta has this wonderful thing called Alberta SuperNet, which, if I recall, all health regions use to use before our government decided to spend hundreds of millions of dollars to merge everything together and spend even more money to use the Shaw network to connect everything. The SuperNet was specifically designed with government offices in mind but nooo, why use something you have already paid for when you can spend more money and use something different.
Re:Limitations (Score:2, Interesting)
It's kind of funny for me to hear someone call Shaw one of the "big boys" after working on a number of Telecom projects in the USA. In the scheme of things Shaw would only rate being maybe a tier 3 player. Their maximum customer base is maybe 8 million potential people .... not households (and I'm presupposing they are in Saskatchewan and Manitoba now otherwise subtract a couple million). And they compete with Telus and Manitoba Tel and Sasktel (or whoever it is there). That's definitely tier 3 or smaller. Heck, Bell Canada and Rogers are considered tier 2 and their market is probably 15 million or more. FYI Sprint USA has 50 million plus accounts, AT&T has at least 300 million accounts.
I told a Telus (Shaw's major and bigger competitor) senior manager in a conversation one time that it was ridiculous that they made employees pay for their coffee and had the gall to close the cafeteria at 2:30. He said "do you know how many employees we have?! 20,000!" I said I had worked on campuses of companies that had more people than that... and they bought the coffee... I guess you don't think much of your employees. He didn't believe there were business campuses that big. People in Canada don't get that we're not that big in the scheme of things world wide and like it or not it's why we have to work with out neighbor and stop insulting them with our Napoleon syndrome antics. Land area doesn't make up for relatively small population.
Re:No Site Level Resiliency? (Score:3, Interesting)
Imagine if the library of Alexandria had backup copies of all those books, manuscripts and other treasures? How about Constantinople? I'm sure there were people that tried to protect that data who believed it was worth more then their life. I hope that brings stuff into perspective.