Forgot your password?
typodupeerror
Government IT

Power Problems Force Seattle To Throttle City Data Center For Days 85

Posted by timothy
from the but-shut-down-is-a-binary dept.
Nerval's Lobster writes with an except from sister site SlashDataCenter: "On Aug. 23, Mayor Mike McGinn of Seattle informed residents that the city would partially shut down its municipal data center for five days including the Labor Day weekend. As a result, city residents will be unable to pay bills, apply for business licenses, or take advantage of other online services. In a Webcast press conference, McGinn isolated the issue as a failure in one of the electrical 'buses' that supplies power to the data center. Because that piece of equipment began overheating, the city had to begin taking servers and applications offline to prevent overloading the system. The maintenance will cost the city $2.1 million of its maintenance budget. A second power bus will remain operational, supplying enough electricity to power redundant systems for critical life and fire safety systems, including 911 services and fire dispatch. The city's Web sites should also be up and running in some capacity."
This discussion has been archived. No new comments can be posted.

Power Problems Force Seattle To Throttle City Data Center For Days

Comments Filter:
  • by Anonymous Coward on Friday August 24, 2012 @08:22PM (#41117739)

    That should help the situation.

    • by Artaxs (1002024)
      Article doesn't mention the power draw from PAX and Guild Wars 2 launching on the same weekend?
  • While looking at the prospect of moving to Seattle, I've read repeatedly read that the city is in political gridlock and seems totally unable to get any meaningful long term infrastructure additions put in place despite wide support for them. It seems to me this is the case in many cities out there, but can anyone say what it's like in their city?
    I suspect that the first ones to finally get something useful done (rather than just repaving a few highways) will be the ones to reap a lot of growth when this re

    • by 93 Escort Wagon (326346) on Friday August 24, 2012 @09:56PM (#41118675)

      I live south of Seattle, and work in the city.

      Any political gridlock is largely because current Mayor McGinn is a joke. Seattle is a fairly liberal city, but McGinn was largely seen as too extremely left-wing to be electable even there; so he remade himself into a pragmatist - a change that lasted until he was sworn in. McGinn made specific promises pre-election that he wouldn't let his personal ideology affect policies where the citizenry clearly differed from him... then he turned around and spent most of his time fighting ideological political battles, ignoring real problems while devoting 100% of his time tilting at his personal windmills.

      • by Anonymous Coward

        It's definitely gotten worse under the current mayor, but we've had gridlock before. We're just now breaking ground on the viaduct replacement over 20 years and one earthquake after we learned that the engineering design was flawed. But the current mayor is going to be a one term mayor, he's been so bad that I actually deeply regret not having voted for Nickels in the primary that year. I remember hearing the results the next day and having an "oh shit" moment when I realized that Nickels wasn't even an opt

        • Yeah, actually you are right that gridlock is not really new up here. McGinn is just such a breathtakingly bad mayor though.

          I find it a bit funny, because in Seattle it's not like there's any real right versus left division - all the crap that goes on is about individual agendas. Plus people want all this stuff and want new laws to save puppies and orphans and transgendered left-handed bicyclists, but don't really want to pay for any of it - it surprises me how anti-tax Seattle can be for such an ostensibly

          • "Plus people want all this stuff and want new laws to save puppies and orphans and transgendered left-handed bicyclists, but don't really want to pay for any of it

            I think you just described every country ever. I'm sure if you went through the cruniform tablets dug up from Babylon you'd eventually find a letter complaining that there aren't enough guards on the street and the taxes are too high.
        • Re: (Score:2, Funny)

          by drainbramage (588291)

          For those of you not from seattle:
          He is complaining that he voted for the democrat instead of the other democrat.

      • by cthulhu11 (842924)

        McGinn made specific promises pre-election that he wouldn't let his personal ideology affect policies where the citizenry clearly differed from him... then he turned around and spent most of his time fighting ideological political battles, ignoring real problems while devoting 100% of his time tilting at his personal windmills.

        Sounds kinda like Schwartz and Sun.

    • by Anonymous Coward

      That's probably the only significant thing that's wrong with the city. There's tons of activists and there's a lot of good folks that believe they're entitled to get their way every time. But, in all honesty, it tends to get sorted out and it really doesn't have as much of a negative impact as you might think. For the most part there isn't a whole lot that really needs fixing that isn't already addressed.

    • by Anonymous Coward

      We should be so lucky. Political gridlock would be wonderful. However, that is not the case. We have very active political drivers in this city.

      I used to only understand such issues theoretically, but I've seen their application in plain use all around me in this city. One of my roommates is an ex-architect who is getting deep in the politics of the council's plans for the capital hill area. It is a constant fight between city planners selling off blocks of space to the highest briber(high rise developers w

    • by symbolset (646467) * on Friday August 24, 2012 @11:56PM (#41119589) Homepage Journal
      Seattle has great parking. You can park your car on I5 for several hours each day without concern that traffic might move forward while you're shopping.
    • Seattle is 95% Democrat and has been for decades.
      The remaining 5% consists of 'magic' voters, libertarians and socialists.
      Most of whom cannot define the terms.
      --
      If you're guessing: I moved away from there.

  • by Frosty Piss (770223) * on Friday August 24, 2012 @08:30PM (#41117849)

    Interesting that this is not on the front page of the Seattle Times. In fact, I can't find it at Washington's biggest paper at all.

    • by cthulhu11 (842924)
      The "Seattle Times" really needs to come clean and just rename itself the "Microsoft Times". This story didn't concern Microcultists, so it doesn't rate their attention.
  • If you lived in podunk nowhere then no probably not, if emergency services continue to operate it wouldn't be a big issue. But for such a large municipality to go dark for 5 days...would definitely be impactful locally and possibly regionally/nationally to a smaller degree. Emergency services are very important but the business of government (no matter how i feel about it from time to time) needs to continue and serve it's people...I am sure (at least i hope) that they looked into portable power generatio
    • by jrmcferren (935335) <robbie,mcferren&gmail,com> on Friday August 24, 2012 @08:41PM (#41117977) Journal

      They have the power, they just can't get it where they need it without equipment overheating. Since it is a busbar overheating you can't just switch over to emergency power to fix it, you have to route power around the issue which is not economically feasible in this case except for the emergency services systems which can use their redundant power supplies.

      • Well being able to keep up emergency services is definitely most important, i don't think we are getting the whole story since either something was added to create extra electrical draw or something is failing. I wonder if that is the 2.1 million spoken of to add capacity...or repair.
      • by dnay (86973)

        They have the power, they just can't get it where they need it without equipment overheating. Since it is a busbar overheating you can't just switch over to emergency power to fix it, you have to route power around the issue which is not economically feasible in this case except for the emergency services systems which can use their redundant power supplies.

        Run down to Autozone and grab a couple dozen jumper cables.

      • by Isaac-1 (233099)

        Am I the only one to think, how many modern servers does the city of Seattle really need? Google says the population is only 608,000 in 2010

        • by TubeSteak (669689)

          Am I the only one to think, how many modern servers does the city of Seattle really need?

          By my calculations, the city of Seattle needs exactly two electrical buses worth of modern servers.

        • Re: (Score:3, Funny)

          by Anonymous Coward

          The city of Seattle, or any modern city, needs exactly three modern servers to provide their public services. And two of them are to provide redundancy for the one that does the actual work. Internally they may need more servers for VDI or some such, or need to physically isolate one service from another. But one modern server is adequate to provide all of the public services Seattle provides, and two more provide geographic redundancy through their fiber network, which could be upgraded to 100 Gig for a

  • If bills don't get paid, there better not be any late fees imposed. The banks could make millions on this.

  • by Anonymous Coward

    iCarly will be pissed.

  • by koan (80826)

    Sounds like Seattle's 911 system is quite fragile.

  • If power problems are downing the city's datacenter for a holiday weekend, couldn't they just rent a few $100/mo servers and run the city apps on them for the downtime and make the problems transparent to the end user? No one-place site is ever safe for important apps, we call that a Single Point of Failure around here.
  • by Lord Ender (156273) on Friday August 24, 2012 @09:22PM (#41118397) Homepage

    Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!

    • It's also the home of Microsoft; and Google is also strongly represented. You can't afford to piss off any of these guys...

    • Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!

      Migrating huge amounts of data and services is very expensive, and especially difficult to do in years when the tax revenue is down. Government is also typically more conservative with new technologies and processes than the private sector., and apprehensive about outsourcing when proper stewardship of citizens' data is their #1 priority.

  • by girlintraining (1395911) on Friday August 24, 2012 @09:58PM (#41118687)

    What I'm trying to figure out is why 911 and emergency services didn't have a separate offsite backup. I mean, how much more mission critical can you get than that? Everytime I see one of these articles I think to myself: Why are they mentioning this if there wasn't some risk of failure? And the answer is... because quite obviously, there was some risk.

    I don't want my cause of death to be "Your call could not be completed as dialed. Please check the number and try your call again later..."

    • by adolf (21054)

      In my experience with 911 and emergency communications (none of which is anywhere near the scale of what Seattle must have), they have power redundancy (consisting of one or more UPS and one or more standby generator), connectivity redundancy (multiple telephone/data circuits going to different places), and physical redundancy.

      So if one 911 PSAP goes completely offline for any reason, there is one or more geographically independent backup PSAPs which can take over in quid-pro-quo fashion.

      Do things get a lit

      • by gsogeek (1146905)

        Efficiency goes to shit, but more hands are easily called in/moved around to help with that in short order.

        So. The 911 phone will still be answered, and your ambulance/fire brigade/armed posse is still within easy grasp.

        While the redundancy is built into the system to allow the call to go somewhere, it may not be a place that can handle the call the best. We try to build the systems to account for these failovers, but even then, calltakers/dispatchers will start working on a sort of "muscle memory" when things get bad and busy. Just because the 911 phone is answered, doesn't mean that the ambulance/fire brigade/armed posse is within easy grasp. It's not as simple as adding capacity if the extra capacity is just as unaware

        • by adolf (21054)

          No system is without faults.

          But in this context (the Seattle non-fiasco), it doesn't seem to be a big deal. Things were/are fine.

          That said: Everyone wants and expects their 911 services to be absolutely bullet-proof, but nobody paying for it gives a fuck about the funding for that. 911 (in these parts) is funded in ways that are more straight-forward than gasoline taxes, and relatively easily understood. And these taxes are currently on the chopping block, for the benefit of no-one and the detriment of

  • Nice, so they're running their mission critical operations on reserve systems. Hope nothing too important happens while they're getting bombed by a /. post.

  • by hawguy (1600213) on Friday August 24, 2012 @11:04PM (#41119241)

    Why don't they just fail over the critical life and fire safety systems to the backup datacenter, and keep normal services up at the primary datacenter while they do the work? They do have a second site, right? Surely no one would host a system deemed "critical" and "life safety" at a single site?

    • Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much money we can save". Such is the way of things.

      If you personally are worried about 911 services being out then go write down the various 7 (or 10-digit if your exchange requires it) numbers for your local emergency services. 911 is not an exclusive to reach the

      • by hawguy (1600213)

        Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much money we can save". Such is the way of things.

        If you personally are worried about 911 services being out then go write down the various 7 (or 10-digit if your exchange requires it) numbers for your local emergency services. 911 is not an exclusive to reach them, just the easiest. Whatever happened to the days of the list of those various numbers on the fridge? I'm not even that old and I remember my parents having the list posted just in case.

        I thought I was already paying for a reliable E-911 service through the 911 service fees we've all been paying on our phone bills for years.

        So what you're saying is that even though we've been paying for 911 for years, we've been paying for cheap, non-redundant service, and it we expect the type of multi-site redundancy that's normally reserved for moderately successful websites, then we need to pay even more? What value are we getting from the hundreds of millions of dollars already collected?

        I've called 9

        • Next you're going to tell me that the USF fees are always used precisely for what they say they're for.

      • by johnnick (188363)

        >Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major
        >issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much >money we can save". Such is the way of things.

        Part of this is also people who are bad at math. I once had a major disagreement with a business guy trying to explain that there was a significant difference between a server that had been 100% availabl

        • With 911 services the other is consolidation. In my area there used to be multiple centers with dispatchers for various agencies in diverse locations. By nature it was redundant. If one had problems the others could assist. Now there's just a single unified emergency communications center for everyone. Even if there is a backup site or plan all of the dispatchers are on duty are only at the unified center and it takes time to shift things around.

    • by CAIMLAS (41445)

      You just don't fail over to an off-site facility when you're short staffed and haven't thoroughly tested your off-site. Very few locations can effectively fail to an off-site location gracefully for one reason or another.

    • by forkazoo (138186)

      Presumably because then, there won't be a backup available for the critical systems. There probably is some extensive backup infrastructure available, but you never activate it unless you genuinely *absolutely have to.* If something bad happens to the active systems while you have voluntarily taken down half your 911 infrastructure, "we didn't want to take down any convenience systems," really won't cut it as an excuse. Besides, the presumed backup probably isn't seamless, doesn't work quite as well, etc

  • overheating power buses / wires are a fire risk and that comes from them being under sized for the load.

    See the towering inferno to see where that can get you.

    • by thegarbz (1787294)

      and that comes from them being under sized for the load.

      While you're technically right they may not be undersized for the load. Very few substations suddenly become so warm because they are sized incorrectly for the the loads they are driving under normal conditions.

      The vast majority of overheating in switchgear comes form either malfunctioning equipment (overdrawing current without tripping out), maintenance problems (dust and such, though more of a problem in remote RMUs), or of faulty installations or faults developing over time causing poor connections.

      Last

  • McGinn had quite a few facts wrong in the press conference. The equipment is working fine now and the overheating only caused a minor amount of downtime. The major issue though was the backup generator never kicked in because as it turns out, the electric starter for the diesel generator is connected to the same bus. Labor Day weekend was then choosen to fix this majorly obvious design deficiency.

  • If you've got enough compute resources to have your own data center, it's probably cheaper for you to have your own data center instead of paying someone else to do it for you. So then, you build your own data center, and you decide on compromising on certain things like power bus redundancy, because in any given data center environment there are a million things that can fail, but you have to prioritize the systems you make redundant by looking at their failure probability and expected failure impact. Y

If you're not part of the solution, you're part of the precipitate.

Working...