Power Problems Force Seattle To Throttle City Data Center For Days 85
Nerval's Lobster writes with an except from sister site SlashDataCenter: "On Aug. 23, Mayor Mike McGinn of Seattle informed residents that the city would partially shut down its municipal data center for five days including the Labor Day weekend. As a result, city residents will be unable to pay bills, apply for business licenses, or take advantage of other online services. In a Webcast press conference, McGinn isolated the issue as a failure in one of the electrical 'buses' that supplies power to the data center. Because that piece of equipment began overheating, the city had to begin taking servers and applications offline to prevent overloading the system. The maintenance will cost the city $2.1 million of its maintenance budget. A second power bus will remain operational, supplying enough electricity to power redundant systems for critical life and fire safety systems, including 911 services and fire dispatch. The city's Web sites should also be up and running in some capacity."
Hey, so let's post it to Slashdot! (Score:4, Funny)
That should help the situation.
Re: (Score:2)
Re: (Score:1)
Problem is that it contained atleast bill paying and other personal information, and the government has very strict laws about how stuff like that is to be stored and processed. Even if they can get over the red tape just for using a private vendor, there are laws saying that they have to use vendors that meet certain special interest criteria, and then automatically pick the lowest price because of budget laws. In the end they get a data center that is down on weekends, holidays, and all other cruft for a
Re: (Score:1)
Feel free to bid out the project, and then see if it's worthwhile. Check out the costs of something being under another entity control.
In this case, they COULD pay the extra to have this fixed while running, but for them, I'm guessing the temporary shutdown over a 3-day weekend is the more cost-effective option. With two days on either side, it's hardly a gross inconvenience. The city's key operations will go on.
Or just assume that contracting out is the better way, and sprinkle on a little magic fairy d
Re: (Score:3)
Re: (Score:3)
I had an almost identical situation happen to me this past spring, too. I was the sysadmin at one of the facilities. It happened right after I gave my two weeks, and damn was I busy. :P I ended up having to take all my UPSes off the mains and run them over some two phase at one point to get additional power onto a secondary genset, because the amp load simply was too high (oops, poor planning - someone forgot to figure high load overhead amperage requirements).
Unlike this situation, my situation only had a
Re: (Score:2)
Diesel here in CA is under $4, usually a few cents less than unleaded, and in any case, generators can use the road-tax-free supply (ala. Home heating oil) which drops the price significantly, still.
Re: (Score:2)
when was the last time a well-run private datacenter was offline for five days barring a natural disaster?
Boeing's datacenter in Bellevue Washington (East of Seattle about 10 miles). About a decade ago, they had to shut down their entire operation because a purpose-designed and built data center had power problems and didn't have the system redundancy they thought it did. In this case, the problem had to do with the use of incorrectly specified parts in some panelboards (main lug bolts). When a few were discovered to be overheating due to loose connections, the extent of the problem was revealed.
That datacente
Looking to move to Seattle... (Score:1)
While looking at the prospect of moving to Seattle, I've read repeatedly read that the city is in political gridlock and seems totally unable to get any meaningful long term infrastructure additions put in place despite wide support for them. It seems to me this is the case in many cities out there, but can anyone say what it's like in their city?
I suspect that the first ones to finally get something useful done (rather than just repaving a few highways) will be the ones to reap a lot of growth when this re
Re:Looking to move to Seattle... (Score:4, Informative)
I live south of Seattle, and work in the city.
Any political gridlock is largely because current Mayor McGinn is a joke. Seattle is a fairly liberal city, but McGinn was largely seen as too extremely left-wing to be electable even there; so he remade himself into a pragmatist - a change that lasted until he was sworn in. McGinn made specific promises pre-election that he wouldn't let his personal ideology affect policies where the citizenry clearly differed from him... then he turned around and spent most of his time fighting ideological political battles, ignoring real problems while devoting 100% of his time tilting at his personal windmills.
Re: (Score:1)
It's definitely gotten worse under the current mayor, but we've had gridlock before. We're just now breaking ground on the viaduct replacement over 20 years and one earthquake after we learned that the engineering design was flawed. But the current mayor is going to be a one term mayor, he's been so bad that I actually deeply regret not having voted for Nickels in the primary that year. I remember hearing the results the next day and having an "oh shit" moment when I realized that Nickels wasn't even an opt
Re: (Score:2)
Yeah, actually you are right that gridlock is not really new up here. McGinn is just such a breathtakingly bad mayor though.
I find it a bit funny, because in Seattle it's not like there's any real right versus left division - all the crap that goes on is about individual agendas. Plus people want all this stuff and want new laws to save puppies and orphans and transgendered left-handed bicyclists, but don't really want to pay for any of it - it surprises me how anti-tax Seattle can be for such an ostensibly
Re: (Score:2)
I think you just described every country ever. I'm sure if you went through the cruniform tablets dug up from Babylon you'd eventually find a letter complaining that there aren't enough guards on the street and the taxes are too high.
Re: (Score:2, Funny)
For those of you not from seattle:
He is complaining that he voted for the democrat instead of the other democrat.
Re: (Score:1)
McGinn made specific promises pre-election that he wouldn't let his personal ideology affect policies where the citizenry clearly differed from him... then he turned around and spent most of his time fighting ideological political battles, ignoring real problems while devoting 100% of his time tilting at his personal windmills.
Sounds kinda like Schwartz and Sun.
Re: (Score:1)
That's probably the only significant thing that's wrong with the city. There's tons of activists and there's a lot of good folks that believe they're entitled to get their way every time. But, in all honesty, it tends to get sorted out and it really doesn't have as much of a negative impact as you might think. For the most part there isn't a whole lot that really needs fixing that isn't already addressed.
Re: (Score:1)
We should be so lucky. Political gridlock would be wonderful. However, that is not the case. We have very active political drivers in this city.
I used to only understand such issues theoretically, but I've seen their application in plain use all around me in this city. One of my roommates is an ex-architect who is getting deep in the politics of the council's plans for the capital hill area. It is a constant fight between city planners selling off blocks of space to the highest briber(high rise developers w
Re: (Score:2)
Please give him some slack, he graduated from a Seattle school.
Re:Looking to move to Seattle... (Score:5, Funny)
What the hell were you reading? (Score:1)
Seattle is 95% Democrat and has been for decades.
The remaining 5% consists of 'magic' voters, libertarians and socialists.
Most of whom cannot define the terms.
--
If you're guessing: I moved away from there.
Seattle Times, Where Are You? (Score:3)
Interesting that this is not on the front page of the Seattle Times. In fact, I can't find it at Washington's biggest paper at all.
Re: (Score:1)
5 days no government, is that so bad? (Score:2)
Re:5 days no government, is that so bad? (Score:4, Insightful)
They have the power, they just can't get it where they need it without equipment overheating. Since it is a busbar overheating you can't just switch over to emergency power to fix it, you have to route power around the issue which is not economically feasible in this case except for the emergency services systems which can use their redundant power supplies.
Re: (Score:2)
Re: (Score:2)
They have the power, they just can't get it where they need it without equipment overheating. Since it is a busbar overheating you can't just switch over to emergency power to fix it, you have to route power around the issue which is not economically feasible in this case except for the emergency services systems which can use their redundant power supplies.
Run down to Autozone and grab a couple dozen jumper cables.
Re: (Score:3)
Am I the only one to think, how many modern servers does the city of Seattle really need? Google says the population is only 608,000 in 2010
Re: (Score:2)
Am I the only one to think, how many modern servers does the city of Seattle really need?
By my calculations, the city of Seattle needs exactly two electrical buses worth of modern servers.
Re: (Score:3, Funny)
The city of Seattle, or any modern city, needs exactly three modern servers to provide their public services. And two of them are to provide redundancy for the one that does the actual work. Internally they may need more servers for VDI or some such, or need to physically isolate one service from another. But one modern server is adequate to provide all of the public services Seattle provides, and two more provide geographic redundancy through their fiber network, which could be upgraded to 100 Gig for a
The dog ate my data center (Score:1)
If bills don't get paid, there better not be any late fees imposed. The banks could make millions on this.
Oh crap.... (Score:1)
iCarly will be pissed.
Re: (Score:3)
Re: (Score:2)
The cloud is somewhere else with their own power problems.
Wow (Score:1)
Sounds like Seattle's 911 system is quite fragile.
Forgetting something? (Score:2)
Re: (Score:2)
No. They should get the money to put in 2:3 redundancy separate data centers, away from downtown, in widely separated locations.
the cloud (Score:3)
Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!
Re: (Score:3)
It's also the home of Microsoft; and Google is also strongly represented. You can't afford to piss off any of these guys...
Re: (Score:2)
Seattle? The home of Amazon? Why on earth don't they just move their datacenter to Amazon Web Services? They could probably do it for less than the $2.1 million they're spending on this single part!
Migrating huge amounts of data and services is very expensive, and especially difficult to do in years when the tax revenue is down. Government is also typically more conservative with new technologies and processes than the private sector., and apprehensive about outsourcing when proper stewardship of citizens' data is their #1 priority.
911 and emergency services (Score:5, Insightful)
What I'm trying to figure out is why 911 and emergency services didn't have a separate offsite backup. I mean, how much more mission critical can you get than that? Everytime I see one of these articles I think to myself: Why are they mentioning this if there wasn't some risk of failure? And the answer is... because quite obviously, there was some risk.
I don't want my cause of death to be "Your call could not be completed as dialed. Please check the number and try your call again later..."
Re: (Score:3)
In my experience with 911 and emergency communications (none of which is anywhere near the scale of what Seattle must have), they have power redundancy (consisting of one or more UPS and one or more standby generator), connectivity redundancy (multiple telephone/data circuits going to different places), and physical redundancy.
So if one 911 PSAP goes completely offline for any reason, there is one or more geographically independent backup PSAPs which can take over in quid-pro-quo fashion.
Do things get a lit
Re: (Score:1)
Efficiency goes to shit, but more hands are easily called in/moved around to help with that in short order.
So. The 911 phone will still be answered, and your ambulance/fire brigade/armed posse is still within easy grasp.
While the redundancy is built into the system to allow the call to go somewhere, it may not be a place that can handle the call the best. We try to build the systems to account for these failovers, but even then, calltakers/dispatchers will start working on a sort of "muscle memory" when things get bad and busy. Just because the 911 phone is answered, doesn't mean that the ambulance/fire brigade/armed posse is within easy grasp. It's not as simple as adding capacity if the extra capacity is just as unaware
Re: (Score:2)
No system is without faults.
But in this context (the Seattle non-fiasco), it doesn't seem to be a big deal. Things were/are fine.
That said: Everyone wants and expects their 911 services to be absolutely bullet-proof, but nobody paying for it gives a fuck about the funding for that. 911 (in these parts) is funded in ways that are more straight-forward than gasoline taxes, and relatively easily understood. And these taxes are currently on the chopping block, for the benefit of no-one and the detriment of
Good plan (Score:1)
Nice, so they're running their mission critical operations on reserve systems. Hope nothing too important happens while they're getting bombed by a /. post.
Use the remote site (Score:3)
Why don't they just fail over the critical life and fire safety systems to the backup datacenter, and keep normal services up at the primary datacenter while they do the work? They do have a second site, right? Surely no one would host a system deemed "critical" and "life safety" at a single site?
Re: (Score:2)
Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much money we can save". Such is the way of things.
If you personally are worried about 911 services being out then go write down the various 7 (or 10-digit if your exchange requires it) numbers for your local emergency services. 911 is not an exclusive to reach the
Re: (Score:3)
Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much money we can save". Such is the way of things.
If you personally are worried about 911 services being out then go write down the various 7 (or 10-digit if your exchange requires it) numbers for your local emergency services. 911 is not an exclusive to reach them, just the easiest. Whatever happened to the days of the list of those various numbers on the fridge? I'm not even that old and I remember my parents having the list posted just in case.
I thought I was already paying for a reliable E-911 service through the 911 service fees we've all been paying on our phone bills for years.
So what you're saying is that even though we've been paying for 911 for years, we've been paying for cheap, non-redundant service, and it we expect the type of multi-site redundancy that's normally reserved for moderately successful websites, then we need to pay even more? What value are we getting from the hundreds of millions of dollars already collected?
I've called 9
Re: (Score:2)
Next you're going to tell me that the USF fees are always used precisely for what they say they're for.
Re: (Score:2)
>Because while things may have been well designed originally or planned including all the fancy redundancy, after years of no major
>issues it becomes a target of its own success: cutbacks and people saying "see, we never needed it, and look at how much >money we can save". Such is the way of things.
Part of this is also people who are bad at math. I once had a major disagreement with a business guy trying to explain that there was a significant difference between a server that had been 100% availabl
Re: (Score:2)
With 911 services the other is consolidation. In my area there used to be multiple centers with dispatchers for various agencies in diverse locations. By nature it was redundant. If one had problems the others could assist. Now there's just a single unified emergency communications center for everyone. Even if there is a backup site or plan all of the dispatchers are on duty are only at the unified center and it takes time to shift things around.
Re: (Score:2)
You just don't fail over to an off-site facility when you're short staffed and haven't thoroughly tested your off-site. Very few locations can effectively fail to an off-site location gracefully for one reason or another.
Re: (Score:2)
Presumably because then, there won't be a backup available for the critical systems. There probably is some extensive backup infrastructure available, but you never activate it unless you genuinely *absolutely have to.* If something bad happens to the active systems while you have voluntarily taken down half your 911 infrastructure, "we didn't want to take down any convenience systems," really won't cut it as an excuse. Besides, the presumed backup probably isn't seamless, doesn't work quite as well, etc
Re:2.1 million? What? (Score:4, Informative)
The datacenter is on the 26th floor of the municipal tower and the overheating bus runs up to that floor. The power company in question is municipally owned, either way it would be the city's problem.
overheating power buses / wires are a fire risk (Score:3)
overheating power buses / wires are a fire risk and that comes from them being under sized for the load.
See the towering inferno to see where that can get you.
Re: (Score:2)
and that comes from them being under sized for the load.
While you're technically right they may not be undersized for the load. Very few substations suddenly become so warm because they are sized incorrectly for the the loads they are driving under normal conditions.
The vast majority of overheating in switchgear comes form either malfunctioning equipment (overdrawing current without tripping out), maintenance problems (dust and such, though more of a problem in remote RMUs), or of faulty installations or faults developing over time causing poor connections.
Last
Story doesn't make sense (Score:2)
McGinn had quite a few facts wrong in the press conference. The equipment is working fine now and the overheating only caused a minor amount of downtime. The major issue though was the backup generator never kicked in because as it turns out, the electric starter for the diesel generator is connected to the same bus. Labor Day weekend was then choosen to fix this majorly obvious design deficiency.
It's all about how you look at costs (Score:2)