Follow Slashdot blog updates by subscribing to our blog RSS feed

 


Forgot your password?
Close
typodupeerror
×
Cloud Data Storage Power IT

Con Ed Says NYC Datacenters Should Get Power Saturday 107

Posted by timothy from the long-weekend-for-sysadmins dept.
Nerval's Lobster writes "The local utility serving most of the New York City area, Con Edison, reported that it should begin supplying utility power to midtown and lower Manhattan by Saturday evening, returning the island's data centers and citizens to some semblance of normalcy. In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power. That should come to an end, for the most part, when utility power is restored. In a possibly worrying note, Verizon warned late on Nov. 1 that its services to business customers could be impacted due to lack of fuel."
This discussion has been archived. No new comments can be posted.

Con Ed Says NYC Datacenters Should Get Power Saturday More

Con Ed Says NYC Datacenters Should Get Power Saturday

Comments Filter:

  • Hopefully the suffering people of this unfortunate city will finally get some help; they seem to going through quite a lot and it is time they got some help to get things back on track. First 9/11 and now this. Have they not suffered enough?

    • First 9/11 and now this. Have they not suffered enough?

      Obviously not, because they are going to re-build.

      By far the smartest thing to do would be to re-build somewhere else. Preferably someplace with more altitude. Unless you really don't believe in AGW, in which case, many have beachfront land to sell you.

      • Re:New York will re-build. (Score:5, Informative)

        by Animats ( 122034 ) on Saturday November 03, 2012 @02:28AM (#41862911) Homepage

        Obviously not, because they are going to re-build.

        Manhattan barely has to rebuild anything. Building codes are tough there - everything has to be brick, concrete, or steel. Building foundations go down to bedrock. Few Manhattan buildings were damaged by the hurricane. One three-story slum had the front facade collapse; the walls and floors held and no one was injured. One construction crane had its boom broken by the wind, but the safety cables held and it didn't fall. That was about it for Manhattan.

        Yes, there was about a half billion gallons of water in subway, railroad, and road tunnels. Was. The MTA has big pumps. They have pumping trains made from old subway [youtube.com] cars which they pushed up to the water with small Diesel locomotives. Half the East River tunnels are already pumped out and some lines under the river are operating. Limited subway service between Manhattan and Brooklyn should resume tomorrow.

        Power never failed for Manhattan above 34th St, and it's back on now for most of lower Manhattan. Even when flooded, underground power lines can be restored rapidly. That will speed up the remaining pumping work. With power back on, New York City's gasoline pipeline is running again, and gas stations are reopening.

        The areas that are severely damaged are single-family residential frame structures in coastal communities. Some of them are totally wiped out. People in the outer boroughs and the Jersey shore are getting cold and hungry. The first supermarket in Far Rockaway reopens at 11 AM Saturday. In Manhattan, as soon as the infrastructure came back up, the city was ready to go. Not so in the 'burbs.

        The idiots who stayed on Fire Island despite a mandatory evacuation order were finally rescued, with great difficulty. The first group of rescuers had to themselves be rescued; they were cut off when water cut all the way across the island. Now the people who built expensive frame houses facing the Atlantic Ocean only a few feet above sea level are whining for Government funding to rebuild.

        • I look forward to the inevitable sickness and decay that follows these inundations, not because I like to see people suffer, but because I can remind you of this comment when it comes up.

          By the time this happens again, though I suspect it will be a much shorter time than you think, we will probably both be too busy to slashdot.

  • Fuel logistics (Score:5, Insightful)

    by hawguy ( 1600213 ) on Friday November 02, 2012 @06:57PM (#41860631)

    In the past few days, data center managers have been forced to add fuel logistics to their list of responsibilities, as most Manhattan data centers have been subsisting on generator power.

    Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work. Few inner city datacenters have a week or more of fuel on-site - most have only days of fuel, and they count on fuel contracts from suppliers to keep them running. And the supplier may not be able to honor the contract in a disaster.

    Suburban and rural datacenters have the space (and less conflict with fire codes since the fuel is not stored in or near an office building) to keep weeks of fuel on hand. The last datacenter that I colocated in had 2 weeks of fuel on-site, and had another week of fuel in a trailer that can be trucked in from their other facility 60 miles away if the roads are passable. They had a spare generator that can be trucked in from that other facility as well. (and this facility could send fuel and a generator to that facility if needed)

    • Re:Fuel logistics (Score:5, Funny)

      by hguorbray ( 967940 ) on Friday November 02, 2012 @07:00PM (#41860667)
      And on the plus side, the IT guys who spent the week bucket brigading now have arms like Hulk....

      -I'm just sayin'

    • Well, given that folks are already drawing firearms in gas lines, http://www.cbsnews.com/8301-201_162-57544187/new-yorker-sean-bailey-accused-of-pulling-gun-in-gas-line/ [cbsnews.com] , part of the plan had better be to contract Mad Max to bring in a tanker of precious juice past The Humungous, Wez, and their pals.

      • Re: (Score:3)

        by hawguy ( 1600213 )

        Well, given that folks are already drawing firearms in gas lines, http://www.cbsnews.com/8301-201_162-57544187/new-yorker-sean-bailey-accused-of-pulling-gun-in-gas-line/ [cbsnews.com] , part of the plan had better be to contract Mad Max to bring in a tanker of precious juice past The Humungous, Wez, and their pals.

        Well the good news is that few drivers in the USA can use the diesel that's needed to feed datacenter generators in their cars, so hopefully the trucks will get through.

        • Re:Fuel logistics (Score:5, Informative)

          by Anonymous Coward on Friday November 02, 2012 @07:42PM (#41861037)

          True, but you can't legally use the fuel that's meant for the data center in your truck to get it there. We did that after Hugo hit the SC coast in Sept 1989 and got caught. The generator fuel has dye in it that will stain the fuel filter. It took nearly six months and several tens of thousands in legal fees to get our truck back. Meanwhile, the servers in our data center in Goose Creek, SC ran out of fuel and nearly put the company out of business. There were dozens of other trucks that got caught at the weight stations over the next few years that also used fuel not meant for use on the road that were also fined and/or confiscated. My father-in-law owns a towing company so they got a lot of towing and storage business from that. Don't underestimate the US government's desire to screw over the little guy and their desire to put companies out of business.

          • Re:Fuel logistics (Score:4, Insightful)

            by kriston ( 7886 ) on Friday November 02, 2012 @10:57PM (#41862231) Homepage Journal

            The government is not "screw[ing] over the little guy" as you so eloquently put it. The non-road diesel fuel is dyed specifically because that fuel's tariff does not carry road tax. Road tax pays for the roads. This is what the situation is really about. So many violators were using non-road fuel that they had to take steps.

            All licensed truck drivers implicitly understand that you do not put non-road/farm fuel into a road vehicle if you intend to keep that vehicle in revenue service. It's intuitively obvious to everyone, but it's too bad your drivers were either ignorant of the law or chose not to follow it.

            • Yeah, God forbid the rules should be relaxed during a disaster. No, the law is the law for a reason, and if anything enforcement should be stepped up after a big storm.

              • "Yeah, God forbid the rules should be relaxed during a disaster. No, the law is the law for a reason, and if anything enforcement should be stepped up during a big storm."

                FTFY.

      • "two cars enter; one car leaves!"

    • Re:Fuel logistics (Score:4)

      by girlintraining ( 1395911 ) on Friday November 02, 2012 @07:36PM (#41860987)

      Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.

      I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.

      As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is. Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions. Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"

      I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.

      • Re:Fuel logistics (Score:5, Insightful)

        by hawguy ( 1600213 ) on Friday November 02, 2012 @08:07PM (#41861195)

        Any datacenter manager that doesn't already have fuel logistics in their disaster plan is in the wrong line of work.

        I don't think any data center manager had a line item in the disaster recovery plan that included having all transportation access cut to the entire island due to flooding of the tunnel and closure of the bridges, for over a week. Everyone is having a problem getting fuel into the city; even mission-critical services like emergency services, hospitals, and telecommunications facilities.

        What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that every sizable datacenter had exactly this scenario in the DR plans. You don't run a $10M facility in a coastal city on an island without including flooding in your DR plan. This was a 100 year event, so it was definitely on everyone's horizon. Having it in the plan doesn't mean that you have a good solution - living in SF means Earthquakes are a big part of our DR plans, but we have no expectation that our facility will survive a 7.5 earthquake intact, or even that our employees will be motivated to come to work when they are worried about their own survival.

        As to your comment that "suburban and rural datacenters have the space", sure... but where's the fiber optic cable hookups and the telecommunications infrastructure located? I'll give you a hint: Not in a barn. Those data centers are located downtown because that's where everything else is.

        Just follow the train lines to find out where the major telecommunication lines are -- I have access to more carriers down on the Peninsula outside of San Francisco than I do downtown. You may be surprised at how much bandwidth runs through Colorado and even Missouri.

        Not only that, but most of the data centers on the island are there because that's where Wall St. is, and milliseconds matter when it comes to high volume trading and financial transactions.

        Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.

        Commercial real estate is at a premium in New York. Actually, all real estate is, leading to the old joke that when a New Yorker hears someone has died, the first question they ask is, "Is their apartment for rent?"

        I'm not sure what your point is? Datacenters have to be built in the city because that's where the carriers are, but commercial real estate is expensive so don't build your datacenter in the city?

        I think it's more likely to assume you've made an error in your reasoning, writing opinions from the comfort of an armchair, than people being paid over six figures who's job depends on balancing everything out exactly and to the nearest penny an hour.

        You're obviously not in NYC if you think "over six figures" means someone is highly paid. Part of my job is planning our IT DR strategy. Fortunately, our Facilities dept is in charge of the generator so I don't need to worry about fuel contracts or keeping it running, but I do need to make sure our data is safe no matter what happens to the building and that we can continue to operate as a business. The only "disaster" that we plan on riding out on the building generator is a localized power outage when we know we'll be able to get fuel once our 3 - 5 days of fuel runs out. If there's a widespread power outage or disaster, our plan is to transition to the remote site since we know we may not be able to keep the generator fueled.

        • What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that...

          There were 47 storms to hit New York last century [wikipedia.org]. Of those, only the storms of 1936, 1944, 1954, 1971, 1976, and 1996. Of those, the only storm which caused the New York subway system to flood and caused significant damage to Manhattan was in 1971. So a couple times a century is not a likely disaster -- most of these storms do not result in the level of devastation seen here.

          Just follow the train lines to find out where the major telecommunication lines are -- I have access to more carriers down on the Peninsula outside of San Francisco than I do downtown. You may be surprised at how much bandwidth runs through Colorado and even Missouri.

          I'm well aware that bandwidth runs through rural areas, but planting a data center in a place conveniently located on top of one that

          • What kind of disaster plan is it that doesn't account for a likely disaster? I can guarantee that...

            There were 47 storms to hit New York last century [wikipedia.org]. Of those, only the storms of 1936, 1944, 1954, 1971, 1976, and 1996. Of those, the only storm which caused the New York subway system to flood and caused significant damage to Manhattan was in 1971. So a couple times a century is not a likely disaster -- most of these storms do not result in the level of devastation seen here.

            a couple of times a century is definitely a likely disaster. Would you not plan for an event with a 2% chance of occuring every year?

            • a couple of times a century is definitely a likely disaster. Would you not plan for an event with a 2% chance of occuring every year?

              Not if it cost me $86 million a year, and my total operating budget was $200 million.

              • Re: (Score:2)

                by hawguy ( 1600213 )

                a couple of times a century is definitely a likely disaster. Would you not plan for an event with a 2% chance of occuring every year?

                Not if it cost me $86 million a year, and my total operating budget was $200 million.

                Where does $86M come from? The $83M number you gave in an earlier thread was based on faulty math and the true number is closer to $400K:

                http://slashdot.org/comments.pl?sid=3224821&cid=41849463 [slashdot.org]

                But since billion dollar fiber links are being laid to cut latency by milliseconds primarily for algorithmic trading, if you really need a datacenter in NYC, $83M/year could be worth it.

                • The $83 million a year would be the cost to rent 1 floor of a 1 square block sized skyscraper. One assumes the electricity would be the other major operating cost, so I fudged an extra $3 mil into the equation. But even $400k against an operating budget of $200mil represents a .2% increase in operating cost to protect against something that has a 1 in 50 chance of occuring. The profit margin of the 25 largest businesses in America is about 8.3%. It's likely our margins would be lower, but even at 8.3%, that

          • Re: (Score:2)

            by hawguy ( 1600213 )

            I'm well aware that bandwidth runs through rural areas, but planting a data center in a place conveniently located on top of one that also bisects a town large enough to host said data center is somewhat rare. When Google announced it needed a data center, only a dozen small towns in the whole country met the requirements. So it's not as common as you think.

            Sounds like there are at least a dozen.

            Sure, latency is a reason to be close to NYC, but I don't think any of the exchanges even have datacenters in the city anymore,they are all across the river. I know NASDAQ has a backup facility in Virginia.

            I seem to recall there were two of them located in the Twin Towers. You know, until the day they were exploded. So don't "think" me anything -- either know, or keep quiet.

            So you're spouting 10 year old information for a datacenter that you already know doesn't exist and you're calling me uninformed?

            http://www.tradersmagazine.com/news/nyse-levitt-wrong-about-backup-readiness-110474-1.html [tradersmagazine.com]
            No U.S. exchanges actually operate in lower Manhattan any longer, even though the New York Stock Exchange maintains a trading floor at 11 Wall Street.

            The data centers that house their operations are all across the Hudson River, in Carteret (Nasdaq), Mah

      • If your disaster recovery plan for a business based on an island doesn't consider that all transportation may be cut, you're not doing a very good job. Otherwise, you're right.

      • Re: (Score:2)

        by kriston ( 7886 )

        That old joke about asking for a dead person's apartment was about the absurd situation called rent control. It wasn't about real estate being at a premium.

        See http://en.wikipedia.org/wiki/Rent_control [wikipedia.org]

  • Great News (Score:3)

    by ohnocitizen ( 1951674 ) on Friday November 02, 2012 @07:01PM (#41860683)
    Power to mid and lower Manhattan means a lot of people who have been without power will finally have it (and all the modern conveniences like refrigeration we rely so heavily on). It also means subway service between Manhattan and Brooklyn will come back sooner, which will be huge.

  • And in the brilliance of the building engineers, the generator is in the basement.

    Which is now filled with 13 feet of water.

    It's going to be fun cleaning up.

    • Re: (Score:2)

      by kriston ( 7886 )

      Even Sirius Satellite Radio decommissioned their midtown satellite uplink facility the moment they acquired the Vernon, NJ uplink facility in the northern exurbs.

  • Dice, get better authors (Score:4)

    by girlintraining ( 1395911 ) on Friday November 02, 2012 @07:42PM (#41861035)

    You guys recently bought slashdot, and let me say, the first few "sponsored links" have been a real disappointment. TFA has a picture that's a screenshot of the ConEd website, and poorly cropped. The information is almost 10 hours out of date at time of posting, and most of the article consists of direct quotes from articles previously submitted to slashdot! Where's the originality? Where's the reporting on why this matters? Journalism includes an analysis of the facts, not just a compilation of them.

  • Laugh (Score:1)

    by koan ( 80826 )

    And that, in a nut shell is why, you don't really want to use "The Cloud".
    Keep your data within reach.

    • Re: (Score:2)

      by hawguy ( 1600213 )

      And that, in a nut shell is why, you don't really want to use "The Cloud".
      Keep your data within reach.

      if you live in NY and keep your data in reach, it's likely that you would have been affected by this storm.

      If you kept it at 2 different cloud providers (or 2 physically separated regions of the same cloud provider), you would have been fine.

      The cloud is as safe as you want it to be, but don't assume that storing data in a single cloud provider's facility is any safer than storing it at your own facility unless you're paying to have your data replicated somewhere else.

    • Actually, this why you want to use the cloud and it's redundancy. If you are using a cloud without redundancy, you doing it wrong.

  • While shopping around for data center resources I was always amused by the data centers being offered on the island of Manhattan. Given the speed of light and the vast abundance of local meet-me rooms and extreme bandwidth on and off the island itself, coupled with the massive off-site capabilities established shortly after 9/11, why would I ever want to tolerate the risk and excessively high cost of a data center in Manhattan?

    Especially after 9/11, the idea just seemed silly. But then I listen to DI.FM a

    • Re: (Score:2)

      by Jawnn ( 445279 )
      Oh to have mod points for this sage observation. I know, I know. My office is about 22 miles from data center where our stuff lives, and it's a PITA when we have to do hands on, so I can sympathize with the desire to have the hardware close by, but at some point, you have to look at the reality. A dense urban environment is tough to serve; with electricity, fuel, food, whatever, after a disaster like this. Placing critical resources in the middle of that is tempting fate. Sure, they got away with if for 50

Slashdot Top Deals

We are each entitled to our own opinion, but no one is entitled to his own facts. -- Patrick Moynihan

Close