Forgot your password?
typodupeerror
Security The Internet United States

A Cyber-Attack On an American City 461

Posted by timothy
from the if-by-one-day-you-mean-already dept.
Bruce Perens writes "Just after midnight on Thursday, April 9, unidentified attackers climbed down four manholes in the Northern California city of Morgan Hill and cut eight fiber cables in what appears to have been an organized attack on the electronic infrastructure of an American city. Its implications, though startling, have gone almost un-reported. So I decided to change that."
This discussion has been archived. No new comments can be posted.

A Cyber-Attack On an American City

Comments Filter:
  • by explosivejared (1186049) * <hagan...jared@@@gmail...com> on Wednesday April 22, 2009 @05:04PM (#27679147)
    We should also consider whether it might be necessary to harden some of the local infrastructure of our communities. The old Bell System used to arrange cables in a ring around a city, so that a cut in any one location could be routed around. It's not clear how much modern telephone companies have continued that practice. It might not have helped in Morgan Hill, as the attackers apparently even disabled an unused cable that could have been used to recover from the broken connections.

    Always assume the enemy knows the system. Hardening wouldn't hurt, but redundancy is the most important thing. Hardening a system tends to make it that much more vulnerable to a single insider. Redundancy mitigates this effect. Having such a small group be able cause so much disruption from such a relatively simple act makes it obvious that the city placed way too much on a single point of failure remaining in tact. Have redundant fiber. Have auxiliary wireless setups. Maintain a base of ham volunteers. Multiply your points of failure.

    Personally, I think this sort of lax infrastructure security has become endemic. The 'war on terror' rhetoric we were fed for so long has us looking for the next suicide jet-liner attack or what have you, completely distorting any real conception the public had of real-world modern security risks.
    • by Red Flayer (890720) on Wednesday April 22, 2009 @05:14PM (#27679301) Journal

      Multiply your points of failure.

      I'm not sure that means what you think it means :)

      Reducing single points of failure is what is needed, which is not the same thing as multiplying the places it is possible to have failure.

      But all the methods you describe have merit, but they also have a huge drawback -- cost. It's hard to get private entities to absorb the cost of redundant fiber, etc, since they will see very little gain from them.

      So is the answer to nationalize our fiber infrastructure? Is that the only way we can make our systems secure?

      • by Xeth (614132) on Wednesday April 22, 2009 @05:57PM (#27679963) Journal
        As with any infrastructure that has national security implications, the answer is yes. Where is the profit incentive to make you triple your costs (at least) to safeguard against an incredibly rare occurrence?
        • by Sun.Jedi (1280674) on Wednesday April 22, 2009 @07:39PM (#27681035) Journal

          As with any infrastructure that has national security implications

          There should not be national security implications, because there shouldn't be anything on the internet or attached to it that could threaten national security.

          • I want a pony (Score:3, Interesting)

            Everyone should get ice cream.

            But seriously, why should there be nothing on the internet or attached to it that could threaten national security?

            The obvious answer to that is because it increases the risk.
            Consider that any radio communication between soldiers in war can potentially be a risk because there is the possibility of interception. Does that mean that radio should not be used? no!

            The more subtle answer to that is that the internet isn't yet perfectly secure or maybe 'secure enough'. The answer to t

          • by Xeth (614132) on Wednesday April 22, 2009 @09:28PM (#27681957) Journal
            You seem to be thinking solely in terms of classified information. That part is easy. The problem is that civilian telecommunications links have become the backbone of our economy. And I don't just mean that in a capital growth sense, I mean that they form the core of the financial transactions that keep day-to-day operations running. Losing those links has the capability of causing as much harm to the U.S. as losing a power plant or piece of military hardware.
          • Re: (Score:3, Insightful)

            by jra (5600)

            Major impact on commerce in a major American city is in fact an issue of national security, and anyone who doesn't think so either has his head in the sand, or doesn't read enough Tom Clancy.

      • by MWoody (222806) on Wednesday April 22, 2009 @06:14PM (#27680141)
        And on a related note, this is why I always insert multiple </b>'s into my text.
      • by bdenton42 (1313735) on Wednesday April 22, 2009 @06:39PM (#27680411)

        Reducing single points of failure is what is needed

        The cost of doing this is enormous, which is why it will never happen 100%. The scale of this outage is no where near what we had in the Chicago area when the Oak Brook central office caught on fire http://catless.ncl.ac.uk/Risks/6.82.html#subj2 [ncl.ac.uk], and that was 20 years ago. I don't think any one system is any more fault tolerant now than it was 20 years ago, but there are now multiple providers which can mitigate it significantly as long as they don't all route through the same cables as was the case here to a large degree.

        In the end any telecom system is vulnerable in localized areas... the trick is to make sure it cannot all be disabled (although software has managed to do so to great effect in the past http://www.soft.com/AppNotes/attcrash.html [soft.com]

        ...

      • by billstewart (78916) on Wednesday April 22, 2009 @07:45PM (#27681093) Journal

        It's not hard to get private entities to build redundant systems as long as they get paid for it - they're trying to sell reliable service to customers, and many kinds of customers need redundancy, and it's very hard to provide even regular reliability without it. If they had had better geographical diversity down there, then the vandals would have had to cut two different manholes in south county to do the job instead of cutting one down there and one up in the location they vandalized. Post-2001, it _is_ harder for businesses to get information on what redundancy is available, because while they all are much more aware that they need it, the governments have pushed the never-tell-anybody-real-locations paranoia - and realistically, while everybody can tell that the large building downtown with no windows and a faded bell logo on the wall is a telco office, the only way they can tell where fibers are is to look for the "Don't Dig Here - Fiber" signs which don't tell you which ones are critical.

        What's hard to get is Right of Way, and governments can sometimes help that but often interfere - highway departments can be really difficult to deal with, compared to railroads which are usually much more helpful because they're in business and you're paying them. It's especially a problem in the area south of San Jose, because the government regulators constrain ex-monopoly-telcos to operating in LATA boundaries, and they're near several LATA boundaries down there (because it used to be mostly empty farmland, and a lot of it has hills that aren't stable enough to put significant housing on, so most of the area is either reservoir watersheds or cattle ranches on one side of the freeway.) It used to be that the only industry down there was one railroad company, some farmers, and biker bars, and it was 30-40 miles from Watsonville up to the San Jose POP, a frequently-flooding river between them and Santa Cruz, and a LATA boundary between them and Monterey. Even so, I found it surprising that one well-placed cable cut was enough - usually there's one direct connection available and if a business customer needs redundancy, you can find them a second connection but it'll cost a lot more because it has to go a lot longer.

        But even in northern Silicon Valley and the peninsula, there are a number of areas that don't have as much redundancy as they'd like because the locations where telcos can cross freeways are limited. From a nationwide carrier perspective, things are better - while there are some constraints, like a limited number of railroads and highways crossing the Rockies, and a few major cities that have limited numbers of bridges and tunnels, so cable cuts out west will cost you a bunch of extra milliseconds, but the carriers do have alternate routes, and the growth of Microsoft and the Phoenix-area financial and high-tech data centers has meant that everybody's got extra capacity on the northern and southern routes as well as I-80.

        The one other source of right-of-way I'm familiar with was a gas pipeline company that ran lots of fiber along their routes. They had a certain advantage over the rest of the industry, because while Bubba the Backhoe Driver might ignore a "telco fiber - don't dig here" sign, a "Gas Pipeline! Explosive! Flammable! Don't Dig Here or You'll Blow Up and Die" sign generally got its point across better.

        Disclaimer: This is entirely my personal opinion, not that of any current or past employer.

    • by Bruce Perens (3872) * <bruce@perens.com> on Wednesday April 22, 2009 @05:18PM (#27679377) Homepage Journal

      Redundancy of means, not just points. That means not just relying on the wired communications infrastructure or things that depend on it like cellular, for your emergency services.

      Redundancy isn't always economically efficient, but we have to do it anyway, and what is worse we have to keep testing it so that it keeps working. This is hard to do if you are a private company with your stockholders baying at your feet for more efficiency.

      • Re: (Score:3, Insightful)

        by Penguinshit (591885)
        I live in one of the areas affected and I will guarantee you that disruption of 911 service means more to me than any of you. Therefore I refuse to use VoIP at home (for a variety of reasons), my cordless phone base has it's own UPS, and I have corded phones stationed throughout the house. I have a generator for when the power goes out longer than the batteries on my life support equipment.

        When I was still working and installed VoIP at my locations across the country each one remained capable of basic inde
    • by couchslug (175151) on Wednesday April 22, 2009 @05:28PM (#27679551)

      "Personally, I think this sort of lax infrastructure security has become endemic."

      That's why the incident under discussion is a good thing in the way that cracker threats and viruses are good.

      Without attacks there is little incentive to build robust systems.

    • by JavaManJim (946878) on Wednesday April 22, 2009 @06:33PM (#27680349)

      Thirty years ago Arco Oil and Gas had full data center backup. Where is this thought today in our attention deficit management world?

      ARCO did NOT depend on their local Plano TX data center. ARCO had a building prepared in Independence KS on top of pipelines that was an empty data center. They had a contract with IBM to get the next big iron off the production lines. That combined with their backup tapes means quick switch over.

      ARCO also never allowed all top executives to travel on the same jet. They flew TWO jets with passengers selected for functional redundancy. Two jets to the same location by the way.

      I like the idea, if possible, of local redundancy. Like hospitals have generators.

      I would appreciate examples of backup and redundancy today. These quiet things are often unnoticed.

      Cheers,
      Jim

      • Re: (Score:3, Interesting)

        by benjamindees (441808)

        hospitals have generators.

        In addition, resources that should not have failed, like the local hospital's internal computer network, proved to be dependent on external resources, leaving the hospital with a "paper system" for the day.

        Hospitals have generators, true. But I know of one hospital that keeps all of it's patient records via remote Windows terminal sessions to a datacenter in the next state.

        Not a small hospital either. A huge one. And it sounds like that is the norm.

        Windows terminal sessions. Not a remote database for redundancy. Not something that can be cached. A hospital, with complete dependence on a single real-time data link across hundreds of miles. Let that sink in.

    • by mcrbids (148650) on Wednesday April 22, 2009 @07:02PM (#27680683) Journal

      Sure, you can do things like reducing single-points-of-failure, beefing up security, but you can do this only to a point. At some point, you realize that society is, by nature, cooperative, and if you remove that basic assumption of cooperation, society will fail.

      There aren't any exceptions to this. There are just too many possible things that can be destroyed by people who desire a society or civilization to perish.

      You can salt fields. The Romans did this thousands of years ago, and the areas they ravaged are, to this day, incapable of meaningful agriculture.

      You can poison drinking water. LSD is pretty easy to make cheaply, and a single pound of it thrown into a public water system would cause mass insanity.

      This list is infinite: You can destroy power lines, you can cut fiber cables, you can make a bomb out of fertilizer and destroy a building or the Golden Gate Bridge or any of a quintillion other things that are both easily done and highly destructive.

      A society is secure when its population are generally happy with it continuing. When a society reaches the point where enough of its population are disenfranchised with it, it will becomes incapable of maintaining the critical infrastructure necessary for a complex civilization. Adding security measures such as multiple points of failure quickly become reasons NOT to fix why anyone would want the civilization to perish in the first place - and thus actually make the civilization LESS secure.

      And that's just the simple truth of it. So, if we want to be secure, we need to clear up the reasons why people would want our culture to fail. These include things like

      A) Not torturing people.

      B) Allowing other countries to be sovereign in their own affairs.

      C) Not being overly greedy with our wealth. Exploitation is only good for the short term - it's a long-term destabilizing force and that's bad for everyone.

      Really, I don't get it. You get people who swear by our Constitution yet somehow think that torturing is OK. Perhaps they should read the 4th and 5th ammendments? This issue is a deep, dark stain on the freedoms we are otherwise so quick to espouse.

      • Re: (Score:3, Insightful)

        by ToasterMonkey (467067)

        There aren't any exceptions to this. There are just too many possible things that can be destroyed by people who desire a society or civilization to perish.

        And that's just the simple truth of it. So, if we want to be secure, we need to clear up the reasons why people would want our culture to fail. These include things like

        A) Not torturing people.

        B) Allowing other countries to be sovereign in their own affairs.

        C) Not being overly greedy with our wealth. Exploitation is only good for the short term - it's a long-term destabilizing force and that's bad for everyone.

        This could be summed up as "Everyone should just get along with each other." Well, duh.
        I'm sure you realize that these only work if you could get everybody to do them. That is the actual problem, you cannot rely on EVERYONE AKWAYS doing what's best for EVERYONE ELSE. You've got to think about the problem differently.. how do you make people behave more predictably, in a somewhat controlled manner? Sorry, truth sucks.

      • Re: (Score:3, Interesting)

        by Khyber (864651)

        "You can salt fields. The Romans did this thousands of years ago, and the areas they ravaged are, to this day, incapable of meaningful agriculture."

        Ever hear of hydroponics? Back then they didn't know soil wasn't a requirement for plant growth. In fact the first published work on hydroponics didn't happen until the late 1600s. In this day and age, we can most certainly use that area for agriculture.

      • Re: (Score:3, Interesting)

        Allowing other countries to be sovereign in their own affairs.

        What about other countries whose affairs constitute of getting involved in your own?

      • Re: (Score:3, Interesting)

        by Hatta (162192)

        You can poison drinking water. LSD is pretty easy to make cheaply, and a single pound of it thrown into a public water system would cause mass insanity.

        Oh, how I wish this were true. First, LSD is a pretty complicated synthesis, it's not kitchen chemistry by any means. Second, the precursors are watched carefully. Third, LSD is not very stable. If there's any chlorine added to the water the LSD would never make it to the tap.

    • ... and that two cable cut sites had been found, I speculated that there were two more sites. Turns out that was the case.

      The SONET network is normally configured in a ring, or a set of interconnected rings and ring segments - a net with MOST nodes being points on a line and a few being points at a Y junction. (It's the cheapest way to insure two geographically diverse paths to every site when you have to dig things up to string your connections.) The rings are configured so that a cut link is automatica

  • Hams FTW (Score:5, Insightful)

    by ipX (197591) on Wednesday April 22, 2009 @05:05PM (#27679167)
    Ham radio operators save the day once again... 'nuff said.
  • by Sir_Lewk (967686) <sirlewk@gmail.REDHATcom minus distro> on Wednesday April 22, 2009 @05:06PM (#27679173)

    Lets not all go blaming terrorist organizations on this one.

    My money is on unionized workers facing layoffs or payroll cuts. They would best know how to hurt the system and this sort of sabotage being linked to unions is not exactly unheard of.

    • Re: (Score:3, Insightful)

      by Red Flayer (890720)

      Lets not all go blaming terrorist organizations on this one.

      Define terrorism.

      Now define terrorist organization.

      If an organized group of people orchestrated this attack in order to bring attention to some goal, wouldn't that make them a terrorist group?

      Admittedly, an attack on property is not the same as an attack on people, but yet... to me this seems textbook.

      • by Sir_Lewk (967686) <sirlewk@gmail.REDHATcom minus distro> on Wednesday April 22, 2009 @05:13PM (#27679289)

        Well, I'd certainlly concede that this could be classified as terrorism but I was refering more to the "ZOMG TALIBAN" kind of terrorists. Modern media interpretation of the word. ;)

        • by couchslug (175151) on Wednesday April 22, 2009 @05:33PM (#27679613)

          ""ZOMG TALIBAN" kind of terrorists. Modern media interpretation of the word. ;)"

          Shortly to turn into "ZOMG Wobbly Anarchist Union Menace to be cleansed with fire and legislation" if formerly-gruntled union workers are found to be the cause...

          • by Fluffeh (1273756) on Wednesday April 22, 2009 @06:23PM (#27680227)
            Hey hey hey! Hold it right there. I think there is LITTLE that strikes fear into the hearts of most nerds like being cut off from the net. So before you go waving your burning flags and crying alla akbar, maybe you should consider whistling like a 14.4k baud, and putting on that "I make Token Ring Networks" t-shirt.

            On slashdot, THAT my friend, is true terror.

            If you want to be a real tard, maybe throw in some rhetoric about head to head gaming via Com1!
      • by MightyYar (622222) on Wednesday April 22, 2009 @05:16PM (#27679347)

        wouldn't that make them a terrorist group?

        I'd presume that some amount of "terror" would need to be created for one to be considered a terrorist. But maybe I'm old-fashioned.

        • Re: (Score:3, Insightful)

          by TubeSteak (669689)

          I'd presume that some amount of "terror" would need to be created for one to be considered a terrorist. But maybe I'm old-fashioned.

          Terrorist acts need not generate terror.
          Part of the definition is that the acts can be designed to intimidate or cause fear.
          Actions that don't fit your 9/11 definition of terrorism are still considered terrorism.

          Ultimately, unless some ideological motivation is discovered, this isn't terrorism, just sabotage.

      • by Anonymous Coward on Wednesday April 22, 2009 @05:17PM (#27679357)

        If an organized group of people orchestrated this attack in order to bring attention to some goal, wouldn't that make them a terrorist group?

        No.

        What makes a terrorist group a terrorist group, is that they inflict, you know, terror .

        Cutting some cables isn't going to (and, in fact, didn't) send the general populace into a panic.

        Yes, it's an inconvenience, but unless they are trying to instill terror in the general populace, they're not terrorists.

        • by TheRaven64 (641858) on Wednesday April 22, 2009 @05:44PM (#27679813) Journal
          They might just be really, really, incompetent terrorists? I don't think we have a word mildinconvenienceists.
        • Re: (Score:3, Insightful)

          by WiseWeasel (92224)

          Note that the attack did knock out emergency response in several cities, and there were reports of increased armed robberies during the communication outage in affected areas. While defining this as terrorism is unwarranted, as it is unlikely that the attack was meant to physically harm anyone directly, it does go beyond mere vandalism when you put people's lives at risk, by preventing them from reaching the hospital, fire department or police in case of emergency.

      • Re: (Score:3, Interesting)

        by Gat0r30y (957941)
        Um, was anyone terrified that the fiber had been cut?
    • by PapayaSF (721268) on Wednesday April 22, 2009 @05:11PM (#27679257) Journal

      sabotage being linked to unions is not exactly unheard of

      Indeed, that's very possible: the contract between the Communication Workers of America and AT&T expired on April 11th [morganhilltimes.com].

      • Re: (Score:3, Insightful)

        by bjourne (1034822)

        Um.. That article explicitly refutes that rumor. Although they used weasel words to deny the suspicions in such a way that the suspicion seems more plausible to a casual reader. E.g. AT&T has not identified any suspects and does not believe Bruce Perens sabotaged their fiber like others have suggested. Although he did post a slashdot article about it two weeks after the incident...

        Blaming those damn commie unions sure is popular.

    • by Elwood P Dowd (16933) <judgmentalist@gmail.com> on Wednesday April 22, 2009 @05:18PM (#27679375) Journal

      Or just regular blackmail:

      http://yro.slashdot.org/article.pl?sid=09/04/20/1427259 [slashdot.org]

      I assumed these were both the same story at first. But the YRO story was 2005, and this one was a few weeks ago.

    • by Sponge Bath (413667) on Wednesday April 22, 2009 @05:31PM (#27679597)

      My money is on unionized workers...

      I think it was management, upset that so few people wore Hawaiian shirts on casual Friday.

    • by TheRaven64 (641858) on Wednesday April 22, 2009 @05:45PM (#27679825) Journal

      My money is on unionized workers

      Well then, ionize the workforce in future.

    • I was voting for the CWA as well...

      This happened the same day the CWA was reported as saying "contract talks with AT&T are not going well", 5 days after most of the employment contracts in California expired and AT&T tried to low-ball the healthcare benefits they'd be giving union workers in the future, and force a series of job cuts. One imagines that, in a down economy, AT&T felt they had their workers over a barrel, since job prospects are tighter these days.

      Here's a telecom industry rags ta

  • Ocean's 14? (Score:5, Funny)

    by Absolut187 (816431) on Wednesday April 22, 2009 @05:08PM (#27679197) Homepage

    Is there a casino nearby?

  • Hams (Score:5, Funny)

    by pete-classic (75983) <hutnick@gmail.com> on Wednesday April 22, 2009 @05:11PM (#27679253) Homepage Journal

    Jesus. Here come four thousand posts about how awesome hams are, and how Internet over power cables sucks, etc.

    My Dad was a ham. Yes, hams are awesome. In their nutty little useful-once-in-a-lifetime, semi-Luddite way.

    We love you, hams. We're glad you're out there. But please, seriously, shut the fuck up. On the Internet. Feel free to blather on your radios.

    -Peter

    • Re:Hams (Score:5, Funny)

      by Red Flayer (890720) on Wednesday April 22, 2009 @05:16PM (#27679345) Journal

      My Dad was a ham.

      He was delicious.

      Yes, hams are awesome.

      I concur. Especially with eggs. Or spiced and pressed into a can.

    • Re: (Score:3, Insightful)

      by idontgno (624372)

      But please, seriously, shut the fuck up. On the Internet. Feel free to blather on your radios.

      Quoth the slashbot. On the Internet.

      +1 Ironic (and not in the Morissette sense)

      Hell, there's no sane reason amateur radio shouldn't be a low-capacity emergency part of the Internet [wikipedia.org].

      You should probably work out your deep-seated issues with your daddy someplace else. Rather than on the Internet.

      • Re: (Score:3, Insightful)

        by pete-classic (75983)

        It's really weird that you interpreted what I said as daddy issues. Was the association of, "A was a B. Bs are awesome." lost on you?

        What I was trying to convey, in fact, is that I respect amateur operators. I just find the obligatory self-congratulations every time there's an article that has anything to do with ham radio annoying.

        I also find it fascinating that you dismiss me as a "slashbot", since I regularly go against the grain here. Come to think of it, you accuse me of being a slashbot when I com

    • Re: (Score:3, Insightful)

      by spire3661 (1038968)

      I wouldnt label the guardians of the ultimate backup communications system as luddites...

  • Eye Opener (Score:5, Interesting)

    by FractalParadox (1347411) on Wednesday April 22, 2009 @05:12PM (#27679273)
    Hopefully this catches enough attention to get people to evaluate their area's utilities similar to the blackout across parts of the US and Canada back in 2003. http://en.wikipedia.org/wiki/2003_North_America_blackout [wikipedia.org]
  • Cyber(?) Attack (Score:5, Insightful)

    by Duradin (1261418) on Wednesday April 22, 2009 @05:13PM (#27679281)

    So now a "cyber" attack includes the physical destruction of hardware/infrastructure without any exploitation of any programming logic?

    • Re: (Score:3, Interesting)

      by Bruce Perens (3872) *
      IMO this is what a real "cyber" attack looks like. It's interesting that someone can do it remotely, but the fact that someone - maybe just one person - was able to be 100% effective in denying communications with not much more than a hacksaw, that's a real hack :-)
    • Re: (Score:3, Informative)

      by stevied (169) *

      I guess it's kinda reasonable to use the term for an attack on the "cyber" domain (by going after its physical substrate) as well as for attacks that occur within that domain. Either way, it screws up people's access to comms.

      • by Chris Burke (6130) on Wednesday April 22, 2009 @06:20PM (#27680199) Homepage

        I guess it's kinda reasonable to use the term for an attack on the "cyber" domain (by going after its physical substrate) as well as for attacks that occur within that domain. Either way, it screws up people's access to comms.

        I don't think it's reasonable, at least not enough that we should accept it and start using "Cyber Attack" to refer to the target of the attack rather than the means. The reason basically boils down to the opposite of attack, which would be Cyber Defense, and what was mentioned earlier on /., the Pentagon Cyber Command.

        If we accept this meaning of Cyber Attack, then that means that an airplane that drops a bomb on an ISP is a "Cyber Attack", while bombing any other form of infrastructure would be a "regular attack". Logically this would also mean that an anti-aircraft gun that is placed near an ISP is a form of "Cyber Defense". Except that isn't logical, it makes no sense. Anti-aircraft defenses should not be under the purview of Cyber Command regardless of where they are located.

        No. I insist that the adjective "Cyber" before the word "Attack" should indicate the means, not the target, in the same way that Cyber Defense should mean securing computer networks, not preventing physical assaults that may or may not happen to hit internet infrastructure.

        This was nothing more than plain ol' sabotage. It's the same as them destroying a sewage line, except the impact was different. If it was a power line, that too would have cut off many forms of communication, is that a cyber attack? No. It's an attack.

    • Re:Cyber(?) Attack (Score:5, Insightful)

      by Darth_brooks (180756) * <{moc.liamg} {ta} {773reppilc}> on Wednesday April 22, 2009 @05:38PM (#27679689) Homepage

      Four words: Denial of Service Attack. You want the hospital's network cut off from the outside world for 8 hours? Congrats. Done. This was Indiana Jones vs. the Scimitar-wielding Arab. I'd like to this this whole situation has encouraged people to start thinking a little bit more outside the box when it comes to infrastructure planning and what "mission critical" really means.

      For example, my last employer took mission critical to heart. They were the regional blood bank, so that mentality was infused (tee hee. I made a pun.) into every aspect of the organization. Microwave links between our sites (and several customer sites. If need be we had the capacity to route traffic in and out through locations that were physically 5-10 miles away), generator power up the wazoo (including written contracts that put us second in line behind the hospital for diesel fuel. on top of the ample reserves we kept on site. Don't know why we weren't natural gas, though I assume that was more capacity than anything else), redundant external power connections to independent grids (which paid off handsomely one day), pneumatic tube connections to two hospitals and a couple other local sites, and a disaster preparedness plan that could have been leather bound and used for Law Office commercials if it wasn't being updated so often.

      Infrastructure and disaster planning require some in depth "disaster porn" level of thought. It's hard to excuse civic services for not being ready to handle this sort of outage. Between that job and working in SE Michigan during the '04 blackout, I've learned a lot about just how ready some places think they are vs. how ready they really are to handle a disaster, be it man made or otherwise.

  • by georgewilliamherbert (211790) on Wednesday April 22, 2009 @05:15PM (#27679325)

    Bruce, the cable cuts were in San Jose and San Carlos. The cable between San Jose and Morgan Hill was cut, but the cut location was in the city of San Jose.

    (otherwise, agree with what you said, hopefully wider audience for this will help...)

    • Re: (Score:3, Informative)

      by Bruce Perens (3872) *
      As far as I am aware, there were four locations entered, and eight cables cut. Do you have the locations for all four? If so, don't put it on Slashdot :-)
      • by Anonymous Coward on Wednesday April 22, 2009 @05:28PM (#27679545)
        I doubt many readers here could physically fit into a manhole, much less survive a climb down any sort of non-motorized ladder.
      • Re: (Score:3, Informative)

        by Bruce Perens (3872) *
        I changed the article to "cables serving the city of Morgan Hill" instead of "in" it.
  • this is interesting (Score:5, Interesting)

    by Satanboy (253169) on Wednesday April 22, 2009 @05:19PM (#27679399)

    We had a similar cyber attack here in columbus, ohio. A disgruntled employee (it is thought) shot the fiber backbone for Time Warner with a .22. I don't believe they ever caught the guy who did this. This one action disrupted the internet for hundreds of companies and thousands of users. It took around 3 days to get the internet back up for everyone.

    This was just one fiber cable, imagine if someone had purposely cut lines downtown?

    The stuff is very centralized and not well protected.

    There needs to be better protection against these sorts of actions, and there needs to be a backup plan in place in case something like this does happen.

  • Discussed on NANOG (Score:4, Informative)

    by lothos (10657) on Wednesday April 22, 2009 @05:22PM (#27679423) Homepage

    This was discussed extensively on the NANOG (North American Network Operators Group) email list.

    It appears that the outage affected multiple carriers including ATT and Alternet.

  • Not a cyber attack (Score:4, Insightful)

    by sunderland56 (621843) on Wednesday April 22, 2009 @05:28PM (#27679539)
    This sounds like a good old physical attack to me, not a cyber attack. Bashing in someone's computer with a hammer is not the same thing as a infiltrating it with a computer virus/worm/etc.
  • What Bruce Left out (Score:5, Informative)

    by Anonymous Coward on Wednesday April 22, 2009 @05:43PM (#27679797)

    The entire Santa Cruz County area was cut off from all telecommunications outside of Point to Point wireless and Satellite. (Comcast customers aside.) Sprint, AT&T, Verizon, long distance for POTS was all down. TFN's were not able to be dialed by any customers. 911, 611, 411 were not functioning. 'Point-to-point' T1's that were aggregated over DS3's in Hayward, were not functioning for area users. Many of the "redundant" network connections for companies in the Monterey Bay area were completely down. Both legs of their "best practice" 2 provider networks were crippled.

    Other than a couple islands of connectivity (namely the Shell Gas station at 41st and Capitola Rd in Capitola, my mother In Law's house, and my Uncle's business) who were lucky enough to only have Satellite service available to them, or were on Comcast, the packets stopped flowing.

    Ironically Comcast services inside the Santa Cruz county were still working. Users of Comcast voice wouldn't have noticed (except for the fact that everyone they called went straight to voicemail.)

    However, inter CO calling was working (you could
    call anyone in the Watsonville-Santa Cruz area if they had a POTS line from a POTS line. Still, corporate communications for nearly everyone in the area (Ag. Brokers, Packers, Pickers, Shippers, Bottlers, etc.) Was down. Commerce came to a halt.

    People couldn't get gas at gas stations around the area unless they had cash. Area banks wouldn't let people inside the bank unless you were making a deposit. People couldn't be players in the game of commerce without little pieces of paper. And so once again, cash was king.

    More cars sat on the side of the road that day then normal between santa cruz and watsonville. Which begs the question how does the regular joe call for help if the call boxes can't talk to a phone switch?

    • Re: (Score:3, Informative)

      by Bruce Perens (3872) *
      I drove through Morgan Hill while this was happening, and got to Carmel by 9 AM, There seemed to be something blocking 156 near 101, I had to turn around and get back on 101 and come in via 68, but I have no indication that the problem was network-related. Business in Carmel seemed to be normal.
  • cloud computing (Score:5, Informative)

    by margaret (79092) on Wednesday April 22, 2009 @05:50PM (#27679871)

    I guess this kinda puts a damper on all the cloud computing hype of late...

  • by malkavian (9512) on Wednesday April 22, 2009 @05:50PM (#27679879) Homepage

    When I first saw the way that one worked, I shook my head, and said "You're joking, right?"..
    Alas, the answer was no. And the reason that it had been designed as a centralised system (well, ok, there's a 'failover' data centre or two) is (according to the designers) that you'll never lose the main and the redundant connections at the same time.
    I seriously hope that they're paying attention to this at the moment. The severing of very few, carefully chosen fibres could quite simply deny a lot of UK hospitals access to their medical records. And if all come on board, then you could deny nearly all hospitals access to the medical records.
    This, as can be imagined, would be rather a bad thing...

  • Oh, Bruce (Score:5, Insightful)

    by fm6 (162816) on Wednesday April 22, 2009 @05:57PM (#27679967) Homepage Journal

    Bruce makes some good points, but he consistently undercuts himself "information" that is poorly sourced, poorly explained, or just plain wrong.

    The question I'm most interested in is why the "internal only" network at Dominican Hospital went down. Bruce doesn't explain this, and I can't find a reference to it elsewhere. I suspect that he just has his facts wrong — Dominican is part of Catholic Healthcare West, and I'd be very surprised if the computers at Dominican didn't rely on servers in a central CHW facility.

    That's still a dangerous vulnerability, just like Bruce says it is. But he'd be more persuasive if he checked his facts.

    And dude, everybody but you knows that that internet technology research was funded by DARPA. Some DARPA personnel are in the Army, but DARPA has never been part of the Army.

    And can we please stop repeating that idiotic myth about the Internet being designed to survive a nuclear attack? It isn't and it wasn't designed to be. The basis of the myth is that early proposals harped on the superior survival characteristic of a decentralized network versus the star topology networks of the time. Not quite the same thing.

    • Re:Oh, Bruce (Score:5, Informative)

      by Bruce Perens (3872) * <bruce@perens.com> on Wednesday April 22, 2009 @06:45PM (#27680491) Homepage Journal

      The way I said it was right. DARPA had Army and other DoD sponsorship. I said the scientists involved designed it to be militarily redundant. The fact is that the military didn't use it that way.

      Unfortunately, the main reference on the hospital is the ham coordinator, as quoted on ARRL's site:

      "While I was meeting with hospital department heads, Bob Wolbert, K6XX, had started our ARES Resource Net on the W6WLS/W6MOW linked repeaters," Pennell told the ARRL. "During the briefing, the hospital determined to implement HICS/SEMS for this emergency. There hadn't been telephones or Internet anywhere since about 2:30 AM. The hospital's phone system did work, but only within the hospital. Their internal computer local area network wasn't working either, so they were instantly on a 'paper system.'"

      The hospital isn't talking about the technical failure.

      • Re: (Score:3, Insightful)

        by fm6 (162816)

        The way I said it was right. DARPA had Army and other DoD sponsorship.

        That's nonsense. DARPA isn't "sponsored" by anybody. It's an arm of the DoD with it's own director and funding sources, completely separate from the army.

        I said the scientists involved designed it to be militarily redundant.

        Which is simply not true, though it's a popular myth.

        The absence of a central node in this kind of network has nothing to do with military requirements. The creators of the technology simply observed that existing networks all had a finite capacity for growth because of their reliance on a master system that supervised all the other systems. The master sys

  • "Manholes?" (Score:4, Informative)

    by Animats (122034) on Wednesday April 22, 2009 @06:14PM (#27680133) Homepage

    The cut location in San Carlos was reported as being at Bing St and Old County Road. That's actually alongside the rail line that runs up the SF Peninsula. There are many fibre optic cables along that right of way. It used to be a Southern Pacific Railroad line, and "Sprint" was originally Southern Pacific Communications.

    There aren't that many long haul fibre optic cable routes. Many of them run along rail lines, because the railroad owns the right of way and doesn't need anyone else's permission to run cables. Often you can run cable for miles without crossing a street, which makes installation much simpler.

  • dupe (Score:5, Informative)

    by krappie (172561) on Wednesday April 22, 2009 @06:19PM (#27680191)

    Its implications, though startling, have gone almost un-reported. So I decided to change that.

    DUUUUUUPE
    http://tech.slashdot.org/article.pl?sid=09/04/09/2044205 [slashdot.org]

  • by illumynite (239768) <illumynite AT gmail DOT com> on Wednesday April 22, 2009 @08:53PM (#27681659) Homepage

    I am the Network Administrator for an ISP (AS4307) in San Martin, CA (between Morgan Hill and Gilroy) that was directly affected by the cuts.

    We are multi-homed by two providers. BOTH providers fiber ran through those SONET rings that were cut. We were COMPLETELY isolated (internet, POTS AND cell) from 2:15am to 10:42pm. Luckily, 90% of our customers are in the Morgan Hill/Gilroy/San Martin/San Jose area, so they were fully aware of what happened.

    As a side note, the cuts were actually in San Jose. I live 3 blocks from where the cuts occured (Monterey Hwy and Cottle Rd. for those interested). And it did not just affect Morgan Hill. Some parts of South San Jose were affected, along with Morgan Hill, San Martin, Gilroy, Watsonville, Santa Cruz, and parts of Hollister.

    What was interesting was when service was restored, customers who lived out of the area who had not heard of the happenings here, called and told us they thought one of two things:

    A) We went out of business
    B) Natural disaster (Earthquake was #1 on the list, considering where we are located)

    We lost no customers over this fiasco, and are now looking at getting a provider that feeds from completely separate fiber (i.e. from the SOUTH)

    Robert Glover
    Director of I.S.
    South Valley Internet (AS4307)

  • Public Safety Nets (Score:4, Insightful)

    by kilodelta (843627) on Wednesday April 22, 2009 @09:02PM (#27681749) Homepage
    Pretty easy to take out public safety trunked systems too. All you need is a hammer and some nails.

    In my city the repeaters are on telephone poles. Just punch a hole through the feedline. If the repeater designer knew their shit they'll detect the high SWR an shut down the oscillator and amplifiers. But I can tell you, I've seen lots of gear that has no such SWR protection.

    You don't even have to go that far. A little conductive grease, or even water in a connector will also reflect lots of RF power back to the emitter.

    It is virtually impossible to protect any given communication medium. You must have several independent means of communication.
  • by The Master Control P (655590) <ejkeever@noSpam.nerdshack.com> on Thursday April 23, 2009 @12:17AM (#27683069)

    Sammy's shoulders hunched down. "They've avoided the killing disasters. They haven't had the war plagues or nuclear war. The governance is still flexible and responsive. There are just the Lord-be-damned technical problems."

    "They are technical symptoms, Sammy, of problems I'm sure the governance understands very well." And can't do a thing about. He remembered back to the cynicism of Gunnar Larson. In a way this conversation was rumbling down the same dead-end street. But Pham Nuwen had had a lifetime to think of solutions. "The flexibility of the governance is its life and its death. They've accepted optimizing pressures for centuries now. Genius and freedom and knowledge of the past have kept them safe, but finally the optimizations have taken them to the point of fragility. The megalopolis moons allowed the richest networking in Human Space but they are also a choke point...."

    But we knew -- I mean, they knew that. There were always safety margins."

    Namqem was a triumph of distributed automation. And every decade it became a little better. Every decade the flexibility of the governance responded to the pressures to optimize resource allocation, and the margins of safety shrank. The downward spiral was far more subtle than the Dawn Age pessimism of Karl Marx of Han Su, and only vaguely related to the insights of Mancur Olson. The governance did not attempt direct management. Free enterprise and individual planning were much more effective. But if you avoid all the classic traps of corruption and central planning and mad intervention, still -- "In the end there will be failures. The governance will have to take a direct hand." If you avoided all other threads, the complexity of your own successes would eventually get you.

    -- Vernor Vinge, A Deepness In The Sky.

    Always we hear that something should be privatized because private industry is more efficient. Yet never does anyone stop to ask whether efficiency is the only concern.

    Thus rather than having a reserve in transmission capacity on our electric grid, since deregulation we simply eat farther and farther into former safety margins. Rather than spend the time to set up proper local mirrors of systems, hospital networks collapse when their Internet connection breaks. It's reasoned that the time-integrated cost of safety margins exceeds the price to be paid when failures they would have prevented occurs.

    And so far, they're mostly right. We have a little more latitude for technical failures on Earth than the fictional inhabitants of Namqem. But eventually, as we hop and skip blithely into privatization of core systems, we're going to pay a horrible price for it. It's sad how many innocent lives it's going to take, but no one listened to those calling for improved maritime safety until Titanic sank either.

  • by jcam2 (248062) on Thursday April 23, 2009 @12:20AM (#27683085) Homepage

    Since I live in the area where this happened and it was reported extensively on the local news, I noticed *many* errors in TFA, such as :

    - Morgan Hill was not specifically targeted .. the cuts were in San Jose and Santa Clara. At most, Morgan Hill was collateral damage.

    - Cables were cut in four different locations, so there was no single point of failure.

    - Hosting everything at your site might help in cases like this, but is your mail really more reliable if managed by a part-time sysadmin on a single $1000 box, or at Google where they have triple-redundant everything?

    • Re: (Score:3, Informative)

      by Bruce Perens (3872) *

      San Jose and Santa Clara had other communications sources and do not seem to have had outages nearly so complete, and didn't (as far as I'm aware) need to get hams to help them run the hospital. So, I focused on Morgan Hill.

      I did mention that redundancy might not have helped this case.

      Yes, one beige box and one operator would be the wrong way to go for a hospital. I think database replication is the best way to handle this.

  • I lived this (Score:3, Informative)

    by Trip6 (1184883) on Thursday April 23, 2009 @12:53AM (#27683295)
    Heading south on 101 from San Jose Thursday morning, I was on my cell as I usually am. It cut out about 4 miles north of my Morgan Hill exit. I thought it was a dead spot. I got to work and realized we had no phones, no internet, no cell if you were a Verizon or AT&T customer. The only link we had was AM radio (KGO), who told us of the outage. We needed an ETA for restoration of service. How? We drove north until we had cell coverage, and called our respective providers. Neither had a clue. We called our spouses outside the DOS area and they said that cables were cut, but still no ETA. Finally we heard on the AM radio they expected to restore service by end of day. We ended up sending our customer service and order entry people home, and the rest of us worked the internal network or paperwork for the day. The phone came back around 4 PM, but the internet and cells never did until the next day. No 911, all the stores and restaurants were pretty much cash only. It was truly eerie. It was front page news for a couple days but has faded from view since. We think it was almost certainly the union, since the first thing the union did was vehemently deny they had anything to do with it.

Mathematicians stand on each other's shoulders. -- Gauss

Working...