Stories
Slash Boxes
Comments

News for nerds, stuff that matters

Slashdot Log In

Log In

Create Account  |  Retrieve Password

The Backhoe, The Internet's Natural Enemy

Posted by CmdrTaco on Thu Jan 19, 2006 01:29 PM
from the brace-yourself-for-impact dept.
Juha-Matti Laurio writes "Experts say last week's Sprint outage is a reminder that with all the attention paid to computer viruses and the latest Windows security holes, the most vulnerable threads in America's critical infrastructures lie literally beneath our feet. A study issued last month by the Common Ground Alliance, or CGA -- an industry group comprised of utilities and construction companies -- calculated that there were more than 675,000 excavation accidents in 2004 in which underground cables or pipelines were damaged." I estimate that one third of those accidents occured within the 5 block radius surrounding my office.
+ -
story
This discussion has been archived. No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
 Full
 Abbreviated
 Hidden
More
Loading... please wait.
  • Cost?? (Score:4, Interesting)

    by pvt_medic (715692) on Thursday January 19 2006, @01:32PM (#14511390)
    I would love to see what all these "oops" cost. Fiber optic is not exactly cheap, and it is a little more complicated than just reconnecting the severed ends. And then take network down time etc.
    • Re:Cost?? (Score:5, Interesting)

      by Heembo (916647) on Thursday January 19 2006, @02:28PM (#14512061) Journal
      I know a guy who lives nearby who is a cut-fiber troubleshooter (and a good one). He will be chilling on the beach, his phone will ring, and he is on the next plane to (wherever). He troubleshoots the problem, extrapolates where the cut is (if they dont know already) spends the flight on the plane-phone, and get a team in action to fix the problem now. Sometime, they have no clue where the cut is and that is a big bitch when you have several hundred or thousand miles of fiber. The dude makes 400$/hr and is well known. Damn, I need a new job!
      • $50,000 for a fusion splicer that can be used in the bottom of a muddy trench (at my last job, I think our benchtop models cost $20K) or $20 for a pair of crimpers? I'd wager repairing fiber is a lot more expensive. Heck, copper hardly even matters if it's clean or not. Either way, you're getting a couple of on-call guys each spending a couple hours getting gear together, driving to the site, doing the fixing, and testing and documenting the repair. Add that to the theoretical cost of down-time that account
  • by Tackhead (54550) on Thursday January 19 2006, @01:34PM (#14511421)
    A tip, by the way, for all who go down to the sea in ships:

    Always carry a length of fiber-optic cable in your pocket. Should you be shipwrecked and find yourself stranded on a desert island, bury the cable in the sand. A few hours later, a guy driving a backhoe will be along to dig it up. Ask him to rescue you.

  • Good logic (Score:5, Insightful)

    by pvt_medic (715692) on Thursday January 19 2006, @01:35PM (#14511438)
    So the DHS wants to protect this infrastructure by making the location of such lines protected. Which of course is not going to help the situation because when you call Dig Safe they wont know whats under you. So you run the risk of severing more cables, and you run the risk of injury to the workers. I tip my hats to them.
  • by _xeno_ (155264) on Thursday January 19 2006, @01:36PM (#14511454) Homepage Journal

    That reminds me of when Qwest cut all telephone lines to my home town - including 911. It made the local news, and the police chief and fire chief were both pretty pissed about it. They had to increase police patrols since no one could just call in a crime, fire, or medical emergency.

    Fortunately nothing serious happened while 911 was out.

    Then Qwest did it again, two days later, on the same line...

    Ah, telecom monopolies.

  • by kyoko21 (198413) on Thursday January 19 2006, @01:37PM (#14511466)
    Information Technologist: 0

    Red Neck: 1

  • by JUSTONEMORELATTE (584508) on Thursday January 19 2006, @01:38PM (#14511480) Homepage
    Evi Nemeth used to tell us how to lay out a fiber ring -- separate egress from the buildings, diverse routes from location to location, etc -- and how NOT to lay out a ring.
    When CU Boulder put in their fiber ring, they ran the spans in separate conduit, which they lay in the same trench. The conduits were not at different depths, nor were they really that far apart (about 3 inches)
    They put the bright orange plastic sheet ("Hey backhoe guy! Stop digging now!") right on top of the conduit, then filled in the trench.

    Surprisingly, it got cut.

  • by Nom du Keyboard (633989) on Thursday January 19 2006, @01:38PM (#14511481)
    That's like saying that the gun kills, and not the person holding the gun. So much for another Slashdot article title.
  • by snowwrestler (896305) on Thursday January 19 2006, @01:38PM (#14511486)
    Holes! define who vee are, und vhere vee are going.
  • by macklin01 (760841) on Thursday January 19 2006, @01:42PM (#14511531) Homepage
    According to the article, in 2004, nearly half of the accidents were caused by on-site workers not checking with the proper support numbers for underground cables and/or pipelines.

    I wonder just how much those incidents would be reduced if companies were fined a stiff penalty for digging without calling these numbers. The type of astronomical fines/penalties levied against virus writers would seem very appropriate in these cases, given the type of economic damage that can be caused by telecom outages.

    I'm glad to see that a national calling center is being established (similar to 911, according to the article). Now, it will be easier for workers to call. But I still think we need the other half: better (financial) incentive to make those calls in the first place. -- Paul
    • by realmolo (574068) on Thursday January 19 2006, @02:09PM (#14511827)
      At least here in Iowa, they ARE fined. It's against the law to not call for a "locate" if you are going to do ANY digging. The law applies to both businesses and individuals.

      Of course, no one ever calls. I work for a local utilities company, and lines get cut ALL the time by contruction crews. Because they almost NEVER call for a locate. It's insane.

      The few times they do call for a locate, we go out and mark the lines, and they cut them ANYWAY. Unbelievable.

    • According to the article, in 2004, nearly half of the accidents were caused by on-site workers not checking with the proper support numbers for underground cables and/or pipelines.

      And a large part of the other half, like this particular incident, is probably because the digger got an erroneous answer from the support number. A contractor for Verizon buried fiber optic cable (for FIOS [verizon.com]) in my neighborhood late last year. Prior to their arrival, the cable, electrical, and natural gas utilities marked the

  • Hah! (Score:5, Funny)

    by Gaewyn L Knight (16566) <vaewyn@NOSPAm.wwwrogue.com> on Thursday January 19 2006, @01:45PM (#14511562) Homepage Journal
    Every time I see a backhoe go by I go into an Elmer Fudd voice and say, "Be wery wery qwiet... I'm hunting fiber"

    For some reason the Servers and Networks guys don't think it is funny.
  • by ispland (460855) on Thursday January 19 2006, @01:47PM (#14511579) Homepage
    Those of use from the telecom world recoginze
    this as "backhoe fade" and ARPA has conducted
    considable research on the effect of fiber
    optic cable to attrace backhoes in the wild...

    ARPA Science Research Funding News Today......

                    ARPA to Fund Network Reliability Research

    Washington, DC -

    The Advanced Research Projects Agency of the DoD announced today they are
    funding a three-year effort to improve the field reliability of
    fiber-optic communications networks. The program is aimed at reducing
    network outages from damage to buried fiber optic cables caused by
    construction machinery. Many telecommunications outages are caused each
    year when machines called "backhoes" dig-up underground fibers, cutting
    them and causing massive service disruptions.

    This phenomenon is commonly referred to as "backhoe fade" and
    the uncanny ability of the construction backhoe to locate buried
    cables will be the focus of this effort.

    Dr. Zweiback Gimfizel of the Marginalia Institute of Technoplasty
    has been designated Principle Investigator on the project and
    held a news conference today and described the proposed line of
    research.

            "We are taking a page from the biologists who discovered
            the magnetic organ in the brains of homing pigeon. This
            organ senses the earth's magnetic field and allows the
            pigeon to track its location.

            "In like manner, our research will focus
            on identifying the specialized organ structure within
            the backhoe that can somehow sense the location of glass
            fibers."

            "The hope is that if this fiber-seeking mechanism can be
            identified, measures can be developed to disguise
            telecommunications cables, thereby creating "stealth"
            fiber bundles which will not attract the attention of
            the rampaging backhoes."

    In another unrelated statement today, ARPA announced the creation of the
    Remote Autonomous Rodent Program which will work on developing specialized
    weapons systems for attacking the underground communications systems of
    adversaries. In recent theater actions, modern fiber-optic communications
    systems have proven quite resilient to traditional attacks and require
    new techniques to disable them.

    Dr. Gimback Zweifizel of Hardly Yardwell University was designated
    Principle Investigator. In a prepared statement, Dr. Zweifizel noted that
    this work program was funded for three years and was to produce a field
    demonstration of a working system. Other details of the project are
    classified.
  • by iibbmm (723967) on Thursday January 19 2006, @01:48PM (#14511600)
    In California we are required to notify USA DigAlert before all excavation. DigAlert then notifies all agencies with pipe in the area. Most of the time, they come out and mark, the other times, nobody does.

    When nobody comes out an marks, and their line gets hit, it's on them. If it's marked and we hit it, it's on us. Accidents happen. Digging around mismarked and unmarked utilities in a big hole in the ground isn't easy.

    Personally I'm more worried about my guys hitting a pressurized gas line than someones precious telco wire. Wire gets fixed in a matter of hours.
  • by AndroidCat (229562) on Thursday January 19 2006, @01:50PM (#14511616) Homepage
    The Common Yellow Backhoe [ellingsondrainage.com]
    The Common Yellow Backhoe [68.232.111.189] attempting to hide from view.
    The Hammer Backhoe [onsiterents.com] evolved to fit particular niches.
  • by djh101010 (656795) * on Thursday January 19 2006, @01:54PM (#14511665) Homepage Journal
    I've actually created my own internet outages with my (now sold) backhoe, twice. Neither of them the obvious. I had a 802.11b feed from a neighbor's house, 1.1 miles away (my hill to his tower). Worked great, almost always. Finally figured out that if I parked my backhoe at --> that end of the back yard, it was enough into the fresnel zone of the wireless link that things got wodgy.

    Next time I created backhoe fade, was again in an unexpected way. I'd been trenching along the driveway, after dutifully and carefully marking the underground phone line to the house (that the brain-trust from the phone company decided to run next to my driveway, despite instructions not to). I carefully and successfully avoided the cable, no worries there. Then, when reaching juuuuuuust a bit too far over, I got the backhoe stuck in the muddy ditch along the road. Apparently, in the effort to get un-stuck, I pressed down on the cable, which then stretched over a rock in the trench and broke.

    The phone company (eventually) got out there and tried to say I dug it up. I showed 'em exactly what happened - yes, I'd been digging. Yes, the wire was marked. Yes, none of my digging was along the wire's path (all true). The cable had clear marks of a pull over a rock, not a cut from a hoe. Shear vs. tension, obvious from inspection.

    Phone company guy didn't want any part of explainations until I (a) bet him that I could dig right (made an X) here and find a big rock with a sharp edge "that you people left in the trench of this improperly installed wire", and (b) pointed out that if he's gonna dig the trench, he's standing in poison ivy while doing so, and I could just go get the backhoe and make it easier for all involved.

    He called his boss, explained the high points of the situation (including the poison ivy, which inexplicably a guy in his job didn't recognize without help), and they fixed the cable no charge. But, I bet I'm one of those statistcs in the article.
  • T-Shirt (Score:3, Funny)

    by MBCook (132727) <foobarsoft@foobarsoft.com> on Thursday January 19 2006, @01:56PM (#14511684) Homepage
    Is it just me, or does this need to be on a T-Shirt? Put the picture of a backhoe in a yellow diamond caution sign with the phrase below it.

    This is defiantly true though. Living in a fairly recent subdivision, back when the construction was closer to my house this would happen all the time. The phone. The cable. The internet. Even the power once.

    I think it's clear what we need to do: go kill all the backhoes.

    Save the internet!

  • by whyde (123448) on Thursday January 19 2006, @02:06PM (#14511796)
    ...That way, when he gets lost, he drops the fiber on the ground, waits ten minutes, then asks the operator of the backhoe cutting the fiber for directions.
  • by Ranger (1783) on Thursday January 19 2006, @02:12PM (#14511868) Homepage
    Backhoes don't care. Why? I'll relate a similar wire cut story. I was called out to our local airport to fix a problem with one of the airlines ground to air radios. This lets ground crews communicate with the flight crew. It is separate from air traffic control. In one room was the radio. In another room several doors down was a monitor speaker that could hear the conversation. I determined there was nothing wrong with the monitor speaker and nothing wrong with the audio and most likely a broken wire. The room separating the radio and speaker was quite large. The wire was run through the ceiling. I had to lift up many a ceiling tile to trace the wire and find the break.

    I found the break. The wire had been cut and tied off. There was barely enough wire to splice the two back together. Once repaired the monitor speaker worked again. I was told later by the airline employees, airport facility workers had redone the ceiling in that one room. To me it appeared the workers found the wire in the way of their job, didn't know or didn't care what it was hooked to and simply cut it and tied it off out of the way.

    Backhoe operators probably have the same mentality. They want to get their work done. If they cut a cable, it doesn't affect them. They are just doing their job. To solve this problem I would recommend burying fiber next to gas lines. The fiber should be coated with a material that bursts into flame 30 seconds after it exposed to air or cut. Not only will the backhoe operator cut the cable he'll break the gas line as well. The 30 seconds delay is to build up enough gas for a nice explosion. Sure it'll be a mess, but that's one backhoe operator who won't cut any more fiber.
  • by smooth wombat (796938) on Thursday January 19 2006, @02:31PM (#14512106) Homepage Journal
    Said in that Australian accent we've come to know and love:

    Today I'm going to show you one of the wonders of nature. If you look down in that hole there you can long fibrous tendrils. Those are fiber optic cables and they snake their way through the ground all over the world.

    Crikey, it's nice to see them. Usually they stay underground so this is really special. Just look at the size of the hole they make as they burrow through the earth.

    Oh look! I didn't expect this. The only known enemy to these folks is coming over to investigate. The backhoe. Look at those nasty pointed teeth. I wouldn't want to get caught by them I'll tell you.

    I'll just walk away so I don't disturb him. This could get real exciting any moment.

    *growl* *snort* dig dig dig dig

    Look at that! This is a real treat. The backhoe is digging up the fiber optic! Look at the way those teeth just dig into the soil and expose those poor buggers. Oh wow, just look at it as it tears those fibers to ribbons.

    I know it may seem cruel to stand by and do nothing but this is part of nature. Someone has to eat and someone has to be eaten.

    But don't worry mate, those fiber optics grow back real quick. In fact, they grow so quickly there will never be a shortage of them no matter how many get eaten by the backhoe.

  • by The Fun Guy (21791) on Thursday January 19 2006, @02:37PM (#14512162) Homepage Journal
    Neal Stephenson has a hilarious comment on this in "Mother Earth Mother Board" [wired.com], in his description of a big project to lay fiber optic cable in the Pacific Rim.

    Q: Why bother running two widely separated routes [for cable from Point A to Point B] over theMalay Peninsula?

    A: Because Thailand, like everywhere else in the world, is full of idiots with backhoes.

    Q: Isn't that a pain in the ass?

    A: You have no idea.
  • by Gandalf_the_Beardy (894476) on Thursday January 19 2006, @03:17PM (#14512585)
    Over here in Blighty I've been digging with a little mini backhoe for foundations for a greenhouse and found pipe - rusty iron about two feet down which for an 8 inch main is actually shallow. Put a crack in it, but no leakage fortunately. So we called round and Transco (gas infrastructure) reckoned it was theirs and sent a man out.

    A short period of digging later and he came out the hole at some speed looking very pale. The said "pipe" had fins on one end and was delivered 60 years ago by some Germans who failed to stop and advise my grandparents of the delivery.....
    • ah here a great human error story. House gets blow up because they connected the wrong gas line. Home in Lexington explodes [boston.com]. If a company that has the maps of all of its own gas lines can do this. Think of the possibilities when DHS tries to classify the listing of all the fiber optic in the US
      • My father-in-law is a dispatcher for Keyspan, the company responsible for that blunder. Apparently, some of the older gas lines are not clearly marked as to whether they are high pressure or low pressure lines (which is what caused this balls-up). The tech hooked a high-pressure feed line up to a low-pressure residential feed line. This caused gas leaks in many homes along this line as the high-pressure gas popped open any weak fittings or connections inside the house. MAJOR screw-up.
    • A possible solution would be to embed RFID tags every 3 feet or so inside the conduit, allowing for easier location. Code embedded in tag would give owner, pipe or line type, and depth.
      • And then the terrorists would just go around with an RFID reader, wait to get a lot of hits, and start digging.

        /Somebody had to bring it up

    • by omeomi (675045) on Thursday January 19 2006, @01:42PM (#14511528) Homepage
      At our company, we are trying to figure out how to use cable over telephone pole (business class cable) as a backup in case we get "dug up", which would provide a new level of reliability, but I am sure somewhere out there there is still some unavoidable single point of failure that no amount of money can overcome.

      A backhoe driver that accidentally digs up your cable, and then backs into the telephone pole?
      • by xsbellx (94649) on Thursday January 19 2006, @02:48PM (#14512268) Homepage
        You obviously have NO experience in the construction trade. For that matter I guess you have very little experience with things moving as a result of frost. Things burried several decades ago can easily move several feet

        There is NO contractor in the world that would accept the responsiblity and/or liablity of locating utility assets (gas, telecom, water, electric and so forth). Each utility provider will dispatch specially trained and equipped technicians to perform this service. The "locater" must be accurate within certain tolerances or the utility assumes the liablity associated with any distruption/repairs including contractor's equipment that was damaged.

        Speaking from experience, I have seen a 60 inch water main broken (locator was wrong), a large telecom cable (something insane like 5,000 pairs) running to a 72 story office building (excavator problem) and countless single line telecom cables (just trying to find the damn things using a shovel but electrical tape works wonders).

        The short answer is, you can have all the centralized documentation/maps whatever that you want but no contractor will ever put a shovel into the ground until the utilities come on-site and say "You can dig here but not there".

        • by gEvil (beta) (945888) on Thursday January 19 2006, @01:43PM (#14511539)
          And if you'd read TFA, you'd see that the contracter did call. They were given the go-ahead to dig.
          • Usually you tell the marking service how deep you're going to be digging. For most jobs they'll mark everything since the stuff isn't buried too deep.

            For a lot of the "middle of nowhere" fiber feeds they bury them at least 6' (2M) deep. An electrical contractor friend of mine was doing a job "just north of middle of nowhere." He'd had the major fiber carrier in the area come out and mark where the bundle was buried. And they assured him it was 6' down -- which worked since he was only digging down about
            • by SmurfButcher Bob (313810) on Thursday January 19 2006, @05:02PM (#14513504) Journal
              You're on drugs. NO utility gives depth info, and I'm not aware of any major one-call center that TAKES depth info from the caller. I know for a fact that we don't.

              Depth is useless; in Arizona, for example, snow-plows are required to "call before they plow". Why? Soil erosion.

              Here in my state, depth is likewise useless; not as much from erosion as it is from grading. Infrastructure goes in first; landscape happens last. It is QUITE common for a 48" deep line to be 24" from the surface after several years.

              And that isn't accounting for things that were discovered when trying to bury the lines in the first place; intersections with other plant means you change height at that location. Hitting Bedrock... means you change height.

              > had the major fiber carrier in the area... assured him it was 6' down

              Not likely. The moron sent to locate the cable may have mentioned the depth in passing, but I work with these "major carriers" and their locators every day, and there is no way in hell they'd say "you're fine to use your backhoe directly on top of my wire up to 5 feet 11 inches deep". Most "Major Carriers", on a long-haul line, will physically PUT A BODY on-site during the dig to enforce the protection of their cables by hand digging over it. If it's an issue, or a very high-value asset, they'll even go so far as to hand-expose it, themselves. They do not, ever, say "sure! Just dig right on top of it".

              Ever.

              What I'd suggest is that you ask your contractor friend to define what he means by "assure". And as you do this, remember that he's getting sued for being at fault... he won't do anything to deflect responsibility, at all... he certainly won't exaggerate what was said, for certain.
        • You shouldn't have to make the call in the first place. Because you'd have to call the phone, power, gas, internet, cable, and about 7 other organizations to figure out if there was anything down there. An easier way would be to have it centallized in a database. You type in where you want to dig, In GPS coordinates, and it tells you what is located underneath, if anything.
          • by krlynch (158571) on Thursday January 19 2006, @02:02PM (#14511754) Homepage
            In many states there's only one number to call, not several. Anywhere you live in MA (and a bunch of other NE states), you call "Dig Safe" at 1 888 DIG SAFE, tell them the date and location of the dig, and they make sure all the appropriate companies are contacted.
          • by chroma (33185) <chroma AT mindspring DOT com> on Thursday January 19 2006, @02:04PM (#14511765) Homepage
            Here in Georgia, USA, at least, you can make one phone call and have all underground gas, cable, phone, sewer, and electric lines located for you. For free. People come from the various services and stick little flags in the ground over the lines.

            I had to do this when I dug up part of my front yard to put in a flower bed.
          • by lowrydr310 (830514) on Thursday January 19 2006, @02:23PM (#14512007)
            Do I smell a business opportunity that utilizes a Google Maps plugin?
          • I agree, totally - we played with that concept over a decade ago using a protype I'd built. We went so far as GETTING board-approval for the concept.

            It didn't work; GPS is good for 20 feet usually, whereas excavators need to know within a bucket-width (24"). That means that the GPS error, combined with OUR error, must be less than 24". Not gonna happen, especially since we're dealing with junk that was buried over a century ago.

            The second reason it didn't work was because retards would transpose digits whi
        • People who have vulnerable shit underground pay to be notified about excavation. Then when someone excavates, they are notified. Everyone with an interest in the area shows up and spraypaints the pavement. If you would like to point out where the flaws are in this system (certainly there are flaws) then I'm all fucking ears. So to speak.

          The flaw is in the fact that all these people have to do the right thing. In this case, if some low-level Sprint employee reads the map wrong, a whole state can be without

        • by mrhartwig (61215) on Thursday January 19 2006, @02:09PM (#14511818)
          If you would like to point out where the flaws are in this system (certainly there are flaws)....

          Simple, easy, flaw (which I'm sure you've already thought of) -- human error. Like the time the construction worker started digging a hole next to my house right on top of the orange paint mark specifying the location of my phone line.

          The funniest thing was the foreman trying to fix the line, since the phone company (thank you, SBC) said they'd take a day or two to get there. He was shocked (literally) to find out that phone lines carry electricity. :-)

    • you need protection from backhoe fade, you have to do the interagency engineering for separate feeds on separate systems from separate directions. will at least triple your cost to bring it up.

      I believe it is called a Sonnet Net. Two completely independant paths that are at no time closer than 25 feet from each other, including the locations where they exit the building. Various telcos offer this.
    • by dosquatch (924618) on Thursday January 19 2006, @02:42PM (#14512223) Journal
      everybody cross-leases dark fiber to everybody else.

      At one of my previous jobs, we had popped for the whole menu of auto-whizbang-failover magic. Redundant routers, redundant switches, redundant connections from separate providers. Protected to the nuts against outages.

      Imagine our surprise when early the first spring after installing all of this, our connection went down. Both T's out. We were more than a little perplexed - the way the odds were explained to us, God himself would've had to smite most of the southeast US to make this happen.

      It turns out that it wasn't God, and there was no smiting involved. Instead, over certain stretches, provider #2 was leasing fiber from provider #1, and one of these stretches ran under the edge of a farmer's field in Georgia. Come spring, the farmer comes out with his backhoe, and... well, you know.

      For as long as I was there, we were guaranteed at least a half a day of outage somewhere around the beginning of spring. Every time, the problem was eventually reported to us as "A fiber cut in Georgia..." They never would tell us if it was the same farmer every time.