The Backhoe, The Internet's Natural Enemy 382
Juha-Matti Laurio writes "Experts say last week's Sprint outage is a reminder that with all the attention paid to computer viruses and the latest Windows security holes, the most vulnerable threads in America's critical infrastructures lie literally beneath our feet. A study issued last month by the Common Ground Alliance, or CGA -- an industry group comprised of utilities and construction companies -- calculated that there were more than 675,000 excavation accidents in 2004 in which underground cables or pipelines were damaged." I estimate that one third of those accidents occured within the 5 block radius surrounding my office.
Cost?? (Score:4, Interesting)
Re:Cost?? (Score:2, Informative)
The only difference is the cost of the equipment needed. Fusion splicing is actually very easy.
Re:Cost?? (Score:3, Informative)
Re:Cost?? (Score:5, Interesting)
it's called backhoe fade in telecom (Score:2)
you need protection from backhoe fade, you have to do the interagency engineering for separate feeds on separate systems from separate directions. will at least triple your cost to bring it up.
Re:it's called backhoe fade in telecom (Score:3, Informative)
I believe it is called a Sonnet Net. Two completely independant paths that are at no time closer than 25 feet from each other, including the locations where they exit the building. Various telcos offer this.
Re:it's called backhoe fade in telecom (Score:2)
you need protection from backhoe fade, you have to do the interagency engineering for separate feeds on separate systems from separate directions. will at least triple your cost to bring it up.
I worked with this one vendor that went on and on about having dual paths that went out separate ends of the buildin
Re:it's called backhoe fade in telecom (Score:5, Funny)
At one of my previous jobs, we had popped for the whole menu of auto-whizbang-failover magic. Redundant routers, redundant switches, redundant connections from separate providers. Protected to the nuts against outages.
Imagine our surprise when early the first spring after installing all of this, our connection went down. Both T's out. We were more than a little perplexed - the way the odds were explained to us, God himself would've had to smite most of the southeast US to make this happen.
It turns out that it wasn't God, and there was no smiting involved. Instead, over certain stretches, provider #2 was leasing fiber from provider #1, and one of these stretches ran under the edge of a farmer's field in Georgia. Come spring, the farmer comes out with his backhoe, and... well, you know.
For as long as I was there, we were guaranteed at least a half a day of outage somewhere around the beginning of spring. Every time, the problem was eventually reported to us as "A fiber cut in Georgia..." They never would tell us if it was the same farmer every time.
Human error... (Score:2, Insightful)
Re:Human error... (Score:3, Interesting)
Re:Human error... (Score:3, Informative)
Re:Human error... (Score:2, Interesting)
The problem is that each service provider has their own idea of when this should be done. Some don't even know where their services run. If they do know, they only check within a few feet of the preposed test hole. So this means that once phone, sewer, and water have agreed, natural gas comes along and says to
Solution (Score:3, Interesting)
Re:Solution (Score:3, Insightful)
/Somebody had to bring it up
it happens anyway (Score:2, Interesting)
The Backhoe, the sailor's best friend. (Score:5, Funny)
Always carry a length of fiber-optic cable in your pocket. Should you be shipwrecked and find yourself stranded on a desert island, bury the cable in the sand. A few hours later, a guy driving a backhoe will be along to dig it up. Ask him to rescue you.
Nah, playing cards are easier (Score:2, Funny)
Hard Problem (Score:2, Insightful)
Re:Hard Problem (Score:5, Funny)
A backhoe driver that accidentally digs up your cable, and then backs into the telephone pole?
Good logic (Score:5, Insightful)
And on the ocean...? (Score:2)
Re:And on the ocean...? (Score:5, Funny)
Based on this article, I'd hazard it's either:
1: Backhoes falling off ships transporting them hitting cables.
2: Submarines with backhoes, no doubt performing black ops at the time.
Re:And on the ocean...? (Score:2)
Re:And on the ocean...? (Score:2, Insightful)
Re:And on the ocean...? (Score:3, Funny)
> The telco is now suing the vessel...
Darn right! Why the hell didn't the ship call Miss Utility and have lines drawn in the water before recklessly dropping an anchor into the water?
Re:And on the ocean...? (Score:3, Interesting)
Re:And on the ocean...? (Score:2)
http://www.marinetrench.com/trenching.html [marinetrench.com]
Re:And on the ocean...? (Score:2)
Apart from that the main cause certainly is anchors close to the coasts.
Ah, yes, Qwest did this to my home town... (Score:5, Interesting)
That reminds me of when Qwest cut all telephone lines to my home town - including 911. It made the local news, and the police chief and fire chief were both pretty pissed about it. They had to increase police patrols since no one could just call in a crime, fire, or medical emergency.
Fortunately nothing serious happened while 911 was out.
Then Qwest did it again, two days later, on the same line...
Ah, telecom monopolies.
Re:Ah, yes, Qwest did this to my home town... (Score:2)
Information Technologist vs. The Red Neck (Score:5, Funny)
Red Neck: 1
Re:Information Technologist vs. The Red Neck (Score:4, Funny)
Information Technologist: 8,050,000
Red Neck: 37,200,000
http://www.googlefight.com/index.php?word1=Inform
They call it an "Accident" (Score:2, Funny)
is this what they were thinking?
Argh i give up! Those conniving small minded cable companies
--
Learn from the backhoe (Score:3, Interesting)
When CU Boulder put in their fiber ring, they ran the spans in separate conduit, which they lay in the same trench. The conduits were not at different depths, nor were they really that far apart (about 3 inches)
They put the bright orange plastic sheet ("Hey backhoe guy! Stop digging now!") right on top of the conduit, then filled in the trench.
Surprisingly, it got cut.
It's the backhoe (Score:3, Funny)
Re:It's the backhoe (Score:2)
Re:It's the backhoe (Score:2)
Thats one slippery slope mostly because if the person who kills with the gun was hired by someone else then that person is also responsible, but what if that person is a business or government.
Things aren't as black as white as it seems when you place blame...
If a government sends off soldiers to war and a soldier accidently shoots a civilian by accident, then is it the gun fault for a bad s
From ze smallest divot to ze largest canyon... (Score:3, Funny)
Backhoes and Sharpened Sticks (Score:2)
"The Internet was impervious to our most powerful magnetic fields, yet in the end it succumbed to a harmless sharpened stick."
Should increase liability / penalties (Score:5, Insightful)
I wonder just how much those incidents would be reduced if companies were fined a stiff penalty for digging without calling these numbers. The type of astronomical fines/penalties levied against virus writers would seem very appropriate in these cases, given the type of economic damage that can be caused by telecom outages.
I'm glad to see that a national calling center is being established (similar to 911, according to the article). Now, it will be easier for workers to call. But I still think we need the other half: better (financial) incentive to make those calls in the first place. -- Paul
Re:Should increase liability / penalties (Score:5, Interesting)
Of course, no one ever calls. I work for a local utilities company, and lines get cut ALL the time by contruction crews. Because they almost NEVER call for a locate. It's insane.
The few times they do call for a locate, we go out and mark the lines, and they cut them ANYWAY. Unbelievable.
Re:Should increase liability / penalties (Score:2)
Re:Should increase liability / penalties (Score:3, Interesting)
And a large part of the other half, like this particular incident, is probably because the digger got an erroneous answer from the support number. A contractor for Verizon buried fiber optic cable (for FIOS [verizon.com]) in my neighborhood late last year. Prior to their arrival, the cable, electrical, and natural gas utilities marked the
Main Pipe (Score:2, Funny)
Rumors said the guy was fired due to failing a drug test.
Re:Main Pipe (Score:2)
Rumors said the guy was fired due to failing a drug test.
Fired? That's nothing, back in Biblical Times he would be totally stoned. [wikipedia.org]
Hah! (Score:5, Funny)
For some reason the Servers and Networks guys don't think it is funny.
Re:Hah! (Score:2, Funny)
Probably because your faux Fudd is flawed; should be:
"Be vewy, vewy quiet... I'm hunting fiber"
Give this subtle nuance a try and report back with your results.
Go! We're waiting!
ARPA identifies backhoe fade effect (Score:4, Funny)
this as "backhoe fade" and ARPA has conducted
considable research on the effect of fiber
optic cable to attrace backhoes in the wild...
ARPA Science Research Funding News Today......
ARPA to Fund Network Reliability Research
Washington, DC -
The Advanced Research Projects Agency of the DoD announced today they are
funding a three-year effort to improve the field reliability of
fiber-optic communications networks. The program is aimed at reducing
network outages from damage to buried fiber optic cables caused by
construction machinery. Many telecommunications outages are caused each
year when machines called "backhoes" dig-up underground fibers, cutting
them and causing massive service disruptions.
This phenomenon is commonly referred to as "backhoe fade" and
the uncanny ability of the construction backhoe to locate buried
cables will be the focus of this effort.
Dr. Zweiback Gimfizel of the Marginalia Institute of Technoplasty
has been designated Principle Investigator on the project and
held a news conference today and described the proposed line of
research.
"We are taking a page from the biologists who discovered
the magnetic organ in the brains of homing pigeon. This
organ senses the earth's magnetic field and allows the
pigeon to track its location.
"In like manner, our research will focus
on identifying the specialized organ structure within
the backhoe that can somehow sense the location of glass
fibers."
"The hope is that if this fiber-seeking mechanism can be
identified, measures can be developed to disguise
telecommunications cables, thereby creating "stealth"
fiber bundles which will not attract the attention of
the rampaging backhoes."
In another unrelated statement today, ARPA announced the creation of the
Remote Autonomous Rodent Program which will work on developing specialized
weapons systems for attacking the underground communications systems of
adversaries. In recent theater actions, modern fiber-optic communications
systems have proven quite resilient to traditional attacks and require
new techniques to disable them.
Dr. Gimback Zweifizel of Hardly Yardwell University was designated
Principle Investigator. In a prepared statement, Dr. Zweifizel noted that
this work program was funded for three years and was to produce a field
demonstration of a working system. Other details of the project are
classified.
I work in the pipeline construction industry (Score:5, Informative)
When nobody comes out an marks, and their line gets hit, it's on them. If it's marked and we hit it, it's on us. Accidents happen. Digging around mismarked and unmarked utilities in a big hole in the ground isn't easy.
Personally I'm more worried about my guys hitting a pressurized gas line than someones precious telco wire. Wire gets fixed in a matter of hours.
How to recognize a backhoe (Score:5, Funny)
The Common Yellow Backhoe [68.232.111.189] attempting to hide from view.
The Hammer Backhoe [onsiterents.com] evolved to fit particular niches.
Re:How to recognize a backhoe (Score:2)
We'll never win the war against stupidity (Score:2)
Anyone remember the 4-story high geyser in Old Montreal last year?
You'd think that backhoe operators would triple-check what's underground in a metropolitan area...
Forget about civilian damage... consider worse (Score:2)
I'm also left wondering why these big players like Spint doesn't have two wires for every important line like this? Cut one wire and the alternate route patches over with a notice. Cut the other and a notice is issued... both without incident to large scale service. If I can imagine it, then I know someone else out there has already thought of it.
DIY Backhoe Fade (Score:5, Funny)
Next time I created backhoe fade, was again in an unexpected way. I'd been trenching along the driveway, after dutifully and carefully marking the underground phone line to the house (that the brain-trust from the phone company decided to run next to my driveway, despite instructions not to). I carefully and successfully avoided the cable, no worries there. Then, when reaching juuuuuuust a bit too far over, I got the backhoe stuck in the muddy ditch along the road. Apparently, in the effort to get un-stuck, I pressed down on the cable, which then stretched over a rock in the trench and broke.
The phone company (eventually) got out there and tried to say I dug it up. I showed 'em exactly what happened - yes, I'd been digging. Yes, the wire was marked. Yes, none of my digging was along the wire's path (all true). The cable had clear marks of a pull over a rock, not a cut from a hoe. Shear vs. tension, obvious from inspection.
Phone company guy didn't want any part of explainations until I (a) bet him that I could dig right (made an X) here and find a big rock with a sharp edge "that you people left in the trench of this improperly installed wire", and (b) pointed out that if he's gonna dig the trench, he's standing in poison ivy while doing so, and I could just go get the backhoe and make it easier for all involved.
He called his boss, explained the high points of the situation (including the poison ivy, which inexplicably a guy in his job didn't recognize without help), and they fixed the cable no charge. But, I bet I'm one of those statistcs in the article.
T-Shirt (Score:3, Funny)
This is defiantly true though. Living in a fairly recent subdivision, back when the construction was closer to my house this would happen all the time. The phone. The cable. The internet. Even the power once.
I think it's clear what we need to do: go kill all the backhoes.
Save the internet!
That needs to be on a t-shirt! (Score:2)
A good sysadmin always carries a few feet of fiber (Score:4, Funny)
bad joke time (Score:2, Funny)
Backhoes Don't Care (Score:5, Insightful)
I found the break. The wire had been cut and tied off. There was barely enough wire to splice the two back together. Once repaired the monitor speaker worked again. I was told later by the airline employees, airport facility workers had redone the ceiling in that one room. To me it appeared the workers found the wire in the way of their job, didn't know or didn't care what it was hooked to and simply cut it and tied it off out of the way.
Backhoe operators probably have the same mentality. They want to get their work done. If they cut a cable, it doesn't affect them. They are just doing their job. To solve this problem I would recommend burying fiber next to gas lines. The fiber should be coated with a material that bursts into flame 30 seconds after it exposed to air or cut. Not only will the backhoe operator cut the cable he'll break the gas line as well. The 30 seconds delay is to build up enough gas for a nice explosion. Sure it'll be a mess, but that's one backhoe operator who won't cut any more fiber.
Huh. And here I thought... (Score:2)
Opportunities for terrorists (Score:2)
Steve Irwin to the rescue! (Score:5, Funny)
Today I'm going to show you one of the wonders of nature. If you look down in that hole there you can long fibrous tendrils. Those are fiber optic cables and they snake their way through the ground all over the world.
Crikey, it's nice to see them. Usually they stay underground so this is really special. Just look at the size of the hole they make as they burrow through the earth.
Oh look! I didn't expect this. The only known enemy to these folks is coming over to investigate. The backhoe. Look at those nasty pointed teeth. I wouldn't want to get caught by them I'll tell you.
I'll just walk away so I don't disturb him. This could get real exciting any moment.
*growl* *snort* dig dig dig dig
Look at that! This is a real treat. The backhoe is digging up the fiber optic! Look at the way those teeth just dig into the soil and expose those poor buggers. Oh wow, just look at it as it tears those fibers to ribbons.
I know it may seem cruel to stand by and do nothing but this is part of nature. Someone has to eat and someone has to be eaten.
But don't worry mate, those fiber optics grow back real quick. In fact, they grow so quickly there will never be a shortage of them no matter how many get eaten by the backhoe.
Charge Providers (Score:2)
Neal Stephenson's observations on backhoes (Score:4, Interesting)
Q: Why bother running two widely separated routes [for cable from Point A to Point B] over theMalay Peninsula?
A: Because Thailand, like everywhere else in the world, is full of idiots with backhoes.
Q: Isn't that a pain in the ass?
A: You have no idea.
Be thankful that you only find fibre (Score:5, Interesting)
A short period of digging later and he came out the hole at some speed looking very pale. The said "pipe" had fins on one end and was delivered 60 years ago by some Germans who failed to stop and advise my grandparents of the delivery.....
Statement from an underground construction worker. (Score:3, Insightful)
Re: I can't wait (Score:2)
Will technical marvels never cease?
Re:Nothing is for certain... (Score:2, Insightful)
Re:Nothing is for certain... (Score:2, Informative)
Re:Nothing is for certain... (Score:4, Informative)
Cable information isn't always right (Score:3, Informative)
For a lot of the "middle of nowhere" fiber feeds they bury them at least 6' (2M) deep. An electrical contractor friend of mine was doing a job "just north of middle of nowhere." He'd had the major fiber carrier in the area come out and mark where the bundle was buried. And they assured him it was 6' down -- which worked since he was only digging down about
To drive home the point (Score:3, Informative)
They were all off by two feet, in the same direction.
We were told they were 6' down, we snapped cable at 1 1/2', good-bye phone at 2'. After that first snag, we dug VERY carefully with hand tools, only to find power not a foot away and a foot further down. Either the ground is shifting in Tennessee, or we've had some REALLY stoned public
Re:Cable information isn't always right (Score:5, Informative)
Depth is useless; in Arizona, for example, snow-plows are required to "call before they plow". Why? Soil erosion.
Here in my state, depth is likewise useless; not as much from erosion as it is from grading. Infrastructure goes in first; landscape happens last. It is QUITE common for a 48" deep line to be 24" from the surface after several years.
And that isn't accounting for things that were discovered when trying to bury the lines in the first place; intersections with other plant means you change height at that location. Hitting Bedrock... means you change height.
> had the major fiber carrier in the area... assured him it was 6' down
Not likely. The moron sent to locate the cable may have mentioned the depth in passing, but I work with these "major carriers" and their locators every day, and there is no way in hell they'd say "you're fine to use your backhoe directly on top of my wire up to 5 feet 11 inches deep". Most "Major Carriers", on a long-haul line, will physically PUT A BODY on-site during the dig to enforce the protection of their cables by hand digging over it. If it's an issue, or a very high-value asset, they'll even go so far as to hand-expose it, themselves. They do not, ever, say "sure! Just dig right on top of it".
Ever.
What I'd suggest is that you ask your contractor friend to define what he means by "assure". And as you do this, remember that he's getting sued for being at fault... he won't do anything to deflect responsibility, at all... he certainly won't exaggerate what was said, for certain.
Re:Cable information isn't always right (Score:3, Funny)
Is that what that was? I thought we'd found Jimmy Hoffa! I guess we can tell CSI:Backhoe that there's no hurry.
Re:Nothing is for certain... (Score:3, Insightful)
Re:Nothing is for certain... (Score:5, Informative)
Re:Nothing is for certain... (Score:2, Funny)
Re:Nothing is for certain... (Score:3, Insightful)
Anyone else bust out laughing at the thought of people trying to swim in a foot and a half pool... or calling a dig service in order to put their kiddy pool in ground?
For parent: it's 18', not 18".
Re:Nothing is for certain... (Score:5, Informative)
I had to do this when I dug up part of my front yard to put in a flower bed.
Re:Nothing is for certain... (Score:2)
Re:Nothing is for certain... (Score:5, Interesting)
Re:Nothing is for certain... (Score:3, Interesting)
It didn't work; GPS is good for 20 feet usually, whereas excavators need to know within a bucket-width (24"). That means that the GPS error, combined with OUR error, must be less than 24". Not gonna happen, especially since we're dealing with junk that was buried over a century ago.
The second reason it didn't work was because retards would transpose digits whi
Re:Nothing is for certain... (Score:3)
Actually, isn't the accuracy of consumer level GPS is insufficient for this task?
You'd think that some of these buried lines predate GPS, wouldn't you?
Re:Nothing is for certain... (Score:5, Insightful)
There is NO contractor in the world that would accept the responsiblity and/or liablity of locating utility assets (gas, telecom, water, electric and so forth). Each utility provider will dispatch specially trained and equipped technicians to perform this service. The "locater" must be accurate within certain tolerances or the utility assumes the liablity associated with any distruption/repairs including contractor's equipment that was damaged.
Speaking from experience, I have seen a 60 inch water main broken (locator was wrong), a large telecom cable (something insane like 5,000 pairs) running to a 72 story office building (excavator problem) and countless single line telecom cables (just trying to find the damn things using a shovel but electrical tape works wonders).
The short answer is, you can have all the centralized documentation/maps whatever that you want but no contractor will ever put a shovel into the ground until the utilities come on-site and say "You can dig here but not there".
Already in use... (Score:3, Interesting)
http://tinyurl.com/6zw7u [tinyurl.com]
Here's yer flaws (Score:3, Insightful)
The flaw is in the fact that all these people have to do the right thing. In this case, if some low-level Sprint employee reads the map wrong, a whole state can be without
Re:Nothing is for certain... (Score:5, Funny)
Simple, easy, flaw (which I'm sure you've already thought of) -- human error. Like the time the construction worker started digging a hole next to my house right on top of the orange paint mark specifying the location of my phone line.
The funniest thing was the foreman trying to fix the line, since the phone company (thank you, SBC) said they'd take a day or two to get there. He was shocked (literally) to find out that phone lines carry electricity. :-)
Re:On related notes... (Score:2)
Is that The Sharpie, or The Shift Key?
Actually they both sound about the same.
Re:Nothing New (Score:3, Interesting)
Anyone remember back in the late 90's when AT&T lost its ENTIRE frame-relay network [networkworld.com]? Some 6,000 or so customers suddenly lost network connectivity?
According to the scuttlebutt around AT&T a piece of construction machinery backed into some sort of switching station and took the whole thing out. 6,000 customers, just like *that*. Try beating that one.
Re:Nothing New (Score:3, Informative)
Re:Nothing New (Score:2)
I imagine that the reality was probably more complex. i.e. A severing of a connection could have started the cascading failure. Or it could have been one big cooincidence. But internally, the blame was placed squarely on a backhoe.
Re:Nothing New (Score:3, Informative)
http://web.archive.org/web/*/http://www.att.com/p
Re:Nothing New (Score:4, Funny)
Re:Nothing New (Score:3, Funny)
Re:internet not interline (Score:2)
Re:lowest bidders what? (Score:2)
It wasn't the people that layed the fiber that were the problem, it was the people to later work, with a backhoe, in the same place, that didn't use the utility marking system to know where they shouldn't dig.
Capitalism at work? I supposed you're comparing the use of contractors to the fabulous infrastructure work done by, say, the Soviet Union? Why, East Berlin was the very picture of sound, robust, aesthet
Re:lowest bidders what? (Score:2)
Re:lowest bidders what? (Score:2)
ObKentBrockman (Score:2)
Re:In Soviet Russia... (Score:2)