A Cyber-Attack On an American City 461
Bruce Perens writes "Just after midnight on Thursday, April 9, unidentified attackers climbed down four manholes in the Northern California city of Morgan Hill and cut eight fiber cables in what appears to have been an organized attack on the electronic infrastructure of an American city. Its implications, though startling, have gone almost un-reported.
So I decided to change that."
Cables were cut in San Jose and San Carlos (Score:5, Informative)
Bruce, the cable cuts were in San Jose and San Carlos. The cable between San Jose and Morgan Hill was cut, but the cut location was in the city of San Jose.
(otherwise, agree with what you said, hopefully wider audience for this will help...)
Re:Hams FTW (Score:5, Informative)
"Any links or info for someone looking at picking it up?"
http://www.arrl.org/ [arrl.org]
http://www.hello-radio.org/ [hello-radio.org]
San Francisco Chronicle Coverage (Score:5, Informative)
Re:Hams FTW (Score:4, Informative)
arrl.org (the American Radio Relay League). Also, the electronic department of most community colleges have a ham club, which offer the tests on a monthly basis.
You don't need to know Morris code any more, but you do need to study up on radio & electronic theory. Radio shack used to sell the Ham license study guides, but I don't know if they have them any more.
Re:Terrorists? Probably not. (Score:4, Informative)
Or just regular blackmail:
http://yro.slashdot.org/article.pl?sid=09/04/20/1427259 [slashdot.org]
I assumed these were both the same story at first. But the YRO story was 2005, and this one was a few weeks ago.
Re:Redundancy, redundancy, redundancy... (Score:2, Informative)
Discussed on NANOG (Score:4, Informative)
This was discussed extensively on the NANOG (North American Network Operators Group) email list.
It appears that the outage affected multiple carriers including ATT and Alternet.
Re:Cyber(?) Attack (Score:3, Informative)
I guess it's kinda reasonable to use the term for an attack on the "cyber" domain (by going after its physical substrate) as well as for attacks that occur within that domain. Either way, it screws up people's access to comms.
Re:Cables were cut in San Jose and San Carlos (Score:3, Informative)
Morgan Hill (Score:2, Informative)
I stayed in Morgan Hill last month.
It's not really a "city", more like a town south of the Bay Area.
Re:Cables were cut in San Jose and San Carlos (Score:3, Informative)
What Bruce Left out (Score:5, Informative)
The entire Santa Cruz County area was cut off from all telecommunications outside of Point to Point wireless and Satellite. (Comcast customers aside.) Sprint, AT&T, Verizon, long distance for POTS was all down. TFN's were not able to be dialed by any customers. 911, 611, 411 were not functioning. 'Point-to-point' T1's that were aggregated over DS3's in Hayward, were not functioning for area users. Many of the "redundant" network connections for companies in the Monterey Bay area were completely down. Both legs of their "best practice" 2 provider networks were crippled.
Other than a couple islands of connectivity (namely the Shell Gas station at 41st and Capitola Rd in Capitola, my mother In Law's house, and my Uncle's business) who were lucky enough to only have Satellite service available to them, or were on Comcast, the packets stopped flowing.
Ironically Comcast services inside the Santa Cruz county were still working. Users of Comcast voice wouldn't have noticed (except for the fact that everyone they called went straight to voicemail.)
However, inter CO calling was working (you could
call anyone in the Watsonville-Santa Cruz area if they had a POTS line from a POTS line. Still, corporate communications for nearly everyone in the area (Ag. Brokers, Packers, Pickers, Shippers, Bottlers, etc.) Was down. Commerce came to a halt.
People couldn't get gas at gas stations around the area unless they had cash. Area banks wouldn't let people inside the bank unless you were making a deposit. People couldn't be players in the game of commerce without little pieces of paper. And so once again, cash was king.
More cars sat on the side of the road that day then normal between santa cruz and watsonville. Which begs the question how does the regular joe call for help if the call boxes can't talk to a phone switch?
cloud computing (Score:5, Informative)
I guess this kinda puts a damper on all the cloud computing hype of late...
The UK centralised electronic medical records... (Score:5, Informative)
When I first saw the way that one worked, I shook my head, and said "You're joking, right?"..
Alas, the answer was no. And the reason that it had been designed as a centralised system (well, ok, there's a 'failover' data centre or two) is (according to the designers) that you'll never lose the main and the redundant connections at the same time.
I seriously hope that they're paying attention to this at the moment. The severing of very few, carefully chosen fibres could quite simply deny a lot of UK hospitals access to their medical records. And if all come on board, then you could deny nearly all hospitals access to the medical records.
This, as can be imagined, would be rather a bad thing...
Re:The hospital (Score:3, Informative)
Re:What Bruce Left out (Score:3, Informative)
"Manholes?" (Score:4, Informative)
The cut location in San Carlos was reported as being at Bing St and Old County Road. That's actually alongside the rail line that runs up the SF Peninsula. There are many fibre optic cables along that right of way. It used to be a Southern Pacific Railroad line, and "Sprint" was originally Southern Pacific Communications.
There aren't that many long haul fibre optic cable routes. Many of them run along rail lines, because the railroad owns the right of way and doesn't need anyone else's permission to run cables. Often you can run cable for miles without crossing a street, which makes installation much simpler.
dupe (Score:5, Informative)
Its implications, though startling, have gone almost un-reported. So I decided to change that.
DUUUUUUPE
http://tech.slashdot.org/article.pl?sid=09/04/09/2044205 [slashdot.org]
Re:Redundancy, redundancy, redundancy... (Score:5, Informative)
Reducing single points of failure is what is needed
The cost of doing this is enormous, which is why it will never happen 100%. The scale of this outage is no where near what we had in the Chicago area when the Oak Brook central office caught on fire http://catless.ncl.ac.uk/Risks/6.82.html#subj2 [ncl.ac.uk], and that was 20 years ago. I don't think any one system is any more fault tolerant now than it was 20 years ago, but there are now multiple providers which can mitigate it significantly as long as they don't all route through the same cables as was the case here to a large degree.
In the end any telecom system is vulnerable in localized areas... the trick is to make sure it cannot all be disabled (although software has managed to do so to great effect in the past http://www.soft.com/AppNotes/attcrash.html [soft.com]
...
Re:Oh, Bruce (Score:5, Informative)
The way I said it was right. DARPA had Army and other DoD sponsorship. I said the scientists involved designed it to be militarily redundant. The fact is that the military didn't use it that way.
Unfortunately, the main reference on the hospital is the ham coordinator, as quoted on ARRL's site:
The hospital isn't talking about the technical failure.
AT&T wants to know: $100k reward (Score:2, Informative)
Well whoever it was, AT&T is offering up $100,000 to find out. Sounds like AT&T might be a little upset.
Below is an excerpt from the article [kionrightnow.com]. The link to the story has an interesting video clip from a local news station (see "video gallery" (flash)). Interesting to me, anyway, as I've never seen a cut fiber cable flopping about. The "play by play" event sequencing was also interesting to see; sounds like it hit the fan about 2am local time.
---begin---
AT&T is Now Offering a $100,000 Reward for Phone Vandalism Information
AT&T is now offering $100,000 reward for information leading to arrest/conviction of those responsible for California phone vandalism. To report information call 408-947-STOP
Police say someone cut the fiber optic cables inside the south San Jose vault on purpose early Thursday morning. ...etc...
---end---
Re:Cables between India and California (Score:3, Informative)
No, that's not correct, though there's a certain amount of Moore's-Law-like behaviour where the newest cable always has a significantly higher capacity than anything built before it. There's a limited number and capacity of cables going from India to Europe through the Mediterranean, but a somewhat larger number going to and/or around Singapore, and from there there's a wide range of cables heading to North America, either more or less directly, plus a bit of connectivity going to North America by way of Australia and even less going to Europe around the southern end of Africa.
For India-Europe, the cables mostly go through the Med, and have been getting cut a lot recently, usually by ships but occasionally by earthquakes. For India and Southern Asia to Japan and North America, almost everything passes between Taiwan and the Philippines, as we discovered in the earthquake a couple of years ago that took out 95% of it at once (and there's now an effort to build some that go around the other side of the Philippines, but the geography's difficult, and there's some growth in land-based cables across Russia and Kazakhstan.) Australia has decent connections to the US, if you don't mind a few thousand extra miles worth of milliseconds, but their connections to Japan that don't go through the Taiwan Straits mostly go via North America, though there's increasing growth in connections via Hawaii and Guam that cuts off some of that distance.
Re:Hams FTW (Score:3, Informative)
All that proves is that you are unable to google for katrina "ham radio" [google.com]
ISP isolated from the world (Score:5, Informative)
I am the Network Administrator for an ISP (AS4307) in San Martin, CA (between Morgan Hill and Gilroy) that was directly affected by the cuts.
We are multi-homed by two providers. BOTH providers fiber ran through those SONET rings that were cut. We were COMPLETELY isolated (internet, POTS AND cell) from 2:15am to 10:42pm. Luckily, 90% of our customers are in the Morgan Hill/Gilroy/San Martin/San Jose area, so they were fully aware of what happened.
As a side note, the cuts were actually in San Jose. I live 3 blocks from where the cuts occured (Monterey Hwy and Cottle Rd. for those interested). And it did not just affect Morgan Hill. Some parts of South San Jose were affected, along with Morgan Hill, San Martin, Gilroy, Watsonville, Santa Cruz, and parts of Hollister.
What was interesting was when service was restored, customers who lived out of the area who had not heard of the happenings here, called and told us they thought one of two things:
A) We went out of business
B) Natural disaster (Earthquake was #1 on the list, considering where we are located)
We lost no customers over this fiasco, and are now looking at getting a provider that feeds from completely separate fiber (i.e. from the SOUTH)
Robert Glover
Director of I.S.
South Valley Internet (AS4307)
When I heard that two regions were out ... (Score:3, Informative)
... and that two cable cut sites had been found, I speculated that there were two more sites. Turns out that was the case.
The SONET network is normally configured in a ring, or a set of interconnected rings and ring segments - a net with MOST nodes being points on a line and a few being points at a Y junction. (It's the cheapest way to insure two geographically diverse paths to every site when you have to dig things up to string your connections.) The rings are configured so that a cut link is automatically bypassed. (The traffic may already be propagating around the ring both ways and the sites just switch to the side that still has good info. Or it may have reserved bandwidth and when a link goes down the sites beside the cut "fold the traffic back" onto the reserved bandwidth.
Packet networks can have similar redundancy characteristics:
- They may be carried on existing SONET infrastructure.
- They may be connected as "Redundant Packet Ring" - essentially the IP equivalent of SONET rings using arbitrary transport methods with the same physical layout.
- Or they may have any of a number of other net-style redundant connections. (But they usually reduce to the same geographic layouts.
- (Or they may be non-redundant or "2x-redundant" with both cables taking the same path. Oops!)
Given this, when I heard that there were two dead patches and that phone service (along with everything else) was out, I figured the dead patches were on rings and that there had to be cuts on two points of each ring to defeat the redundancy.
Now we hear that there were indeed four manholes entered and cables cut in each.
So it sounds to me like the system ALREADY had the redundancy built in - but the attackers knew about it and deliberately made the multiple-location cuts needed to defeat the backups.
Re:Redundancy, redundancy, redundancy... (Score:4, Informative)
Re:Redundancy, redundancy, redundancy... (Score:5, Informative)
Using tax money to pay for stuff doesn't make it cheaper - it just hides the cost. If anything, the guy laying fiber for the government will probably make more money than the guy laying fiber commercially.
Using tax money to provide goods and services does two things: it hides the true cost by shifting the burden of payment onto other people and it eliminates choice. Those are both bad, but for different reasons.
The first is bad because you're deciding how to spend other people's money, and they don't have any choice in the matter. Even the tiniest gain in performance is worth almost any cost if you're not the one paying for it.
Besides that, eliminating what little choice there is in broadband connectivity would be bad because the government would undoubtedly contract the work out to an existing telco.
Re:Society is cooperative in nature (Score:3, Informative)
Specifically, the US Constitution makes any ratified treaty binding law in the US. The US ratified the Geneva Conventions, which prohibit torture of enemy combatants, in 1955. Therefore, under the US Constitution, the torture of enemy combatants is a violation of US law.
Many errors in this story (Score:3, Informative)
Since I live in the area where this happened and it was reported extensively on the local news, I noticed *many* errors in TFA, such as :
- Morgan Hill was not specifically targeted .. the cuts were in San Jose and Santa Clara. At most, Morgan Hill was collateral damage.
- Cables were cut in four different locations, so there was no single point of failure.
- Hosting everything at your site might help in cases like this, but is your mail really more reliable if managed by a part-time sysadmin on a single $1000 box, or at Google where they have triple-redundant everything?
I lived this (Score:3, Informative)
Re:Many errors in this story (Score:3, Informative)
San Jose and Santa Clara had other communications sources and do not seem to have had outages nearly so complete, and didn't (as far as I'm aware) need to get hams to help them run the hospital. So, I focused on Morgan Hill.
I did mention that redundancy might not have helped this case.
Yes, one beige box and one operator would be the wrong way to go for a hospital. I think database replication is the best way to handle this.
Re:Redundancy, redundancy, redundancy... (Score:2, Informative)
I just have to say this. You're an idiot.
Nothing you said about a public fiber project is true. My city rolled out fiber to every home in the city, at no cost to taxpayers.
It doesn't eliminate choice, it enables it. The city can run fiber and offere services over it, and it can also open it up to other to offer the same services on the fiber. Because the organization only needs to pay for upgrades / maintence, not make a profit, the cost to provide services is lower.
Even if you were to use tax money.. so what? My tax dollars are stolen from me for a number of things I don't support and I'm willing to bet you do. So tough shit.
Your "reasons" for not wanting fiber as infrastructure are totally off base; or do you argue that the road system would be better if private companies managed it?