FBI Says Military Had Counterfeit Cisco Routers

There are new developments in the case of the counterfeit Cisco routers, which we have been discussing for some time. The NYTimes updates the story after an FBI PowerPoint presentation made its way onto the Web. It seems that experts at Cisco have examined some of the counterfeit routers in detail and proclaimed that they contain no back doors. Others don't believe we can be so sure. "Last month, [DARPA] began distributing chips with hidden Trojan horse circuitry to military contractors who are participating in the agency's Trusted Integrated Circuits program. The goal is to test forensic techniques for finding hidden electronic trap doors, which can be maddeningly elusive... The threat was demonstrated in April when a team of computer scientists from the University of Illinois presented a paper at a technical conference in San Francisco detailing how they had modified a Sun Microsystems SPARC microprocessor... The researchers were able to create a stealth system that would allow them to automatically log in to a computer and steal passwords."

And outsourcing....

Verification of the producer is essential here - and this is perhaps the moment where outsourcing will bite us in the ass. While you can only buy american made cisco routers, there is no doubt some chipsets made in it are manafactured overseas.

Re:And outsourcing....

While essential, it's not the only step. Automatic tests of the router hardware, random checking, and employee control are all necessary steps if we really want our government networks to be secure.

Re:And outsourcing....

Even automatic checking and testing can be subverted by a determined adversary. For example consider the program, approved by President Reagan and beginning in 1982, whereby the CIA arranged for the Soviets, who were actively attempting to acquire western technology and hardware, to receive natural gas pipeline software and equipment that was designed to "go haywire" after a set amount of time in normal operation. When the pipeline software and hardware, which would have appeared totally normal at first even if the Soviets had bothered to test it, eventually went haywire (i.e. it ran the ultra high pressure scenario) the resulting explosion was so large that it was detected by satellites designed to monitor nuclear explosions from space. The following article [msn.com] from the Washington Post describes this and other programs and operations that took place during the Cold War as part of a coordinated CIA effort to slip bad technology to our enemies.

Re:free software distributes the effort.

Items with high capital costs don't work well as "open source;" basically, the manufacturing plants costs so many billions of dollars that no one who isn't doing proprietary work could afford it. Even if you could open source chip design (a dicey proposition, since there are many fewer EE Phds that want to donate time than there are CS Phds,) there are still difficulties with the actual manufacturing, and we would still need to guarantee the physical chips, which are individual, and cannot be "re-compiled;" if you think there may be an issue with a batch, you can't start over without paying for new chips.

Maybe, however, I am missing something about the procedure you are proposing; what parts would be open source?

Re:And outsourcing....

I would agree on this one 100%. Outsourcing is in part responsible for this, although, we can't ever be 100% sure of goods manufactured domestically. These companies could care less about counterfeiting Cisco routers so let's say Cisco were to pull the contract. What's to stop the outsourced producer from selling these counterfeits in their own country without the Cisco name? Maybe they already are because presumably (as I understand) because IP law is hard or impossible to enforce in some countries.

As the NSA already seems to be certifying comm. gear in the military (or might even make the chips for it). Perhaps even for other departments like the FBI. I see one possibility of this that the NSA certifies routers (or makes them itself) or at least makes them in the USA. I don't work with routers nor am I familiar with their manufacturer. I guess my last point, pertaining at least to the FBI investigation, would be invalid if Cisco makes some routers in the USA except, as you indicate, for some chipsets. Though even on chipset in itself could pose a significant risk.

I'm just surpised that the FBI is even making a "presentation" to anyone on this; regardless of wether the presentation leaked or not.

Re:

As you say, even domestically-produced hardware can theoretically have trojans in it, so it should be standard practice to certify everything they use, regardless of where it came from.

The outsourcing boogeyman has nothing to do with this - relying on the

Re:And outsourcing....

It's worth noting you can do everything a Cisco router can do with a Linux box.

Except connect to a SONNET network. Or a DS3 interface. Or aggregate multiple T1s. Or suport terabit switching and routing speeds.

Re:And outsourcing....

I don't know if that will be enough. I remember there was a story from the Cold War on how the CIA spied on the KGB. The KGB used Xerox copiers in their offices. I don't know if the CIA convinced Xerox to modify the copier or they modified it before the KGB received it, but the copier would record all copies to flash memory. Every so often, the CIA would have to retrieve the memory. The KGB eventually got suspicious that one machine seemed to be serviced all the time while the other one wasn't. They weighed both machines and found a tiny difference in weights. Eventually they found an extra board. That's my recollection of it. I can't seem to google for the backstory. Even if you bought 100% American parts, there is not guarantee that it wasn't tampered with during a routine repair and maintenance or tampered with in the manufacturing process.

Re:And outsourcing....

http://www.parascope.com/articles/0197/xerox.htm [parascope.com]

"Counterfeit" not an issue...

From what I understand, the counterfeit routers are made in the same factories by the same people who make the real routers; they just keep the assembly line running past the hours that Cisco is paying them for.

In this case, if Cisco is comparing the counterfeit routers to their legit ones, they should always be the same.

The question this doesn't answer is this: does the LEGIT Cisco equipment contain back doors? How can Cisco be sure it doesn't? Most of the components are manufactured offshore and the assembly is done offshore. Have they examined each part with an electron microscope to verify it doesn't do anything more than what the spec says it should do?

They can't just watch for network activity; these routers might be filtering and caching data waiting for the eventual physical removal of the router in the next upgrade cycle -- or, they might all have a kill switch built in, so someone can remotely take out ALL routers. There are an infinite number of possibilities to look for, and since Cisco doesn't manufacture everything in-house, they really don't have much hope of detecting that none of the infinite possible modifications have been made.

"Partnership"

Anne McCaffrey wrote a book called PartnerShip [amazon.com] with a plot very similar to this situation. The villian provides chips to the Galaxy, including the military. When nearly everyone has upgraded, it turns out that he can remotely control every device, including military hardware, controlled by the chips. That's enough of a spoiler. How can such a grand and well planned scheme be defeated? You'll have to read to find out...

Re:"Counterfeit" not an issue...

I am generally for free trade and against protectionism, but I am leaning more and more towards the need for a law that makes it mandatory that all gear (guns, routers, computers, coffee makers, etc.) purchased by the Government for any use that is even remotely sensitive be made in the US by US owned companies. That won't necessarily solve this kind of problem, but it would certainly make it far easier to prosecute entities who do things that threaten our national security.

Re:"Counterfeit" not an issue...

As being someone who recently has purchased several Cisco products on Ebay lately, I can tell you that the counterfeit items are not made on the same assembly line. There are several design differences between them.I use the "Andover test" to tell if I'm purchasing authentic Cisco cards.

If I did purchase a card or Cisco product that did pass the Andover test, then chances are that it was manufactured on the same assembly line, but then you would most likely see a report of a duplicate mac address on a "genuine" Cisco product somewhere. So yes it's a possibility, but highly unlikely IMHO.

Selling out the back door

the counterfeit routers are made in the same factories by the same people who make the real routers; they just keep the assembly line running past the hours that Cisco is paying them for.

That happens ALL the time. I've visited manufacturing plants in China and I've seen it happen with my own eyes. Selling out the back door is not surprising at all. In fact this is why I'm less worried than I might otherwise be about the gear having back doors or being otherwise compromised. Simplest explanation is just theft in one form or another.

does the LEGIT Cisco equipment contain back doors?

Very good question. Got to be worrisome to the US military and security agencies. Much/most off the shelf hardware is made outside the US where it wouldn't me much of a stretch to imagine backdoors have been added by foreign governments. Same worries that other governments have about US made/designed software and hardware. And of course if you really want to get tin-foil-hat about it one has to wonder if our own government has had back doors installed. Very unlikely to be sure, but clearly possible.

That said, it's pretty low on the list of likely threats. Pretty hard to know exactly what gear will be placed where and what it will give you access to. Plus even with a back door, places with sensitive data are more likely to be monitoring the traffic which is harder to hide.

Re:"Counterfeit" not an issue...

From what I understand, the counterfeit routers are made in the same factories by the same people who make the real routers; they just keep the assembly line running past the hours that Cisco is paying them for.

I keep hearing this. But look at the images of the hardware side by side [cachefly.net] ... Is it the same? No it's not. Clearly these two boards are not from the same manufacturing line.

Not a big surprise.

I work for a company that sells used electronics on eBay. We'll occasionally buy cheap gear over eBay too, then resell it at a profit. For many months now we've had a huge problem with counterfeit Cisco cards. It's amazing how detailed the counterfeiters are. My boss wrote up a detailed guide on how to spot fakes. Google "counterfeit cisco wic".

Fear Fear Fear

Be afraid. Be very afraid. Vote for those that seek to protect you.

This seems like a scare tactic to "warn" people about the dangers of fake hardware/software. Expect a big push around these types of "stories" as more bills like PRO-IP go through congress and as the creation of the IP & Copyright Czar in the Whitehouse gets a big push.

It's a concern but seems to point more to incompetence rather than some difficult-to-spot threat. Why are government agencies not buying directly from Cisco? Seems they should have some sort of corporate connection.

"We must protect our precious bodily fluids."

Re:Fear Fear Fear

1) This has nothing to do with IP rights.

2) It's a concern when you consider the potential effects of this kind of infiltration. Buying directly from Cisco, in no way, protects you from this problem. The hardware is still made overseas in some factory by a bunch of people who may not like the US very much (which is true of 99% of the planet right now).

Apparently you lack the imagination to see how ugly this can get. Fortunately DARPA isn't run by you.

/Light Bulb Flashes Overhead

So that's why my crappy Linksys wifi access points have to be rebooted every week or so. Damn commies!!!

You reap what you sow

CIA slipped bugs to Soviets [msn.com]

Technical details of malicious hardware

For those of you who are interested, you can find more technical details of how we designed and implemented malicious hardware from here [uiuc.edu]

-- computer scientists from University of Illinois

it is important to carefully inspected new gear.

if your new rack mount routers and switches say "crisco" on the front you may have a problem.

How many back doors? Who has the keys?

The question is not whether Cisco routers have back doors. That has to be assumed. If I was running NSA over the last several decades, I would have my people deep inside every communication equipment manufacturer. The manufacturers management might not even know about it.

The NSA surely has arranged to have one or more back doors designed into virtually every kind of communications switch. The only Cisco employees who would know about them would be the NSA people who work inside Cisco, and some regular Cisco employees who have been cleared. If this has not been done, the NSA senior managers should be fired or jailed.

The real questions are: How many back doors are there? and who has the keys? The (assumed) NSA back door might not be the only one. There is a possibility that the Chinese or Indian chip-fab or software contractors have also installed back doors for their own governments.

With billion-gate machines, a few thousand extra gates would be hard to see. If the extra logic looks like instruction-cache, but just has a little extra code, it would be almost impossible.

Re:

I merely skimmed one article which said that Cisco examined the routers and found no backdoors. The Ministry of Peace is probably just trying to test the sneakiness of their own snooping electronics in the name of "national security". The trojans which are

Re:

More like any company that outsources and doesn't perform internal quality control of what they are reselling should be made criminal in this instance of reselling to governmental agencies. Buy a Cisco, throw it in a private LAN sandbox, fire up Wireshark.