San Fran Hunts For Mystery Device On City Network 821
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
MAC search (Score:5, Informative)
Um, do what any network admin does with a rouge device. Search out what port its MAC address is connected to and then start tracing the cable?
I'm fairly certain most all current managed switches allow for this. Even with unmanaged ones you can hunt down which unmanaged switch it is connected to and snoop from there.
Re:MAC search (Score:3, Informative)
It's probably hidden in a wall cavity somewhere, a bit like that Netware server in the news a few years back.
Please - It's San Francisco or simply "The City" (Score:2, Informative)
Tourists...
Re:The story keeps changing. (Score:5, Informative)
Re:MAC search (Score:5, Informative)
Apparently this was why he refused to give out the admin passwords - he thought, and so far, it appears that he is correct, that they are all morons.
FoxHunt (Score:5, Informative)
2> It's easy to find wireless devices... I've personally been doing it since the 1980's.. it's called a fox hunt [wikipedia.org] here in the Chicago area. We used to get 1 minute of transmission every 5... with WiFi you can just ping the dang thing... how easy is that?
--Mike--
You're an 1D10T (Score:5, Informative)
1) They were firing the guy, so he was no longer in the employ of the city, so his boss, was no longer his boss.
2) You don't know what you're talking about. Every IP address on the network should be known. Either through DHCP or static IP address map. A ping sweep should reveal any IP address in use, that shouldn't be. From the ping sweep, one can arp the unknown IPs to get a MAC address, and do a lookup on the Manufacturer code to know what KIND of device the MAC could be. one could use NMAP to try to discover type of device as well. Then you start going to every port on every switch with rogue IPs hanging off it, and manually looking at what is attached at the other end.
As for wireless access points, if you don't have control over them, you pull the freakin plug. Unsecured Access points and open access points should be VLANed off from administrative networked, including not allowing VPN tunnels from unsecured and open wireless access point.
If the boss allows crap like that on the network, he is an idiot, and shouldn't have the Passwords and access codes to anything.
Re:Please - It's San Francisco or simply "The City (Score:2, Informative)
Re:You're an 1D10T (Score:5, Informative)
cisco command "show cdp neighbors" (Score:2, Informative)
No power outage in the Terry Childs case? (Score:5, Informative)
http://weblog.infoworld.com/venezia/archives/018376.html [infoworld.com]
An insider claims that the power outage that Terry Childs was accused of using to sabotage the San Francisco network was not a planned outage.
TAGS: Problems, San Francisco's FiberWAN, Terry Childs
If you've been following the Terry Childs case to any degree, you probably know that one of the key allegations keeping him in prison on $5 million bail is that he had willfully planned to cause the network to fail during a planned power outage at the DTIS One Market Plaza Datacenter on July 19th. According to credible information I've recently received, that power outage was only going to affect the cubes and offices in that building, but not the datacenter itself.
Thus, there never was a plan to power down the network core. Thus, there's no way that Childs could have tried to engineer the failure of the network during this planned power outage, since the network core would not have lost power.
[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]
The evidence supporting this claim comes from someone certainly in a position to know: Ramon Pabros, the DTIS Datacenter Supervisor himself. Pabros has been employed by San Francisco's DTIS for a surprising 41 years. He's been the Datacenter Supervisor since 1984. He's been running datacenters for the City of San Francisco since Ronald Reagan's first term, the introduction of the Macintosh, and the second season of The A-Team. It's probably safe to say that he knows what he's doing.
According to my source, he will testify to the fact that he discussed the power outage with Childs several weeks before the outage, and at least 10 days before Childs' arrest. He will also state that Childs specifically asked for confirmation that the datacenter itself would not be affected, and was reassured that it would not lose power.
With this statement, the City's allegations that Childs planned to cause the failure of the FiberWAN basically collapse.
Now, I'm admittedly a stranger to San Francisco politics, and am certainly not a lawyer, but if the DA was going to make these accusations against Childs, shouldn't they have talked to Pabros? If the OMP Datacenter was not going to lose power on that date, then this charge against Childs is essentially the same as charging someone with planning to burgle a store that doesn't exist.
But then again, this is the same DA's office that placed valid group usernames and passwords into the public record, and an IT department that ran public, unprotected websites containing internal emails, core network details, as well as usernames and passwords.
I suppose I really shouldn't be surprised at all.
UPDATE: It appears that Pabros has just announced he will be retiring, effective next Wednesday. I can't help but wonder if one event has anything to do with the other. I do know that there have been a number of odd layoffs from San Francisco's DTIS in the past two weeks.
Posted by Paul Venezia on September 8, 2008 08:48 AM
Re:The story keeps changing. (Score:5, Informative)
Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.
By the time his boss thought to ask for the password(s), he had already been fired. Any obligation he had to his boss had disappeared. The same goes for documentation and written procedures - I'm not going to document anything after I've been sacked. In this case the guy had been arguing for written procedures to be put in place, but no one in authority would sign them off as any failures would then be their ultimate responsibility. It should be the managers that are taking flack for this, as so often with IT cock ups.
Re:MAC search (Score:5, Informative)
How, then, can they use the management functions of the equipment if they can't get to it?
Terry Childs provided the passwords [wired.com] to the mayor on July 22. The city "...[was] able to regain complete control of the network," according to the deputy director of the Department of Technology Information Services.
Re:The story keeps changing. (Score:5, Informative)
and I do development on some software [airwave.com] that will use RF data from your existing wireless access points to triangulate and display the physical location of every user and device on your network!
So you can call me, uh, Jerry Siegel, I guess? :| that's not as impressive...
Re:The story keeps changing. (Score:3, Informative)
No joke. You'll pay $300+/hr for a top guy from a place like IBM Global Services or HP's technical consulting group.
Re:to quote bash.org... (Score:1, Informative)
Because it's rouge!
Re:Simple: (Score:5, Informative)
An EMP disrupts electronics by inducing massive currents in the thin circuitry of the circuit boards and integrated chips. They're permanently burned. They won't power-cycle, they'll just fry.
Naw... if you really want to power-cycle it, just disrupt the electrical service to the entire city. You'd probably have to leave it off for a fair length of time, though, in case the device was on UPS.
Re:FoxHunt (Score:5, Informative)
it was covered on /.
http://www.techweb.com/wire/story/TWB20010409S0012 [techweb.com]
Re:MAC search (Score:3, Informative)
I'm sure the rocket scientist in GP post was just using the switched ports (4, on the typical Linksys home router product) on the LAN side, not realizing that the router's firmware was still active even if it was just being used as a dumb switch.
Hell, if you've gone from wired to wireless at your house, and you happen to have this old router laying around now, and you need a few more 10/100ports ... whadday gonna do, run out and buy a new switch, or grab the one you've already got? And who's gonna think of disabling the DHCP service at first, if it was configured running months ago when you decommisioned it?
A perfectly forseeable screw-up, and another good example why lusers aren't allowed to plug their crap into the network.
In the immortal words of some faux-German sage: "Das Network is nicht fur gefingerpoken und mittengrabben."
Re:The story keeps changing. (Score:3, Informative)
regarding point 2:
Trivialy easy to determine if it's a wireless device. TURN OFF THE WIRELESS NETWORK. If the device goes away, it's wireless. Then simply change the security configuration on the network, and problem solved. The offending device is no longer on the network, and its physical location is irrelevant.
Elsewise, if it doesn't go away, it's a wired device, and normal network investigation should work just fine.
I should have R TFA... (Score:3, Informative)
Childs, who has worked for the city for five years but faced firing for alleged poor performance...
Illuminating, but mostly in that it shows all parties in a very dim kind of light. Under the circumstances, I would have hesitated to employ the guy in this capacity anyway...
Re:FoxHunt (Score:3, Informative)
I've heard the same story, only with coax cable (yes, I've been around that long...).
If their network is CAT-5 based on smart switches (HP, Cisco or any somewhat decent network vendor, heck even Linksys and Netgear have smart switches) they can find it quite simply by running a simple command on the switches, I even have a perl script that will do it for me.
Of course, it wouldn't surprise me if they still use 'dumb' switches or if the consultants simply said among them: let's get these stupid , just sit around and rack up the charges.
Re:No power outage in the Terry Childs case? (Score:4, Informative)
Managed or Unmanaged (Score:3, Informative)
Well, if it's on a managed network then the IP needs to be mapped to a MAC address (and port on a switch) and the port turned off. Once that's done, tracing the cable to a physical port should take no more than a day.
If it's on an unmanaged switch things get a little more annoying, but you should still be able to track an IP and MAC address to the switch using any open source network tool like WireShark. Find the switch. Pull the cable out of the port, or if you're feeling really adventurous you could bring a replacement switch in and start playing "Is that it?" until you find the bugger.
Must be a slow day for something this ho-hum to make it on /.
Re:not necessarily wrong... (Score:4, Informative)
Re:I've Changed my mind. (Score:4, Informative)
Who modded this insightful? Part of the reason he was getting canned was because he was PUSHING for the sort of documentation and recovery plans you're snarling about. None of the PHBs wanted to put their names on it because if they came up short, it would be their asses on it.
Re:The story keeps changing. (Score:2, Informative)
No, city property belongs to the city and not to the admin. Failing to allow the city access to or control of its own property is a crime known as "conversion." This guy should be thinking about his personality disorders between prison showers.
Re:You're an 1D10T (Score:3, Informative)
Packets might have nothing to do with a terminal server. He could be dialed into it using a modem and controlling the console port of an otherwise legitimate Unixish server.
Re:Simple: (Score:2, Informative)
Here ya go (Score:4, Informative)
UNC in 2001 [techweb.com]
How about this approach? (Score:4, Informative)
I used this once to track down which server room a system was located in and while it's not perfect for all occasions, it might help.
Ok, first if you can get an IP for the device, perform a traceroute from 3 or 4 separate sites. Identify it's Gateway if possible, also if find see if you can determine from the traceroutes if it has a common parent node that it's traffic is going through.
Once you've found the most common system talking to it, go to that system and perform ping tests to other systems where you know their physical location in proximity to the system your at, and are only 1 hop away (if possible). The key here is to make sure that all of your samples share as much of the same route as possible to minimize signal noise in your data set you're going to build.
See if you can develop a correlation between ping times and amount of network cable to your sample set. Compare that data to the ping times on your mystery device and you *potentially* have a physical range now in hand to perform your search.
I'll be the first to admit that this approach has limited success based on how your infrastructure is built, but it might help.
Re:How about this approach? (Score:4, Informative)
OR, one could do a traceroute to the IP and check the ARP tables of that gateway.
The problem I suspect is that like most governments, they're still using a mix of very old technology. This thing might not even be running IP. Of course, one then presumes to ask "How did they know it's there in the first place."
Re:Mod Parent Up (Score:3, Informative)
Re:Admin code of ethics. (Score:3, Informative)
Which, considering the rest of the FUD around this case, doesn't surprise me.
Re:Simple: (Score:3, Informative)
It shouldn't take any competent netadm more than 5 minutes to track down any device to a specific port on a switch. There are no logs to look at. What do you think is logged that you'd want to look at to track it? Seriously, it's incredibly simple to do. The thing has an IP and for that IP to be useable on the network it must be in the RIB (read: route table). With less than a minute's work a netadm should be able to track down that route to the router that's originating it. I don't care how big your network it. It should take less than a minute. Once you've found the router originating the route you've almost certainly found the router with an L3 interface in the same broadcast domain as the target device (the router could also be redistributing a static route in which case the static route would point you to the device in possession of said prefix, or a trail of bread crumbs of multiple static routes that will eventually lead you to the device using that prefix). If the router is part of that broadcast domain then it will have an entry in the ARP table for the target IP and will give you the device's MAC. From that router's config the netadm can determine where all that broadcast domain is accessible. Ie, what L2 switches downstream of the router have that VLAN on it. The netadm can examine the CAM table (SAT in Cabletron-speak, bridge forward table in generic terms) to figure out which interface the target's MAC is associated with. That will point him to the correct downstream switch. The netadm will do the same thing on that switch to track the target device their the broadcast domain until he find the one access interface that the target device is connected to. Once he finds that interface he visits that wiring closet and tracks the cable down manually to the target device.
Really, it's much easier than it sounds. Once you've done this once or twice it will become second nature. This should not take a competent netadm more than 5 minutes. I don't care how big the network is. This isn't rocket science. The City of San Francisco is just trying to make their case sound worse than it really is. It would take a truly incompetent IT department to not be able to find that device. I would say that it was impossible to be that incompetent but I'm sure someone would try to prove me wrong.