San Fran Hunts For Mystery Device On City Network 821
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
Re:MAC search (Score:3, Interesting)
Exactly, hell I can sit down with my laptop and tell you what switch it's connected to in 20 minutes. Bet you $50.00 the community strings on all their network gear is still set to public and private :)
Are the IT people they hire completely dysfunctional? Or do they do what most cities do and not actually hire IT people or networking admins because they command a real salary instead of the $12.00 an hour that someone handy with computers get's...
This is a job for nmap (Score:5, Interesting)
Hey! Fyodor! They need your number! [insecure.org]
Fyodor spent much of this summer scanning tens of millions of IPs on the Internet (plus collecting data contributed by some enterprises) to determine the most commonly open ports. Nmap now uses that empirical data to scan more effectively.
Zenmap Topology and Aggregation features were added, as discussed in the next news item.
Hundreds of OS detection signatures were added, bringing the total to 1,503.
Seven new Nmap Scripting Engine (NSE) scripts were added. These automate routing AS number lookups, "Kaminsky" DNS bug vulnerability checking, brute force POP3 authentication cracking, SNMP querying and brute forcing, and whois lookups against target IP space. Many valuable libraries were added as well.
Many performance improvements and bug fixes were implemented. In particular, Nmap now works again on Windows 2000.
With just nmap, my old buddies at Farm9 could have sussed this out in a few hours. I think they are still around - as Red Siren / Getronics. [getronics.com]
Ahh. I miss running netcat at 3 AM!
Just remember. (Score:5, Interesting)
You think they've learned anything about the gear since then? No wonder they're having problems.
Malice and stupidity. (Score:5, Interesting)
Why is Slashdot linking to stories that paint the network administrator as a bad guy when he's so obviously surrounded by morons? These are the same people who published all of their user names and passwords [slashdot.org]. That puts the cost of this "hijacking" into perspective. The cost of trusting their employee with the powers required to do the job was zero.
Re:MAC search (Score:3, Interesting)
I worked for a company where they cheaped out on the switch infrastructure and bought low-end Dell switches for the entire network. The kind that don't let you see the MAC address table.
Some guy decided to bring in his Linksys router from home so he could use his laptop and his desktop at the same time (instead of, you know, asking IT to add a second port at his desk). Problem was he left DHCP running on the thing, which obviously led to some confusion. Took forever to find it.
Then again it sounds like the city of 'cisco bough nothing but Cisco gear, so who knows what's really going on here...
Re:Please - It's San Francisco or simply "The City (Score:3, Interesting)
Re:Simple: (Score:5, Interesting)
Re:The story keeps changing. (Score:4, Interesting)
Now, as regards passwords and what not, I would be inclined to agree - you've got no right as a professional to lock out the owner of the kit, from their stuff. However I'd also say escalating it higher because there's 'serious ethical implications' in some situations isn't unreasonable. Not that this necessarily relates to this particular case - I don't know the details, so I won't comment - I just wanted to point out that there are good and valid reasons not to comply with a demand like this from your direct 'boss'.
Re:The story keeps changing. (Score:5, Interesting)
Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.
My obligation to my employer (in this case the city of San Francisco) trumps my obligation to my PHB. If I think my PHB is a moron and is going to cause a shitload of damage to my employer then I think I could make a good case for refusing to give him the passwords.
Of course that's not where it would end.... I would have to explain to his boss what the problem was -- or go even further up the chain of command if he was also a moron.
Assuming that they have wireless on their network, there's no way to find wireless devices
Wireless devices still have MAC addresses. By tracing the MAC address you'd get a switch port. If that switch port has an AP plugged into it then you know it's a wireless device and probably know it's general location (the AP doesn't have limitless range).
there's no real way to find exactly where wireless devices are, as far as I know
Oh, there's a way [wikipedia.org].... it's just out of the reach of most of us.
Re:Sparcstation In The Wall (Score:4, Interesting)
Re:Siding with the network guy (Score:5, Interesting)
Well, the fact that they're contracting outside Cisco experts now suggests nobody else there was technically competent enough to manage the network.
The fact that the network stayed up and running without a hitch, while he was in jail and nobody else had access, suggests he did know what he was doing, and refusing to allow anyone to access the routers to make changes seems to work quite well to keep the system working.
The fact that his supervisors are moronic and useless is no small thing, either.
His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.
Hey, you're smarter than J. Michael Cook! (Score:2, Interesting)
In fact, you just proved you are smarter than all of these guys [comcast.com].
Oh, sorry, that wasn't much of a compliment, was it?
So that's a good point .. (Score:5, Interesting)
Who is actually the OWNER of the system? The boss? Isn't he employed by the same company as the sysadmin? Don't they both have an obligation to safeguard the OWNER'S property and interests? If the sysadmin refuses to hand over the password to sensitive equipment & systems to a (perceived) inept superior-- as long as that guy DOESN'T own the company-- isn't he actually performing his responsibility to the real owner? Which in this case would be the city, and the personification of the city would be the mayor-- and that's exactly who he DID give the passwords to. So it seems to me like he did precisely what he was supposed to do in terms of safeguarding the network and sensitive equipment. Of course he should probably be then fired for failing to keep backups, conops, continuity planning, etc. But that's a different matter.
Re:FoxHunt (Score:5, Interesting)
There is an old, probably apocryphal tale from the days of Novel Netware and IPX of the forgotten server. A loan machine runs headless with a quiet fan and no lights in a corner of a room. New remodeling puts the server behind sheet rock and there it sits walled up and running for years. One day a power spike causes a head crash and suddenly a national billing system dies. It takes a tech tracing a cat5 cable into a wall to find it.
Re:Admin code of ethics. (Score:3, Interesting)
Unfortunately IT professionals aren't in as much of a seller's market now as they were before. Getting another job isn't always as easy and beneficial as it used to be - and when you add in the new kids coming out of school looking for work, available IT positions can quickly become races to the bottom in terms of salary.
So as much as an admin would prefer to take the moral high ground, they also have to look out for number one. Everything is a trade-off nowadays, unfortunately.
Re:The story keeps changing. (Score:3, Interesting)
When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.
Honeypot ? (Score:2, Interesting)
It could even be a Honeypot...
Re:MAC search (Score:3, Interesting)
Not at all. I dealt with this very issue twice for the same organization. They bought wireless routers and wanted to use them like access points. They put port 1 on the network and placed a computer on port 2, never using the WAN port. This is better setup than using the WAN port because you can't as easily access the computers behind the WAN port. The problem was they wouldn't disable DHCP causing all sorts of issues. Twice I went in and explained that they MUST disable DHCP if they want to use the router in this fashion and last I heard they reset the routers again and were having the same issues. Of course, my name gets dragged in the mud because they think I'm the idiot.
Re:The story keeps changing. (Score:5, Interesting)
They could always do something crazy like track the MAC to a port and go trace the cable to find the device, I guess that wouldn't make such a good story though.
If they're using Cisco switches and it's linked via copper then they could probably work out where it is without leaving their seats, use the inbuilt tdr [cisco.com] to find out how long the cable is, then use the location of the switch and a bit of common sense to work out where the device is likely to be.
If it's a terminal server then it's not likely to be hanging off a 3km long fibre somewhere in a duct under the city. It'll be within serial cable distance of all the other kit, more than likely in their main computer room with some bloody great octal cables hanging out the back. I suspect it'd take someone clued up approx 5 minutes to identify it as it will look rather different to any of their other routers purely due to the cabling run to/from it.
The more I read about this "ebil admin" story the less I believe any of it.
Re:Simple: (Score:4, Interesting)
Reminds me of a guy I knew who used piezoelectric fire lighters (it's the one used in stoves) to test the watchdogs on circuits he built.
He fired it over the processor and the interference would be enough to disturb it (electrically isolated of couse, the spark would not go to the device, only the EM interference).
Re:Malice and stupidity. (Score:5, Interesting)
There do appear to be a lot of morons involved in this scenario, and Childs was one of them. Basically what he said was "I am smarter than all of you, so I will do things my way, and trust me, you'll be better off."
Either I have bad luck or I keep on finding people who think exactly that way. We have even had meetings where all agreed on a specific solution to the problem. Right after my boss say well we are going to do it this other way, we know better. Even if the other way was a better solution.
Some people have egos that are way too big fir their own good. I am not saying I am perfect. I use solution that I know work. If there are better ones please show me. I have no issue changing my way of doing things for a better one. I know a lot of people who will not change. Even when a better way is show to them.
Re:Simple: Local Incompetence in Play? (Score:2, Interesting)
"All they have to do is look for the small black box with a lone, onerous blinking red LED."
Not to be a grammar/word-choice "Nazi", but I think you meant "ominous".
But, after all this time, one might expect that the NSA would have been on top of this. Anytime a city government fails to locate rogue devices that could compromise local/state/federal/international investigations, the criminals and the undercover agents/officers, and witnesses, as well as payroll and other HR information, the FBI, NSA, and other agencies should take over that aspect where the locals prove incompetent.
Re:The story keeps changing. (Score:3, Interesting)
Disclaimer: I am a sys admin, but not for the municipality of San Francisco, so my ignorance of their network architecture might be masking something that makes this procedure non-trivial. For the life of me, I can't image what, however.
It could happen-harder to find that it seems (Score:3, Interesting)
In a big network I could see this happening. I know--computer rooms are supposed to be pristine with every wire perfectly aligned and in place with everything perfectly labeled and mapped--NOT! Most computer rooms I've been in, including my own, are somewhat less than ideal. They kind of grew with no plan. Need more space? Run a jumper. One of the Field Engineers who worked on one of our minis just laughed and said we weren't really that bad--you should see banks--they're the worst. In other words, poor housekeeping is widespread and tolerated. A typical terminal server could be 1RU or even a blade, or a box sitting loose on top of the rack where you can't see it. If I were really devious I would put a small terminal server in a bigger box. If this were intentionally hidden it could be in the ceiling hooked to a 128 port hub in the rafters itself and you'd never even know it. It's a bird's nest of Cat5 around a hub, all looking the same. I'll just bet it's a Class B network, so you've got a tremendous number of possibilities. And if you used virtual networks on Cisco hubs or did some bizarre subnets that simply confounds matters. I feel very confident that I could hide a box in my building that even the pros would have a hard time finding. Of course you could start turning off power until the device disappeared to try to pin down its location, but my guess is no one wants to do that just because someone lost a box. Too funny.
Re:Simple: (Score:5, Interesting)
Re:Mod Parent Up (Score:5, Interesting)
What it would have (if it is similar to how I use them, and yes I am a WAN specialist) is a phone-line for dial in access in case of emergencies.
See MRV's InReach [mrv.com] product line for more information.
...though it could have a MAC address on the network, just saying it doesn't have too, and if it is "mysterious" and / or put there maliciously, in all liklihood will not, or it will be spoofed to prevent detection.
What's the problem? (Score:4, Interesting)
It shouldn't be that difficult to find a piece of h/w on a network.
Interrogate the switches to find the IP/MAC address corresponding to the device you are trying to log on to. In the event that this Childs guy is deviously smart (i.e. patched the switch software to conceal a particular device) one can still use a stand-alone sniffer to trace packets through a system.
Its possible that the 'terminal server' might be virtual, just an app. running on some other piece of hardware that doesn't necessarily have "ACME Terminal Server" and a wining LED on the front. But tracing the network to that particular box isn't difficult (maybe time consuming).
If these people are really that dumb, I can understand why Childs kept them off the system. Reading some of the stories about him, it wouldn't surprise me if he left a bunch of 'dead ends', like phony terminal servers that nobody could find. Or wireless access points not plugged into anything but plastered inside a wall to drive security auditors nuts.
Comment removed (Score:2, Interesting)
Re:Simple co-dependency (Score:4, Interesting)
You may want to stop reading what the city says, and find out what really happened.
http://it.slashdot.org/comments.pl?sid=960957&cid=24963255 [slashdot.org]
Re:Simple: (Score:5, Interesting)
The real question, though, is this: If your alternate personality made the bomb, does your present consciousness have the subliminal knowledge of which wire defuses it?
Depends on when it was I guess.
Back in 2001 I did some emergency wiring work that had to be done in 72 hours at our shop.
Now, we are only there 10 weeks a year, so after the end of the 10 weeks it was forgotten about.
I was very sleep deprived and manic when I finished the job, and to this day I have NO idea how I did some of the connections I did. I just hope and pray it all keeps working. Some day some part of it will fail, and I'll have to re-do the entire building.
Note to self:
When sleep deprived, always work from the list, and write down what you did. One thing at a time, and document everything.
Re:Simple: (Score:5, Interesting)
Not at all uncommon. I've got 3 fucking servers in my system room that nobody knows what they hell they are for. The are all running 2.4 kenels so they are as old as the fucking hills. Nobody knows what the passwds are to get into them so I can't log in and find out what they do. And naturally the previous systems administrator that installed them didn't document shit.
The only thing that is known about them is they used to do something important just nobody remembers what it was. Management is to afraid that they might still be doing something important and won't let me yank them out to find out what they do. So while management sits there with their collective heads up their collective asses these three servers sit there taking up space in my racks on my network.
When these thing do finally fall over I hope they are doing something important.
Re:Mod Parent Up (Score:3, Interesting)
Wait, you mean blame it all on the guy who left (be it through death or a cushy new job) isn't standard practice everywhere?
I had to actually threaten legal action against a former employer who repeatedly claimed all the failures after I left were sabotage. Maybe its my fault for not grooming a successor, but there was some truth when I suggested my knowledge deserved higher pay.
More technical info on the device (Score:5, Interesting)
"From what I can see, it's a device running Cisco IOS that was accessed via telnet. I could generate an identical screenshot to the one entered into evidence in about five minutes using an elderly Cisco 2924-XL Ethernet switch -- a device that's certainly not a terminal server. It's completely unclear to me how they could have possibly come to the conclusion that this is a "terminal server" -- the evidence presented to the court certainly does not support that theory."
Venezia also uncovers additional technical errors in the prosecution's case, which appears to be unraveling [slashdot.org] with the recent news that the DTIS Datacenter Supervisor Ramon Pabros will testify on Childs' behalf [infoworld.com]. Since coming forward, Pabros has announced he will be retiring from the DTIS, effective Sept. 17. Coincidence?
VMware (Score:3, Interesting)
Someone loaded vmware server on their desktop that has an extra network card.
Re:Simple: (Score:1, Interesting)
I was going to say nmap it, find an old ssh/telnet/ftp exploit, nail it, then use a root escalation exploit to get root access so you can change the passwords :D
Need to make sure ftp or whatever isn't the critical remote service though...
Re:Simple: (Score:5, Interesting)
always work from the list, and write down what you did. One thing at a time, and document everything.
This seems sensible under all conditions. Being tired is no excuse for being sloppy.
I have a sleep disorder.
There are times when, for no real discernible reason, my brain decides that I will not be sleeping for a few days. Sometimes upwards of 100 hours.
When you have been awake for 4 days, (at least in my case) you get a serious case of "While I'm at it" syndrome.
Tasks that can not be completed in 10 minutes (or without getting up) are nigh impossible. I can still work, but I am extremely easily distracted and will often forget why I am in the room I was in.
Example: I went to the fridge to get some water, and decided that I should clean it while I was there, then decide to do the dishes since I threw stuff out of the fridge, then decide to do the laundry since I had no clean towels, and while I was in the basement doing the laundry I noticed that I needed to organize the basement and throw out old computer parts. Meanwhile, upstairs, my glass of water has long since evaporated, and the task I was doing before that is long forgotten.
Thus, when I get like that, I work from a list, and only what is on the list gets done, in the order it went on the list.
Re:Simple: (Score:3, Interesting)