San Fran Hunts For Mystery Device On City Network 821
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
The story keeps changing. (Score:5, Insightful)
From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.
Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.
MAC address ... switch port ... it should be easy.
Re:The story keeps changing. (Score:4, Insightful)
2. Assuming that they have wireless on their network, there's no way to find wireless devices, since they can be put inside of locked buildings. Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.
Re:The story keeps changing. (Score:5, Insightful)
And exactly how would superman find it? Xray vision? How would he then know he found it?
Siding with the network guy (Score:5, Insightful)
Man, the more I read about this story, the more inclined I am to believe the network admin.
He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.
I've Changed my mind. (Score:2, Insightful)
When I first heard what the rogue-SF-admin had done, I was very negative on his actions.
Now, that once again, and now at least for the third time, I hear of absolute stupidity and ineptness on the group at sf, I am certain the so called rogue was right on the ball from the beginning.
Re:The story keeps changing. (Score:5, Insightful)
Re:The story keeps changing. (Score:3, Insightful)
2) It's a freaking terminal server. How many wireless terminal servers have you seen?
Re:The story keeps changing. (Score:3, Insightful)
The other end of that wireless device plugs into a wire, which has a MAC and then runs to a switch port.
The City of SF is undermining its case! (Score:4, Insightful)
not necessarily wrong... (Score:5, Insightful)
your employer's passwords are NOT yours, no matter how stupid you think your boss is.
Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?
Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.
Re:I've Changed my mind. (Score:2, Insightful)
Oh yeah, let's give him a break. Oops, he's been by hit by a bus. Where's his disaster recovery plan? That's right, there isn't one. He fscked his employer with his trumped up little admin attitude. Like most admins, he's on a power trip because he has root access on a network. The shit should have been fired, then sued him into oblivion for illegally locking up infrastructure that doesn't belong to him. Give him a few months jail time to top it off, he'll never get a decent job again.
Re:to quote bash.org... (Score:5, Insightful)
The admin might not be stupid he might be an ass
1) He placed a rouge device (his personal property) on the SF network
2) He set all the network devices on the network to lose all info on a reboot
3) He will hand over the passwords (after jail) to all the devices except the rogue
You can make equipment hard to find ( mac masquerading comes to mind )... I'm only adequate in terms of networking but I am pretty sure someone who is really good can play a mean game of hide and seek. Who knows *what* he was doing with that device? and were I the network admin I would have to *on principle alone* rebuild everything after this guy left..
Mod Parent Up (Score:5, Insightful)
I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.
No documentation?
No change control?
No diagrams?
What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.
I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.
Re:MAC search (Score:5, Insightful)
I learned early on, that most people don't see the difference between a $12 hour high school geek and a $75 hr network administrator. All most people see is that both do roughly the same job and there is $63 hour difference.
Most of the time, the $12 hr guy is doing most of the same work as the $75 hour guy. The big difference is when crap like this comes up, the $12 hour guy can spend years trying to figure out what the $75 hr guy can figure out in 5 minutes.
Even when the $12 hr guy screws up, the response is "But he was cheaper". It is cheaper to keep a $12 hr guy trying to keep crapware off a computer, rather than a $75 hour guy who doesn't allow crapware in the first place.
The point I'm making, is that a $75 hr guy is worth it, but only to people where time has real value. People who place no value on TIME, don't care about anything other than $ per HR
Re:You're an 1D10T (Score:5, Insightful)
I wish I had mod point for you.
Chances are that internal policies prevent the use of "hacker" tools to secure the network.
Again, the PHBs are idiots!
Simple co-dependency (Score:3, Insightful)
If you find that you are "holding the place together", IT-wise, you are likely part of the co-dependency and are part of the problem.
IT and the other management have both agreed to ignore each other, literally or otherwise, allowing each (and the individual personalities) to do things "their way"; damn the best practices, good management, logical, financial, or even legal issues.
Except when things go wrong.
Like a breakup, they can get ugly. And, as the IT guy, you will always lose for it is not your Business, but theirs. You are simply hired help.
Re:The story keeps changing. (Score:5, Insightful)
If Superman had any IT skills, he'd perform a traceroute to determine the devices gateway. Once the gateway was determined, block the mac address from accessing the network. If the admin of that device is worth his salt, he'll change the mac address and continue. They could then specifically enable allowed devices and forbid all others.
Forget finding it, make the network inaccessible.
City of SF Admins, if this proves to be your resolution, you owe me $150 for 1 hour of my time. Sorry, I do not bill in lower increments.
Re:to quote bash.org... (Score:5, Insightful)
I wonder if this one is just a complete misunderstanding. One article says that they were set to lose configuration files on "reset". That's pretty typical -- if you have some device you don't have the password to, you can do a full factory reset and get it back to the default password, but that also wipes the configuration files. He might have told his incompetent bosses that, and they thought he meant they'd lose the files on a reboot instead.
Anyway, if this guy is what they're making him out to be, they need to completely wipe and reconfigure the network anyway; it's the only way to be sure he didn't leave a few presents for them.
Re:Simple: (Score:1, Insightful)
Re:not necessarily wrong... (Score:5, Insightful)
Agreed.
If a boss I don't entirely trust demanded my password, I'd offer to upgrade his account to the same privileges at mine, but he'd NOT get MY password.
The reason is that if he does something stupid that will show up in logfiles, he can damn well do it on his account and get logged doing so ;-)
Re:Simple co-dependency (Score:2, Insightful)
When you're fighting fires, you're failing.
Re:Mod Parent Up (Score:5, Insightful)
What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.
Well, every Stalin needs his Trotsky!
Re:Mod Parent Up (Score:5, Insightful)
Admin code of ethics. (Score:5, Insightful)
What would you think of a doctor who, because some exec somewhere decided he should, pushed the WRONG medication / procedure to you?
Where does your ethical responsibility end and the boss's desires begin?
To me there isn't even a question. Fire me. Go ahead. I will get another job.
Re:You're an 1D10T (Score:5, Insightful)
Yes, both of those are true (Mac, Ping). Even NMAP responses can be spoofed. However the likelihood of all three being done is not likely. However NMAP will reveal a used IP, and a mac table somewhere will identify what port it is hanging on. Packets have to be routed to it somehow.
And I agree with your last point. I'm a Libertarian. ;)
Re:The story keeps changing. (Score:4, Insightful)
That and setting up the routers so they lose their configuration on reset. Even if your boss is an idiot, you get your concerns on the record and a direct instruction on the record and then do what you're fucking well told.
Re:Siding with the network guy (Score:3, Insightful)
The part that he seem right on is that his management was so inept, that they could not be trusted to touch anything.
I have not seen anything to indicate he "built a house of cards".
It looks more like he built a palace and put a secure wall around it so the local hoodlums (his management) could not vandalize it.
I think he got a little too attached to his work and cannot let go. (obsessive compulsive?)
Though his actions were extremely dumb, I am not sure I would agree with reprehensible... I might leave that word for describing his (non)management team.
Re:Mod Parent Up (Score:5, Insightful)
I took a gig recovering documentation and re-establishing procedures for a great admin who died as well. He really did great docs, but no one had ever used them, and they couldn't figure out the 'copy file piopoiop.dfj to the \asic\wer\2344\sdf.msdfn folder' sort of directions.
And the crew there immediately set to removing, replacing, and destroying all of his systems. He was a Novell hardliner (so was I), and when he was gone, his boss succumbed and the Windows bigots prevailed. Much taxpayer money was spent replacing perfectly functional systems. Mind you their clients were still running Novell, so there was some disconnect when they would get a request for support and start saying 'you have to upgrade (ha!) to Windows'. Their clients, for reasons best left undisclosed, could not upgrade. Both physically impossible and logistically impractical. Start with being 60-1600 meters below the ocean surface, and it only gets more difficult from there.
I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.
And yes, finding a device is not impossible. Finding the connection to the network is the obvious first step. After that, well, kill it.
Unless it's hiding. That would be unfortunate.
ps- This guy, by many accounts, was brilliant. And a little off the wall. Goes together.
Re:The story keeps changing. (Score:1, Insightful)
I would agree, except that his jackassery is what got him thrown into jail in the first place. I don't care how seriously he takes his job or how incompetent his supervisors are, no admin has the right to withhold such important information from his or her employer. If it bothered him so much to put that information into the hands of morons then he should have immediately walked off the job after handing it over. It's no longer his problem after that.
Re:Siding with the network guy (Score:2, Insightful)
but he may have been technically right.
The best kind of right!
Re:The story keeps changing. (Score:3, Insightful)
What makes you assume it's a wireless device?
The article doesn't say anything like that.
It's probably wired into the network, stuffed in a closet or a ceiling somewhere. Perhaps it does have another interface, a wireless one -- but it could be a backdoor without that too. Or perhaps it's not a backdoor at all.
In any event, that they are trying to find it via legal means rather than network means does indeed suggest that they're incompetent. And even if they can't physically find it, they should be able to disable it easily enough.
Re:Mod Parent Up (Score:2, Insightful)
1. If you employ someone to look after your security, you don't put obstructions in his way when he does it.
2. If you employ someone to look after your security, be very very nice to him.
Re:not necessarily wrong... (Score:4, Insightful)
I'm confused, does any admin ever give up his own account password?
In my company we have a blanket policy, never give out passwords, ever... as admin I don't need someone else's password to get into their mailbox and retrieve information that's needed by another employee while the content owner is out of contact. Of course I always notify the mailbox owner that I had to go in as I have to have a specific reason.
Are there environments out there where you would be expected to give up your password? I can understand keeping a password database for service accounts which all admins should be able to access if they manage it but I can't imagine a scenario when I'd need someone else's password. Even if the thing is encrypted, I have the recovery key so again I don't need their password.
Re:I've Changed my mind. (Score:4, Insightful)
Oh yeah, let's give him a break. Oops, he's been by hit by a bus. Where's his disaster recovery plan? That's right, there isn't one.
My bet is, it's sitting right in the middle of his old desk blotter, in a fat manila folder marked "Disaster Recovery and Service Continuity Plans". These clowns would never find it there in a million years. The infamous missing passwords are probably in a letter-size envelope in the top left desk drawer, too.
Re:Mod Parent Up (Score:1, Insightful)
No documentation?
No change control?
No diagrams?
If you're the guy in charge of planning and setting it up, then don't all those things become your responsibility, too? Or at the very least, overseeing that they're done? So who's the idiot again?
Don't mod that "funny". (Score:5, Insightful)
It appears that the idiot "boss" is attempting to generate support for the claim that this guy is a "problem" by paying unreasonable amounts to "repair" the "damage" he did.
It's difficult to "prove" that a guy did millions of dollars of "damage" ... without a bill for millions of dollars of "repairs".
Any competent network admin could map out the network and document it for FAR less than the hundreds of thousands of dollars that is being thrown about.
Re:Malice and stupidity. (Score:3, Insightful)
There do appear to be a lot of morons involved in this scenario, and Childs was one of them. Basically what he said was "I am smarter than all of you, so I will do things my way, and trust me, you'll be better off."
Except they weren't, because he doesn't appear to be anywhere near as smart as he thinks he is. Even if he was smarter than the gaggle of incompetents he worked for.
Re:Mod Parent Up (Score:4, Insightful)
I don't get it. The thing's gotta have a mac address that can be found on a switch somewhere. That'll give you a port number and a patch cable to follow until it's found.
Nah, it's way more fun to blame the guy in prison.
Re:not necessarily wrong... (Score:1, Insightful)
From what I can tell, he was facing a firing, and the password he refused to give up is the password for devices on the network. I understand he is the admin and all, but they aren't his devices and its not his network. Also, anything you do with company resources, e-mail, instant messaging, is property of your employer.
Re:What is this ``terminal server'' thing anyway? (Score:2, Insightful)
I personally don't follow the confusion over what this box is. They indicate it has "router like" login - if they use Cisco's, it's most likely an old cisco terminal server plugged in somewhere. If they can reach it on the network, I'm having a hard time understanding why they can't narrow down where it is. I'm guessing they don't physically label their hardware? What?! I mean, if you can traceroute to it, you can get a MAC address which will give you the device mfg. From there it's a matter of following the cables form the last hop surely to likely boxes. What the hell am I missing here?
Perhaps the article is overly simplistic in its description. Perhaps they've done all this and still can't find it. The MAC address has been changed or tracing 900 cables is taking them a while or something. But I still wouldn't be talking to the press admitting my own departments incompetence. I mean sheesh!
Not always (Score:4, Insightful)
When users ask for Admin privilages, they should be told to go fsck themselves. No matter who they are.
I'm a software developer. For the first few weeks working here IT wouldn't give me admin rights on my own box. I couldn't install software.
So I sat here and did nothing. Not because that's what I wanted. But because that's all I could do, until they gave me permissions on my machine.
Generally speaking, you're right. Most people in a business should be locked down. But not everyone. Depends on the person - depends on the work they're doing.
Re:Malice and stupidity. (Score:4, Insightful)
Big assumption.
They probably deleted all those "useless files" on the fileserver when they fired him.
And the "terminal server" is probably his iPhone...
Re:Simple: (Score:5, Insightful)
Re:Siding with the network guy (Score:4, Insightful)
Then you've never worked for the kind of clueless idiots this guy was working for. Supervisors do NOT need access. Any competent manager knows that's the case. What's needed is more than one competent individual to have access, with backup keys kept in sealed envelopes that are kept in a safe with only logged access to it in case both are hit by a bus on the same day.
BTW, did you miss the part of the case where for _years_ the admin in question begged, _BEGGED_ for someone else who was competent to be hired so he wasn't a single point of failure? That he continually pointed out that there was no DR plan whatsoever?
Nope, this guy made a serious error in judgment in not making sure that the mayor's office had the access information ahead of time. His supervisors are clearly incapable of administering that network and shouldn't be let anywhere near a console.
Re:Simple: (Score:4, Insightful)
Oh man, that is so hilarious. I love this part especially:
I cannot find any information in my MCSE bootcamp journal on how to handle this
Just more proof that MCSE certification is completely useless other than for getting a job. :)
Re:Malice and stupidity. (Score:4, Insightful)
I disagree.
It isn't that simple; it seems that there is waaaaay more to the story that some ego tripping sysadmin.
Everytime another piece of the story or fact about what happened comes out it seems to vindicate Mr. Childs to some degree (not that his judgement was flawless in how this was handled, but still).
Is he still locked up? If so it's a travesty.
It seems like those who are trying to have him tarred and feathered constantly want to make it look like he's some super-e-terrorist who was holding the entire city for ransom and has dealt an economic blow from which the city will never recover.
I am not saying everything he did was right, or that he committed no wrongs here; but I think it's pretty obvious that this was viewed as a pissing match by those in the city who wanted him to hand over that information and they have gone to great lengths to make it look like something much more malicious than it was in the press.
He may have had very good reason to protect it; (I mean aside from the fact that it appears as though those who wanted him to hand it over were incompetent) - because I don't think anyone would put their own ass on the line for jailtime and the loss of their job unless there was something else going on. I am not saying I know this to be true, just that that is how it appears to me based on the available information.
At this point I view anything coming from the anti-Child's side of this issue with a healthy does of skepticism and try to read through the sensationalization. Something has always stunk about this situation.
Re:You're an 1D10T (Score:3, Insightful)
If you have SNMP and a Winders PC;
log.txt (list of your switches / routers)
192.168.1.1
192.168.1.2
file1.bat /f "tokens=1,2,3,4,5,6,7,8,9,10" %%i in (log.txt) do call distcmd.bat %%i %%j %%k %%l %%m %%n %%o %%p %%q %%r
for
distcmd.bat (change public to be your snmp community snmputil is from net-snmp-5.4.1-3.win32.exe / free) .1.3.6.1.2.1.17.4.3.1.1 >> %1.MAC.log .1.3.6.1.2.1.17.4.3.1.2 >> %1.PORT.log
snmputil walk %1 public
snmputil walk %1 public
Now you have a list of every port and ever MAC that runs through that port (don't forget about uplink ports having all MACs listed to them.)
In the file you'll have to do some snmp decimal to hex conversions; .17.4.3.1.1.0.23.164.215.49.153 .0.23.164.215.49.153 Put those number in calc and convert each to a hex and you get
MAC is 00:17:a4:d7:31:99
SNMP walking gives a result of which is
the same MAC, but converted to SNMP notation this becomes
Part of this is the MAC address in decimal
the mac address in hex.
So in the output of these batch files .17.4.3.1.2.0.64.140.109.101.123
Variable =
Value = Integer32 24
Means that on port 24 there is MAC address 00:40:8c:6d:65:7b which is the .0.64.140.109.101.123 converted to Hex through Calc.
Anyway, ping the device, arp -a and locate the MAC address, dump the above against all your routers/switches, convert your MAC address you are searching for to SNMP decimal, and search the .txt files for a switch with a port where that is the only MAC on the port and you've found the device (or a hub between the device.)
Re:Mod Parent Up (Score:4, Insightful)
Re:Simple: (Score:5, Insightful)
Must Consult Someone Experienced
Minesweeper Consultant and Solitaire Expert
Re:Simple: (Score:4, Insightful)
Not really. A network admin should be able to track down the thing, but it will take a lot of work to scan network logs. From the network standpoint, it doesn't matter if the gateway is running on a PC, or running on a VM inside a PC... the network traffic looks the same.
Re:Mod Parent Up (Score:4, Insightful)
I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.
From the article...
And there you have it folks, a million-dollar employee; over-worked and under-appreciated by a management too incompetent to understand the issues the guy dealt with much less manage him and his work effectively. Sadly, it's not a very uncommon story.
One of the fun bug-a-boos that show up in these stories is the cost of damage an intruder (or in this case, rogue employee) "causes" the target. I've been on the inside of a number of US Government incidents and seen the cost estimate damages. To someone on the outside, they seem pretty insane. The question that the public often asks is something like "how can changing one password cause so much damage?" But the numbers I've seen are pretty much on target (plus or minus some variance for estimates) - they represent real expenses associated with work to properly ensure the systems are truely owned by their rightful owners again. And they cover resources (i.e. hard drives) lost to criminal investigative bodies / evidence lockers. But the real gotcha to these things is that these expenses either should have been spent as part of the normal management cycle without an attached incident or, even better, could have been a fraction of the eventual cost if the resources were spent to improve the environment or hire proper talent in the first place.
The new WarLords (Score:5, Insightful)
I'm reminded of a conversation I had some 25 years ago with a co-worker IBM mainframe technician. IBM management was incensed that uneducated morons turning screwdrivers could make 70k a year. Back then as much as what they were paying top MBA stuff shirt types. They were on a mission to get salary levels down to "reality" paying these screwdriver wielding monkeys what they were (in their minds) really worth.
Attitudes have changed but not a lot. 93% of companies that loose their data center for 10 days or more due to a disaster filed for bankruptcy within one year. 50% filed bankruptcy immediately (National Archives & Records Administration in Washington) [google.com]. One can't say the same thing about those over paid MBAs.
It may be awhile before IT matures into a "profession" like doctor or lawyer however I personally believe we're holding the keys. The world can't function now without us.
-[d]-
Re:Siding with the network guy (Score:4, Insightful)
His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.
Thou shalt not expose the government's incompetence.
Re:No power outage in the Terry Childs case? (Score:2, Insightful)
Re:Mod Parent Up (Score:5, Insightful)
Could be an IBM 3174 like device too, running SNA. Fact is, the article and and court filings aren't clarifying any of this and leave the door open for mass amounts of conjecture and sensationalizing, both in the media and on Slashdot. Which, of course, is exactly what everyone is doing...
Re:The story keeps changing. (Score:3, Insightful)
Routers will usually lose the configuration when you do a reset (as opposed to power-cycling the device), and I'm not surprised that some incompetent superior didn't know the difference between a reset and a reboot.
Re:Simple: (Score:5, Insightful)
and your not sniffing the traffic to these boxes why?
Re:You're an 1D10T (Score:5, Insightful)
>But everyone who supports more government ought to take a look at the incompetence here.
Im one of those crazies who doesnt support more or less government. Just better government.
Uh? Not that easy. (Score:3, Insightful)
Sometimes you inherit the fires. Oftimes they may be created by other people, and frankly, without enough co-operation by management (either dealing with consistent firestarters or by hiring supporting staff), you cannot make yourself redundant.
There's only so much time in the day for a given person to do a given set of tasks.
Re:Simple: (Score:3, Insightful)
If you're really lazy, you could also unplug their network cables and see what breaks... :P
Re:Simple: (Score:5, Insightful)
1. Boot from floppy, optical media, network, etc. /mnt -o rw /mnt
2. mount [/dev/sda1|/dev/hda1]
3. chroot
4. passwd root [password]
5. ??????
6. PROFIT!
No yanking to do. A reboot and 5 minutes of down time. Bang. Dead. Done.
Re:Simple: (Score:5, Insightful)
It may also mock you with nonexistent cake.
Re:Simple: (Score:5, Insightful)
I have a huge admiration for your honesty. You are an exceptional person.
Re:Siding with the network guy (Score:3, Insightful)
The problem that I think you're overlooking is that this guy was stuck in a siege mentality. For years he'd been asking for help and his management chose to ignore him and apparently, in some cases, actively blocked his requests from going further up the chain to someone who'd listen.
Now consider the fact that he was working 50-60 hour weeks and was on call 24/7/365. I've been there, done that, hated the T shirt. Believe me, after a few years of that (and he had apparently been suffering with those kinds of working conditions for 5 or 6), you'd be a little nuts, too.
Did he make an error in judgment? Yes. Was it egregious and irresponsible? Not under the circumstances. In my view, quite the opposite. Was it criminal? Maybe. But since when is the law necessarily about common sense? :(
Re:Simple: (Score:4, Insightful)
Holy crap, +5 insightful? I like my karma as much as anyone else, so no complaints, but... huh?