Slashdot Log In
San Fran Hunts For Mystery Device On City Network
Posted by
CmdrTaco
on Thu Sep 11, 2008 09:53 AM
from the they-just-can't-get-a-break dept.
from the they-just-can't-get-a-break dept.
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
Related Stories
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
Simple: (Score:5, Funny)
Power cycle it with a city-wide EMP.
Re:Simple: (Score:5, Funny)
Parent
Re:Simple: (Score:5, Interesting)
Parent
Re:Simple: (Score:5, Funny)
Parent
Re:Simple: (Score:5, Funny)
Don't forget the obligatory RED and BLUE wires. Every small black box with lone onerous blinking red LED MUST have red and blue wires. Its a rule.
Parent
Re:Onerous (Score:5, Funny)
it's a very big LED.
Parent
The story keeps changing. (Score:5, Insightful)
From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.
Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.
MAC address ... switch port ... it should be easy.
Re:The story keeps changing. (Score:5, Funny)
Second that motion. I'd say these guys are like the Marx Brothers of network administration, except they don't know the Secret Woid, so it looks like they're a couple notches down.
Parent
Re:The story keeps changing. (Score:5, Insightful)
Parent
Re:The story keeps changing. (Score:5, Insightful)
And exactly how would superman find it? Xray vision? How would he then know he found it?
Parent
Re:The story keeps changing. (Score:5, Insightful)
If Superman had any IT skills, he'd perform a traceroute to determine the devices gateway. Once the gateway was determined, block the mac address from accessing the network. If the admin of that device is worth his salt, he'll change the mac address and continue. They could then specifically enable allowed devices and forbid all others.
Forget finding it, make the network inaccessible.
City of SF Admins, if this proves to be your resolution, you owe me $150 for 1 hour of my time. Sorry, I do not bill in lower increments.
Parent
Re:The story keeps changing. (Score:5, Informative)
Parent
Re:The story keeps changing. (Score:5, Funny)
I CAN find a wireless device It's called Radio direction finding, with the right gear you can do it, and I have located 802.11g devices with it. It's not hard.
so you may start calling me SUPERMAN.
Parent
Re:The story keeps changing. (Score:5, Informative)
and I do development on some software [airwave.com] that will use RF data from your existing wireless access points to triangulate and display the physical location of every user and device on your network!
So you can call me, uh, Jerry Siegel, I guess? :| that's not as impressive...
Parent
FoxHunt (Score:5, Informative)
2> It's easy to find wireless devices... I've personally been doing it since the 1980's.. it's called a fox hunt [wikipedia.org] here in the Chicago area. We used to get 1 minute of transmission every 5... with WiFi you can just ping the dang thing... how easy is that?
--Mike--
Parent
Re:FoxHunt (Score:5, Interesting)
There is an old, probably apocryphal tale from the days of Novel Netware and IPX of the forgotten server. A loan machine runs headless with a quiet fan and no lights in a corner of a room. New remodeling puts the server behind sheet rock and there it sits walled up and running for years. One day a power spike causes a head crash and suddenly a national billing system dies. It takes a tech tracing a cat5 cable into a wall to find it.
Parent
You're an 1D10T (Score:5, Informative)
1) They were firing the guy, so he was no longer in the employ of the city, so his boss, was no longer his boss.
2) You don't know what you're talking about. Every IP address on the network should be known. Either through DHCP or static IP address map. A ping sweep should reveal any IP address in use, that shouldn't be. From the ping sweep, one can arp the unknown IPs to get a MAC address, and do a lookup on the Manufacturer code to know what KIND of device the MAC could be. one could use NMAP to try to discover type of device as well. Then you start going to every port on every switch with rogue IPs hanging off it, and manually looking at what is attached at the other end.
As for wireless access points, if you don't have control over them, you pull the freakin plug. Unsecured Access points and open access points should be VLANed off from administrative networked, including not allowing VPN tunnels from unsecured and open wireless access point.
If the boss allows crap like that on the network, he is an idiot, and shouldn't have the Passwords and access codes to anything.
Parent
Re:You're an 1D10T (Score:5, Informative)
Parent
Re:You're an 1D10T (Score:5, Insightful)
Yes, both of those are true (Mac, Ping). Even NMAP responses can be spoofed. However the likelihood of all three being done is not likely. However NMAP will reveal a used IP, and a mac table somewhere will identify what port it is hanging on. Packets have to be routed to it somehow.
And I agree with your last point. I'm a Libertarian. ;)
Parent
Re:You're an 1D10T (Score:5, Insightful)
I wish I had mod point for you.
Chances are that internal policies prevent the use of "hacker" tools to secure the network.
Again, the PHBs are idiots!
Parent
not necessarily wrong... (Score:5, Insightful)
your employer's passwords are NOT yours, no matter how stupid you think your boss is.
Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?
Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.
Parent
Re:not necessarily wrong... (Score:5, Insightful)
Agreed.
If a boss I don't entirely trust demanded my password, I'd offer to upgrade his account to the same privileges at mine, but he'd NOT get MY password.
The reason is that if he does something stupid that will show up in logfiles, he can damn well do it on his account and get logged doing so ;-)
Parent
Admin code of ethics. (Score:5, Insightful)
What would you think of a doctor who, because some exec somewhere decided he should, pushed the WRONG medication / procedure to you?
Where does your ethical responsibility end and the boss's desires begin?
To me there isn't even a question. Fire me. Go ahead. I will get another job.
Parent
Re:The story keeps changing. (Score:5, Informative)
Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.
By the time his boss thought to ask for the password(s), he had already been fired. Any obligation he had to his boss had disappeared. The same goes for documentation and written procedures - I'm not going to document anything after I've been sacked. In this case the guy had been arguing for written procedures to be put in place, but no one in authority would sign them off as any failures would then be their ultimate responsibility. It should be the managers that are taking flack for this, as so often with IT cock ups.
Parent
Re:The story keeps changing. (Score:5, Interesting)
Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.
My obligation to my employer (in this case the city of San Francisco) trumps my obligation to my PHB. If I think my PHB is a moron and is going to cause a shitload of damage to my employer then I think I could make a good case for refusing to give him the passwords.
Of course that's not where it would end.... I would have to explain to his boss what the problem was -- or go even further up the chain of command if he was also a moron.
Assuming that they have wireless on their network, there's no way to find wireless devices
Wireless devices still have MAC addresses. By tracing the MAC address you'd get a switch port. If that switch port has an AP plugged into it then you know it's a wireless device and probably know it's general location (the AP doesn't have limitless range).
there's no real way to find exactly where wireless devices are, as far as I know
Oh, there's a way [wikipedia.org].... it's just out of the reach of most of us.
Parent
So that's a good point .. (Score:5, Interesting)
Who is actually the OWNER of the system? The boss? Isn't he employed by the same company as the sysadmin? Don't they both have an obligation to safeguard the OWNER'S property and interests? If the sysadmin refuses to hand over the password to sensitive equipment & systems to a (perceived) inept superior-- as long as that guy DOESN'T own the company-- isn't he actually performing his responsibility to the real owner? Which in this case would be the city, and the personification of the city would be the mayor-- and that's exactly who he DID give the passwords to. So it seems to me like he did precisely what he was supposed to do in terms of safeguarding the network and sensitive equipment. Of course he should probably be then fired for failing to keep backups, conops, continuity planning, etc. But that's a different matter.
Parent
Malice and stupidity. (Score:5, Interesting)
Why is Slashdot linking to stories that paint the network administrator as a bad guy when he's so obviously surrounded by morons? These are the same people who published all of their user names and passwords [slashdot.org]. That puts the cost of this "hijacking" into perspective. The cost of trusting their employee with the powers required to do the job was zero.
Parent
Mod Parent Up (Score:5, Insightful)
I'd like to add that while the way he handled being surrounded by idiots was wrong, he was clearly surrounded by idiots.
No documentation?
No change control?
No diagrams?
What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.
I worked one place where a guy with a great deal of responsibility died. (here today dead tomorrow kind of thing) His peers blamed *everything* on him simply because they could. This sounds like the same thing.
Parent
Re:Mod Parent Up (Score:5, Insightful)
What really rubs me the wrong way is how you haven't heard a single word from the admin and yet he is blamed for everything.
Well, every Stalin needs his Trotsky!
Parent
Re:Mod Parent Up (Score:5, Insightful)
Parent
Re:Mod Parent Up (Score:5, Insightful)
I took a gig recovering documentation and re-establishing procedures for a great admin who died as well. He really did great docs, but no one had ever used them, and they couldn't figure out the 'copy file piopoiop.dfj to the \asic\wer\2344\sdf.msdfn folder' sort of directions.
And the crew there immediately set to removing, replacing, and destroying all of his systems. He was a Novell hardliner (so was I), and when he was gone, his boss succumbed and the Windows bigots prevailed. Much taxpayer money was spent replacing perfectly functional systems. Mind you their clients were still running Novell, so there was some disconnect when they would get a request for support and start saying 'you have to upgrade (ha!) to Windows'. Their clients, for reasons best left undisclosed, could not upgrade. Both physically impossible and logistically impractical. Start with being 60-1600 meters below the ocean surface, and it only gets more difficult from there.
I'm a little surprised that SF hasn't worked this out. There are plenty of outfits eager to do what is necessary, for a fee of course.
And yes, finding a device is not impossible. Finding the connection to the network is the obvious first step. After that, well, kill it.
Unless it's hiding. That would be unfortunate.
ps- This guy, by many accounts, was brilliant. And a little off the wall. Goes together.
Parent
Re:Malice and stupidity. (Score:5, Funny)
You mean like the VP of the United States? That has been done before.
Parent
MAC search (Score:5, Informative)
Um, do what any network admin does with a rouge device. Search out what port its MAC address is connected to and then start tracing the cable?
I'm fairly certain most all current managed switches allow for this. Even with unmanaged ones you can hunt down which unmanaged switch it is connected to and snoop from there.
Re:MAC search (Score:5, Funny)
I'd think that a red device would be easy to spot in a server room.
Parent
Re:MAC search (Score:5, Informative)
Apparently this was why he refused to give out the admin passwords - he thought, and so far, it appears that he is correct, that they are all morons.
Parent
Re:MAC search (Score:5, Informative)
How, then, can they use the management functions of the equipment if they can't get to it?
Terry Childs provided the passwords [wired.com] to the mayor on July 22. The city "...[was] able to regain complete control of the network," according to the deputy director of the Department of Technology Information Services.
Parent
Re:MAC search (Score:5, Insightful)
I learned early on, that most people don't see the difference between a $12 hour high school geek and a $75 hr network administrator. All most people see is that both do roughly the same job and there is $63 hour difference.
Most of the time, the $12 hr guy is doing most of the same work as the $75 hour guy. The big difference is when crap like this comes up, the $12 hour guy can spend years trying to figure out what the $75 hr guy can figure out in 5 minutes.
Even when the $12 hr guy screws up, the response is "But he was cheaper". It is cheaper to keep a $12 hr guy trying to keep crapware off a computer, rather than a $75 hour guy who doesn't allow crapware in the first place.
The point I'm making, is that a $75 hr guy is worth it, but only to people where time has real value. People who place no value on TIME, don't care about anything other than $ per HR
Parent
to quote bash.org... (Score:5, Funny)
<erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.
Re:to quote bash.org... (Score:5, Insightful)
The admin might not be stupid he might be an ass
1) He placed a rouge device (his personal property) on the SF network
2) He set all the network devices on the network to lose all info on a reboot
3) He will hand over the passwords (after jail) to all the devices except the rogue
You can make equipment hard to find ( mac masquerading comes to mind )... I'm only adequate in terms of networking but I am pretty sure someone who is really good can play a mean game of hide and seek. Who knows *what* he was doing with that device? and were I the network admin I would have to *on principle alone* rebuild everything after this guy left..
Parent
Re:to quote bash.org... (Score:5, Funny)
He placed a rouge device (his personal property) on the SF network
My guess is it'll be next to his guyliner
Parent
Re:to quote bash.org... (Score:5, Insightful)
I wonder if this one is just a complete misunderstanding. One article says that they were set to lose configuration files on "reset". That's pretty typical -- if you have some device you don't have the password to, you can do a full factory reset and get it back to the default password, but that also wipes the configuration files. He might have told his incompetent bosses that, and they thought he meant they'd lose the files on a reboot instead.
Anyway, if this guy is what they're making him out to be, they need to completely wipe and reconfigure the network anyway; it's the only way to be sure he didn't leave a few presents for them.
Parent
This is a job for nmap (Score:5, Interesting)
Hey! Fyodor! They need your number! [insecure.org]
Fyodor spent much of this summer scanning tens of millions of IPs on the Internet (plus collecting data contributed by some enterprises) to determine the most commonly open ports. Nmap now uses that empirical data to scan more effectively.
Zenmap Topology and Aggregation features were added, as discussed in the next news item.
Hundreds of OS detection signatures were added, bringing the total to 1,503.
Seven new Nmap Scripting Engine (NSE) scripts were added. These automate routing AS number lookups, "Kaminsky" DNS bug vulnerability checking, brute force POP3 authentication cracking, SNMP querying and brute forcing, and whois lookups against target IP space. Many valuable libraries were added as well.
Many performance improvements and bug fixes were implemented. In particular, Nmap now works again on Windows 2000.
With just nmap, my old buddies at Farm9 could have sussed this out in a few hours. I think they are still around - as Red Siren / Getronics. [getronics.com]
Ahh. I miss running netcat at 3 AM!
Siding with the network guy (Score:5, Insightful)
Man, the more I read about this story, the more inclined I am to believe the network admin.
He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.
Re:Siding with the network guy (Score:5, Interesting)
Well, the fact that they're contracting outside Cisco experts now suggests nobody else there was technically competent enough to manage the network.
The fact that the network stayed up and running without a hitch, while he was in jail and nobody else had access, suggests he did know what he was doing, and refusing to allow anyone to access the routers to make changes seems to work quite well to keep the system working.
The fact that his supervisors are moronic and useless is no small thing, either.
His actions were extremely stupid, but I fail to see why this idiot's relatively non-disruptive actions rise to the level of criminal prosecution.
Parent
The scene when they find the server (Score:5, Funny)
As Indy deciphered the symbols, he found the correct sequence of tiles to push. The huge stone door slowly opened. Indy grabbed a torch and headed inside. At the end of the long room, there it was on the throne: A massive server. It was archaic, and it appeared to be attached to a punch card reader. Along the sides of the room, there were two rows statutes of archers pointed at the center. Indy made his way slowly to the monitor and keyboard of the server. He brushed away the dust and hit the spacebar. The screen turned on slowly and it displayed:
SCO Server 1.0
Your license has expired. You owe use $699.
>_
Suddenly the archers rotated positions and were aimed at Indy.
"Oh boy."
Sparcstation In The Wall (Score:5, Funny)
Just remember. (Score:5, Interesting)
You think they've learned anything about the gear since then? No wonder they're having problems.
Where to look... (Score:5, Funny)
No power outage in the Terry Childs case? (Score:5, Informative)
http://weblog.infoworld.com/venezia/archives/018376.html [infoworld.com]
An insider claims that the power outage that Terry Childs was accused of using to sabotage the San Francisco network was not a planned outage.
TAGS: Problems, San Francisco's FiberWAN, Terry Childs
If you've been following the Terry Childs case to any degree, you probably know that one of the key allegations keeping him in prison on $5 million bail is that he had willfully planned to cause the network to fail during a planned power outage at the DTIS One Market Plaza Datacenter on July 19th. According to credible information I've recently received, that power outage was only going to affect the cubes and offices in that building, but not the datacenter itself.
Thus, there never was a plan to power down the network core. Thus, there's no way that Childs could have tried to engineer the failure of the network during this planned power outage, since the network core would not have lost power.
[ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]
The evidence supporting this claim comes from someone certainly in a position to know: Ramon Pabros, the DTIS Datacenter Supervisor himself. Pabros has been employed by San Francisco's DTIS for a surprising 41 years. He's been the Datacenter Supervisor since 1984. He's been running datacenters for the City of San Francisco since Ronald Reagan's first term, the introduction of the Macintosh, and the second season of The A-Team. It's probably safe to say that he knows what he's doing.
According to my source, he will testify to the fact that he discussed the power outage with Childs several weeks before the outage, and at least 10 days before Childs' arrest. He will also state that Childs specifically asked for confirmation that the datacenter itself would not be affected, and was reassured that it would not lose power.
With this statement, the City's allegations that Childs planned to cause the failure of the FiberWAN basically collapse.
Now, I'm admittedly a stranger to San Francisco politics, and am certainly not a lawyer, but if the DA was going to make these accusations against Childs, shouldn't they have talked to Pabros? If the OMP Datacenter was not going to lose power on that date, then this charge against Childs is essentially the same as charging someone with planning to burgle a store that doesn't exist.
But then again, this is the same DA's office that placed valid group usernames and passwords into the public record, and an IT department that ran public, unprotected websites containing internal emails, core network details, as well as usernames and passwords.
I suppose I really shouldn't be surprised at all.
UPDATE: It appears that Pabros has just announced he will be retiring, effective next Wednesday. I can't help but wonder if one event has anything to do with the other. I do know that there have been a number of odd layoffs from San Francisco's DTIS in the past two weeks.
Posted by Paul Venezia on September 8, 2008 08:48 AM
Road trip (Score:5, Funny)
There are now dozens of cars packed full of cheetos cheap laptops and foul smelling individuals travelling near, or perhaps at the speed limit, towards san francisco. They're full of people thinking the same thing, "Shit if they can't find a wired device, they sure as hell can't find a wireless one!"