Forgot your password?
typodupeerror
Security

San Fran Hunts For Mystery Device On City Network 821

Posted by CmdrTaco
from the they-just-can't-get-a-break dept.
alphadogg writes "With costs related to a rogue network administrator's hijacking of the city's network now estimated at $1 million, city officials say they are searching for a mysterious networking device hidden somewhere on the network. The device, referred to as a 'terminal server' in court documents, appears to be a router that was installed to provide remote access to the city's Fiber WAN network, which connects municipal computer and telecommunication systems throughout the city. City officials haven't been able to log in to the device, however, because they do not have the username and password. In fact, the city's Department of Telecommunications and Information Services isn't even certain where the device is located, court filings state."
This discussion has been archived. No new comments can be posted.

San Fran Hunts For Mystery Device On City Network

Comments Filter:
  • Simple: (Score:5, Funny)

    by SilentBob0727 (974090) on Thursday September 11, 2008 @09:55AM (#24962357) Homepage

    Power cycle it with a city-wide EMP.

  • by khasim (1285) <brandioch.conner@gmail.com> on Thursday September 11, 2008 @09:55AM (#24962363)

    From what I've read, his "hijacking" was limited to refusing to give the passwords to his boss whom he considered an idiot.

    Given that they cannot hunt down a single device on the network, I'd have to agree with that assessment.

    MAC address ... switch port ... it should be easy.

    • by DogDude (805747) on Thursday September 11, 2008 @10:01AM (#24962489) Homepage
      1. Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.

      2. Assuming that they have wireless on their network, there's no way to find wireless devices, since they can be put inside of locked buildings. Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.
      • ... Unless your name is "Superman", there's no real way to find exactly where wireless devices are, as far as I know.

        And exactly how would superman find it? Xray vision? How would he then know he found it?

      • by Lumpy (12016) on Thursday September 11, 2008 @10:09AM (#24962699) Homepage

        I CAN find a wireless device It's called Radio direction finding, with the right gear you can do it, and I have located 802.11g devices with it. It's not hard.

        so you may start calling me SUPERMAN.

      • FoxHunt (Score:5, Informative)

        by ka9dgx (72702) * on Thursday September 11, 2008 @10:10AM (#24962717) Homepage Journal
        1> Yes.. people could be hurt because the network in question is used to save lives, so it's OK not to hand the keys to an idiot.

        2> It's easy to find wireless devices... I've personally been doing it since the 1980's.. it's called a fox hunt [wikipedia.org] here in the Chicago area. We used to get 1 minute of transmission every 5... with WiFi you can just ping the dang thing... how easy is that?

        --Mike--

      • You're an 1D10T (Score:5, Informative)

        by Archangel Michael (180766) on Thursday September 11, 2008 @10:12AM (#24962757) Journal

        1) They were firing the guy, so he was no longer in the employ of the city, so his boss, was no longer his boss.

        2) You don't know what you're talking about. Every IP address on the network should be known. Either through DHCP or static IP address map. A ping sweep should reveal any IP address in use, that shouldn't be. From the ping sweep, one can arp the unknown IPs to get a MAC address, and do a lookup on the Manufacturer code to know what KIND of device the MAC could be. one could use NMAP to try to discover type of device as well. Then you start going to every port on every switch with rogue IPs hanging off it, and manually looking at what is attached at the other end.

        As for wireless access points, if you don't have control over them, you pull the freakin plug. Unsecured Access points and open access points should be VLANed off from administrative networked, including not allowing VPN tunnels from unsecured and open wireless access point.

        If the boss allows crap like that on the network, he is an idiot, and shouldn't have the Passwords and access codes to anything.

      • by damn_registrars (1103043) <damn.registrars@gmail.com> on Thursday September 11, 2008 @10:14AM (#24962785) Homepage Journal

        your employer's passwords are NOT yours, no matter how stupid you think your boss is.

        Refusing to give out passwords to higher-ups is not always the wrong thing to do. If you are the network admin, and your job is to maintain security of the network, wouldn't it be reasonable to refuse to hand out passwords to people outside of the network administration roles?

        Although I can say that an admin can make that choice at his or her own peril. After all, the higher-ups can always opt to fire the admin and replace him or her with someone who is willing to seek security of their job over security of the network they are paid to administer.

      • by LizardKing (5245) on Thursday September 11, 2008 @10:42AM (#24963297)

        Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.

        By the time his boss thought to ask for the password(s), he had already been fired. Any obligation he had to his boss had disappeared. The same goes for documentation and written procedures - I'm not going to document anything after I've been sacked. In this case the guy had been arguing for written procedures to be put in place, but no one in authority would sign them off as any failures would then be their ultimate responsibility. It should be the managers that are taking flack for this, as so often with IT cock ups.

      • by Shakrai (717556) on Thursday September 11, 2008 @10:46AM (#24963367) Journal

        Your boss is your boss. Unless there's the chance that somebody could be physically hurt, your employer's passwords are NOT yours, no matter how stupid you think your boss is.

        My obligation to my employer (in this case the city of San Francisco) trumps my obligation to my PHB. If I think my PHB is a moron and is going to cause a shitload of damage to my employer then I think I could make a good case for refusing to give him the passwords.

        Of course that's not where it would end.... I would have to explain to his boss what the problem was -- or go even further up the chain of command if he was also a moron.

        Assuming that they have wireless on their network, there's no way to find wireless devices

        Wireless devices still have MAC addresses. By tracing the MAC address you'd get a switch port. If that switch port has an AP plugged into it then you know it's a wireless device and probably know it's general location (the AP doesn't have limitless range).

        there's no real way to find exactly where wireless devices are, as far as I know

        Oh, there's a way [wikipedia.org].... it's just out of the reach of most of us.

    • by Fx.Dr (915071) <exterminans@pala ... com ['the' in ga> on Thursday September 11, 2008 @10:03AM (#24962533)
      ...his boss whom he considered an idiot...I'd have to agree with that assessment

      Second that motion. I'd say these guys are like the Marx Brothers of network administration, except they don't know the Secret Woid, so it looks like they're a couple notches down.
    • by moderatorrater (1095745) on Thursday September 11, 2008 @10:06AM (#24962637)
      Agreed. If they're still having problems at this point, they're incompetent jackasses. However, that's not an excuse for the employee to be a jackass too.
  • MAC search (Score:5, Informative)

    by jeffy210 (214759) on Thursday September 11, 2008 @09:56AM (#24962381)

    Um, do what any network admin does with a rouge device. Search out what port its MAC address is connected to and then start tracing the cable?

    I'm fairly certain most all current managed switches allow for this. Even with unmanaged ones you can hunt down which unmanaged switch it is connected to and snoop from there.

    • by Yvan256 (722131) on Thursday September 11, 2008 @09:59AM (#24962453) Homepage Journal

      I'd think that a red device would be easy to spot in a server room.

    • by StandardCell (589682) on Thursday September 11, 2008 @10:12AM (#24962755)
      If the city can't even complete one of the most basic network administration tasks of finding a physical device on a network, I think they have absolutely no right to accuse anyone of "hijacking" their network. I hope the defense attorney for Terry Childs brings this up.
  • by SomeGuyFromCA (197979) on Thursday September 11, 2008 @10:00AM (#24962467) Journal

    <erno> hm. I've lost a machine.. literally _lost_. it responds to ping, it works completely, I just can't figure out where in my apartment it is.

    • by FireStormZ (1315639) on Thursday September 11, 2008 @10:15AM (#24962819)

      The admin might not be stupid he might be an ass

      1) He placed a rouge device (his personal property) on the SF network
      2) He set all the network devices on the network to lose all info on a reboot
      3) He will hand over the passwords (after jail) to all the devices except the rogue

      You can make equipment hard to find ( mac masquerading comes to mind )... I'm only adequate in terms of networking but I am pretty sure someone who is really good can play a mean game of hide and seek. Who knows *what* he was doing with that device? and were I the network admin I would have to *on principle alone* rebuild everything after this guy left..

  • by Jeremiah Cornelius (137) * on Thursday September 11, 2008 @10:01AM (#24962477) Homepage Journal

    Hey! Fyodor! They need your number! [insecure.org]

    Fyodor spent much of this summer scanning tens of millions of IPs on the Internet (plus collecting data contributed by some enterprises) to determine the most commonly open ports. Nmap now uses that empirical data to scan more effectively.
    Zenmap Topology and Aggregation features were added, as discussed in the next news item.
    Hundreds of OS detection signatures were added, bringing the total to 1,503.
    Seven new Nmap Scripting Engine (NSE) scripts were added. These automate routing AS number lookups, "Kaminsky" DNS bug vulnerability checking, brute force POP3 authentication cracking, SNMP querying and brute forcing, and whois lookups against target IP space. Many valuable libraries were added as well.
    Many performance improvements and bug fixes were implemented. In particular, Nmap now works again on Windows 2000.

    With just nmap, my old buddies at Farm9 could have sussed this out in a few hours. I think they are still around - as Red Siren / Getronics. [getronics.com]

    Ahh. I miss running netcat at 3 AM!

  • by John Jamieson (890438) on Thursday September 11, 2008 @10:05AM (#24962583)

    Man, the more I read about this story, the more inclined I am to believe the network admin.

    He may be incredibly bull-headed and lacking social self preservation techniques, but he may have been technically right.

  • by UnknowingFool (672806) on Thursday September 11, 2008 @10:05AM (#24962593)
    I'm sure the scene will be like this:

    As Indy deciphered the symbols, he found the correct sequence of tiles to push. The huge stone door slowly opened. Indy grabbed a torch and headed inside. At the end of the long room, there it was on the throne: A massive server. It was archaic, and it appeared to be attached to a punch card reader. Along the sides of the room, there were two rows statutes of archers pointed at the center. Indy made his way slowly to the monitor and keyboard of the server. He brushed away the dust and hit the spacebar. The screen turned on slowly and it displayed:

    SCO Server 1.0

    Your license has expired. You owe use $699.
    >_

    Suddenly the archers rotated positions and were aimed at Indy.

    "Oh boy."

  • by gentimjs (930934) on Thursday September 11, 2008 @10:07AM (#24962661) Journal
    I recall hearing a story about a Sun Sparcstation 2 at my old college that had accidentilly got sealed inside a wall by construction folks when re-working the building the CS lab was in to eliminate a few closets for structural support reasons.. nobody could find it (shock!), but kept using it as a DNS server for another six years. It was found about 2 years after it stopped responding to ping when some component (nvram?) let out, and it started beeping after a power flicker.
  • Just remember. (Score:5, Interesting)

    by AltGrendel (175092) <ag-slashdot @ e x i t0.us> on Thursday September 11, 2008 @10:10AM (#24962725) Homepage
    These are the guys that the "rogue" admin said were too stupid to run the thing in the first place.

    You think they've learned anything about the gear since then? No wonder they're having problems.

  • by s0litaire (1205168) * on Thursday September 11, 2008 @10:18AM (#24962867)
    Did they try the Rouge Admin's office. It's probably that beige box under his desk... Either that or he made up the device and it does not exist, he's laughing at them ripping the place apart trying to find it :D
  • by Joe The Dragon (967727) on Thursday September 11, 2008 @10:40AM (#24963255)

    http://weblog.infoworld.com/venezia/archives/018376.html [infoworld.com]

    An insider claims that the power outage that Terry Childs was accused of using to sabotage the San Francisco network was not a planned outage.

    TAGS: Problems, San Francisco's FiberWAN, Terry Childs

    If you've been following the Terry Childs case to any degree, you probably know that one of the key allegations keeping him in prison on $5 million bail is that he had willfully planned to cause the network to fail during a planned power outage at the DTIS One Market Plaza Datacenter on July 19th. According to credible information I've recently received, that power outage was only going to affect the cubes and offices in that building, but not the datacenter itself.

    Thus, there never was a plan to power down the network core. Thus, there's no way that Childs could have tried to engineer the failure of the network during this planned power outage, since the network core would not have lost power.

    [ Follow the Terry Childs saga with InfoWorld special report: Terry Childs: Admin gone rogue. ]

    The evidence supporting this claim comes from someone certainly in a position to know: Ramon Pabros, the DTIS Datacenter Supervisor himself. Pabros has been employed by San Francisco's DTIS for a surprising 41 years. He's been the Datacenter Supervisor since 1984. He's been running datacenters for the City of San Francisco since Ronald Reagan's first term, the introduction of the Macintosh, and the second season of The A-Team. It's probably safe to say that he knows what he's doing.

    According to my source, he will testify to the fact that he discussed the power outage with Childs several weeks before the outage, and at least 10 days before Childs' arrest. He will also state that Childs specifically asked for confirmation that the datacenter itself would not be affected, and was reassured that it would not lose power.

    With this statement, the City's allegations that Childs planned to cause the failure of the FiberWAN basically collapse.

    Now, I'm admittedly a stranger to San Francisco politics, and am certainly not a lawyer, but if the DA was going to make these accusations against Childs, shouldn't they have talked to Pabros? If the OMP Datacenter was not going to lose power on that date, then this charge against Childs is essentially the same as charging someone with planning to burgle a store that doesn't exist.

    But then again, this is the same DA's office that placed valid group usernames and passwords into the public record, and an IT department that ran public, unprotected websites containing internal emails, core network details, as well as usernames and passwords.

    I suppose I really shouldn't be surprised at all.

    UPDATE: It appears that Pabros has just announced he will be retiring, effective next Wednesday. I can't help but wonder if one event has anything to do with the other. I do know that there have been a number of odd layoffs from San Francisco's DTIS in the past two weeks.

    Posted by Paul Venezia on September 8, 2008 08:48 AM

  • Road trip (Score:5, Funny)

    by Oriumpor (446718) on Thursday September 11, 2008 @10:41AM (#24963261) Homepage Journal

    There are now dozens of cars packed full of cheetos cheap laptops and foul smelling individuals travelling near, or perhaps at the speed limit, towards san francisco. They're full of people thinking the same thing, "Shit if they can't find a wired device, they sure as hell can't find a wireless one!"

  • by aclarke (307017) <spam&clarke,ca> on Thursday September 11, 2008 @11:15AM (#24963855) Homepage
    I went to a boarding school in Kenya for high school. The system of bells ran across the campus of several hundred acres and many buildings in a closed loop, with all the bells in series. The system ran through the main office, with the Super Secure Bell System locked in a cabinet there so nobody could access it. Penalty for messing with the system of bells was said to be expulsion.

    The problem was, that all you had to do to get all the bells on campus to ring was to wire the loop back into the mains.

    We took a clock from the darkroom in the photo lab, and ran two wires through the face plate. We then ran another strip of wire along the minute hand, so whenever the minute hand swept by a certain point on the clock every hour, it would complete the circuit for about 30 seconds and ring every bell on campus.

    We then hid this contraption under a pile of wood in the attic of the wood shop. Right after convocation when I could no longer be expelled, I ran into the building and turned it on.

    Apparently the bells rang off and on mysteriously for most of the next month of holiday until they managed to follow the loop and find the device. Good times.
  • The new WarLords (Score:5, Insightful)

    by DeanFox (729620) * <spam,myname&gmail,com> on Thursday September 11, 2008 @12:19PM (#24965017)

    I'm reminded of a conversation I had some 25 years ago with a co-worker IBM mainframe technician. IBM management was incensed that uneducated morons turning screwdrivers could make 70k a year. Back then as much as what they were paying top MBA stuff shirt types. They were on a mission to get salary levels down to "reality" paying these screwdriver wielding monkeys what they were (in their minds) really worth.

    Attitudes have changed but not a lot. 93% of companies that loose their data center for 10 days or more due to a disaster filed for bankruptcy within one year. 50% filed bankruptcy immediately (National Archives & Records Administration in Washington) [google.com]. One can't say the same thing about those over paid MBAs.

    It may be awhile before IT matures into a "profession" like doctor or lawyer however I personally believe we're holding the keys. The world can't function now without us.

    -[d]-
  • by snydeq (1272828) on Thursday September 11, 2008 @01:04PM (#24965831)
    Paul Venezia digs a little deeper into this so-called "terminal server" [infoworld.com] today in his blog:

    "From what I can see, it's a device running Cisco IOS that was accessed via telnet. I could generate an identical screenshot to the one entered into evidence in about five minutes using an elderly Cisco 2924-XL Ethernet switch -- a device that's certainly not a terminal server. It's completely unclear to me how they could have possibly come to the conclusion that this is a "terminal server" -- the evidence presented to the court certainly does not support that theory."

    Venezia also uncovers additional technical errors in the prosecution's case, which appears to be unraveling [slashdot.org] with the recent news that the DTIS Datacenter Supervisor Ramon Pabros will testify on Childs' behalf [infoworld.com]. Since coming forward, Pabros has announced he will be retiring from the DTIS, effective Sept. 17. Coincidence?

Get hold of portable property. -- Charles Dickens, "Great Expectations"

Working...