Forgot your password?
typodupeerror
Government Security News

SF Admin Gives Up Keys To Hijacked City Network 581

Posted by timothy
from the please-let-this-be-the-end- dept.
snydeq writes "Jailed IT admin Terry Childs relinquished his hold over San Francisco's multimillion-dollar FiberWAN, handing his administrative passwords over to San Francisco Mayor Gavin Newsom, who was 'the only person he felt he could trust.' Childs is still being held on $5 million bail for his lockout of the city's FiberWAN, a case that has been called into question since an insider came forward with details about both the network and Childs himself. The case hinges on No Service Password Recovery commands Childs allegedly configured onto several Cisco devices, as well as dial-up and DSL modems the SFPD has discovered that would allow unauthorized connections to the FiberWAN. Childs intends to 'expose the utter mismanagement, negligence, and corruption at DTIS, which if left unchecked, will in fact place the City of San Francisco in danger,' according to his motion. The Department of Telecom and IS has cut 200 of its 350 IT positions since 2000 — pressure that may have contributed to Childs' actions, according to interviews with current and former DTIS staffers. Newsom secured the passwords without first telling the DTIS that he was meeting with Childs."
This discussion has been archived. No new comments can be posted.

SF Admin Gives Up Keys To Hijacked City Network

Comments Filter:
  • by UnknowingFool (672806) on Wednesday July 23, 2008 @11:31AM (#24305193)
    From my viewpoint, it appears that Mr. Childs wasn't so much a malevolent person as much as he was paranoid and protective. We've all met this admin before. He won't give you any rights that you may need to do your job because you could screw "his computers". I'm not saying what he did was right or legal but he may not be the white cat stroking, maniacally-laughing villain that the initial news reports made him to be.
    • by MightyMartian (840721) on Wednesday July 23, 2008 @11:35AM (#24305243) Journal

      ... but he may not be the white cat stroking, maniacally-laughing villain that the initial news reports made him to be.

      No, instead, he's a paranoid monomaniacal prima donna. If it is was me, I'd rather be a white cat-stroking schemer bent on world domination, because the former demonstrates a sick mind.

    • by kesuki (321456) on Wednesday July 23, 2008 @11:41AM (#24305361) Journal

      I'd say the guy is probably suffering from mental illness.

      there are a lot of people who simply never get diagnosed, because they seem to be able to function normally without medication... myself i've had lesser symptoms dating all the way to childhood, but until i had a 'severe' hospitalization requireing symptoms people just didn't think that i was bad off enough.

    • by pieterh (196118) on Wednesday July 23, 2008 @11:47AM (#24305451) Homepage

      Anyone having spent that much effort creating a network - and succeeding - would become paranoid and protective of it. I challenge anyone to invest so much in any project and then happily see it messed up by people who are less competent.

      However the situation is still messed up, the City should never have allowed one person to take on so much responsibility, and at the first sign that he was becoming indispensable, they should have moved him to another project.

      If someone is essential for a project, replace him as soon as you can...

      In fact the whole story is a good case study for outsourcing - a small, competent network firm would have done as good a job, and treated the incompetent managers simply as clients, not bosses.

      The blame lies squarely with the City, not Childs.

      • by UnknowingFool (672806) on Wednesday July 23, 2008 @11:56AM (#24305645)
        I think sometimes people need to see the bigger picture. In my youth, I thought that becoming indispensable meant I was a valuable employee, and I had job security. But I had an epiphany at 2am one morning when I was fixing a problem. I COULD be the only one to fix this problem and be stuck fixing these problems forever. Or I could trust someone else and train them to fix these problems. Could my company find it easier to replace me? Sure, but it's just a job; I'll get other ones. The lost time I could have spent at 2am doing other things (like at home with my family) was worth the compromise. Any of you who missed out on anything because you were at work know what I mean.
        • by Hektor_Troy (262592) on Wednesday July 23, 2008 @12:09PM (#24305859)

          I'm not sure we want to know what you do with your family at 2 AM in the morning. Wife/girlfriend, sure .. but not the rest of your family

        • Being indispensable in one role will prevent you from being promoted. I was up for a coveted project but it was assigned to a less qualified person because I was too indispensable in my current project. Lest you think management was just letting me down softly, they had me train the person who was assigned the coveted project. That's right. They had me train the person who took the good job, and had me stay on my less-good project because I was really good at my current project.

          I'm now trying to become a manager on the other project. They'll probably say that I don't have any experience in the field and promote the person who has it now, but we'll see.

          • by Grishnakh (216268) on Wednesday July 23, 2008 @04:16PM (#24310187)

            This assumes that people actually want to be promoted. Personally, as an embedded software engineer, I have no desire to ever go higher than I currently am (which is basically "peon"). Now, I'd certainly like to work on more interesting projects, and leave less interesting ones to less-experienced people, but when you say the word "promotion", that basically means moving "up" into management. I'll pass on that.

            If I were in your position, getting stuck with crappy projects while underexperienced people get the better projects, the answer is simple: find a new job. I've found in my 10-year career that the way to progress is to simply change jobs when things get too boring, or the pay too low. You get a huge pay increase (since companies typically don't give very generous raises to loyal employees), you get to pick from several positions at different places, and you get a nice change of scenery and coworkers. Promotions are for losers who want to become managers, thereby losing all their marketable technical skills, and only being valuable to their current company. They can't easily bail out, move to another company and get the same or better pay, like I can. And instead of interesting work, they have to waste all their time in boring meetings. If I wanted to do horribly boring work that I hated, I could have picked an entirely different profession like law and made far more money than any manager at these tech companies I've work at.

        • by dubl-u (51156) * <2523987012&pota,to> on Wednesday July 23, 2008 @01:06PM (#24306977)

          I thought that becoming indispensable meant I was a valuable employee, and I had job security.

          When I was similarly obsessed with handling every problem myself, a friend said to me, "The graveyards are full of indispensable people."

      • by Dekortage (697532) on Wednesday July 23, 2008 @11:58AM (#24305677) Homepage

        If someone is essential for a project, replace him as soon as you can.

        Replace them? No. Distribute their responsibilities and knowledge? Yes. You still want the brainchild around to give input and support; it's just that you need backup in case they get hit by a bus (or paralyzing delusions of grandeur).

      • by Rene S. Hollan (1943) on Wednesday July 23, 2008 @01:02PM (#24306913)
        Anyone having spent that much effort creating a network - and succeeding - would become paranoid and protective of it. I challenge anyone to invest so much in any project and then happily see it messed up by people who are less competent.

        Even more so when it's one's job to make sure it runs correctly. IOW, if he does provide access to others, and they screw up, it's his fault.

        While paranoia may have something to do with it, I've found that a reasonable dose of it is healthy. I think, however, that it's more of a case of normal admin paranoia (which is a good thing), coupled with "damned if you do, damned if you don't" workplace policy (e.g. "give him access and you will suffer if he screws up"), that likely led to this paranoia growing to the degree it has.

        Sane, logical, people are often driven "crazy" when forced to work in an illogical environment (where no matter what one does, one is "wrong", and the "correct" choice is the one which has the lowest product of "consequences" multiplied by "likelihood of discovery").

        I recently took the "high road", and paid a hefty price for it: I had been hired to be subcontracted to a large "three letter" telecom firm. The project manager at this firm wanted my SSN. They had no legitimate reason for it: I was being paid (and had SS withholdings taken) by my employer, not them. The manager claimed that it was necessary for me to get an access badge. I responded that I didn't mind signing in every day and getting a temp badge. The manager then claimed it was necessary for a "background check". I responded that I would happily provide my SSN and any other identifying data (date of birth, drivers' license, etc.) to any reputable, well known, background-checking company of their choice. I was reported as "difficult" and reqested that I be removed. While my employer strongly defended my position, in the end, it was not reasonable that I be paid to be idle, we parted ways amicably, and I quicky found (much better!) work elsewhere.

        Turns out the manager in question was allegedly driving those working "for" her as slaves and threatening them with derogatory credit reports from bogus lenders if they did not comply, using one instance of prior theft to justify such "background checks" requiring the SSN.

        It was easy, (though expensive), to walk away from that job. But, in this case, lives may very well hang in the balance no matter what choice Childs makes: refusing raises the possibility of the network "going down". Complying, where he has reasonable belief that others will have a good chance of making the network go down, causing havoc in the city, could border on criminal negligence. As far as criminal charges, what could stick? "Theft?" (of access codes).

        Should he be fired? Perhaps, for insubordination. But, if the management of the city is so bad that backup systems aren't in place, and properly trained IT staff to run networks, this might be the only way to raise awareness of the problem that could really impact lives of the city's residence. He may have very well done a very good deed -- as the story unfolds we'll know more. At this point, like with all accused, he should receive the benefit of the doubt.

        But, regardless of whether he did the "right thing": No good deed goes unpunished, and he should realize this.

    • by houstonbofh (602064) on Wednesday July 23, 2008 @11:48AM (#24305497)
      There is a fine line between the white-cat-stroking genius in James Bond, and the crazy cat lady from the Simpson's.
  • by Stanistani (808333) on Wednesday July 23, 2008 @11:35AM (#24305241) Homepage Journal

    This story has a real obvious 'bad guy' in Childs.

    Arrogant, supposedly unstable, egotistical.

    But there are odd, contrary, little pieces of this tale that intrigue me.

    I'd like to see some comprehensive treatment of this tragicomedy written a year from now, when the dust has settled, and Childs' side of the story can be heard as well.

  • by 4pins (858270) on Wednesday July 23, 2008 @11:35AM (#24305247) Homepage
    Did anyone else wonder why a SourceForge administrator had the keys to a city's network.
  • by Van Cutter Romney (973766) <sriram.venkataramaniNO@SPAMgeemail.com> on Wednesday July 23, 2008 @11:35AM (#24305251)
    He was just too embarrassed by the password - ibonkedmymom.
    • by antifoidulus (807088) on Wednesday July 23, 2008 @12:14PM (#24305949) Homepage Journal
      ibonkedmymom

      Worst....Apple product.....EVER!
    • by Rorschach1 (174480) on Wednesday July 23, 2008 @12:36PM (#24306429) Homepage

      Hey, it's possible. I'll never forget the first time we ran l0phtcrack on our 3000-user domain, back before any password policy was in place. It cracked 60% or 70% of the passwords in the first 30 minutes, and the list was full of good blackmail material.

      I remember scanning down the password list and coming across 'nosexforme'. Then I looked at the user name and collapsed laughing. The guy was someone everyone in the department knew, and he was a friendly, personable guy. Everyone ALSO knew his wife - the Ice Queen, who worked on the financial side and ruled over our department with an iron fist.

      Every person who came over to the console to see what the laughter was about did the same thing - looked at the password, followed the line over to the username, figured out who it was, realized the implication, and collapsed laughing.

      Ok, so it was maybe a little unprofessional of us, but we couldn't help it. And we all liked the guy, so it really engendered more sympathy than ridicule... and reinforced what we all thought of his wife anyway.

    • Re: (Score:3, Funny)

      by Neodudeman (1259256)
      I actually heard the password was '12345.' Coincidentally, I also hear Mayor Gavin Newsom has that same code on his luggage.
  • by grandbastard (1312837) on Wednesday July 23, 2008 @11:36AM (#24305257)

    "Childs intends to 'expose the utter mismanagement, negligence, and corruption at DTIS, which if left unchecked, will in fact place the City of San Francisco in danger,' according to his motion."

    The fact that one employee had complete control over the network should be enough of a sign. Of course this is management, so they're all likely still confused on what's going on and need to have another meeting.

  • Miserable Slashdot (Score:5, Insightful)

    by db32 (862117) on Wednesday July 23, 2008 @11:52AM (#24305567) Journal
    So...I certainly don't know if this guy is crazy or not, but there are a few things that I am surprised the /. crowd really hasn't bothered with.
    1. The problems between IT and Management are so bad across the board that there is a famous cartoon relating these problems. This famous cartoon spawned the "PHB" reference. So...to listen to an IT guy complain of incompetent management shouldn't be a surprise at all. Please everyone, raise your hand if you have been handed complete and utter bullshit requirements or policies that some "PHB" without a technical clue has demanded that you implement. Now...raise your hand if you were stupid enough to EVER give them administrative rights over ANYTHING.
    2. The media has a fucking field day with "evil hackers". This is so bad that the world "hacker" now means criminal and hordes of geeks wimpering and moaning about how the media stole the word. So...the media reporting on yet another "evil hacker holding city hostage" should be taken with a grain of salt. Sensationalist crap reported by people that have less than 0 IT understanding to the masses who also have less than 0 IT understanding. Million to one odds says that if they actually reported the more technical facts of this case the ratings would be near 0 and this story would have never gotten to be so high profile.
    3. He did give the password to the person at the top of the chain of responsibility for this. Which to me sounds like the most appropriate thing to do. If you are so concerned that everyone is an incompetent fool then your only option is to go straight to the top. Imagine how much trouble this guy would be in if he gave out these passwords to a bunch of corrupt and incompetent folks who did bring the city down? At least this way everything continued functioning.

    Finally...and most concerning to me is a quote from the article.
    But without access to either Childs' passwords or the backup configuration files, administrators would have to essentially re-configure their entire network, an error-prone and time-consuming possibility, Chase said. "It's basically like playing 3D chess," he said. "In that situation, you're stuck interviewing everybody at every site getting anecdotal stories of who's connected to what. And then you're guaranteed to miss something."
    Really...so basically these people didn't document ANYTHING. Because config files or not, rebulding your network if you bothered to document things isn't all that hard, it's just time consuming. But straight from their man there they would be stuck interviewing people for anecdotal stories becaues they were too incompetent to bother documenting the network. Nevermind that they seem to have cut their IT staff from 350 to 100 over the last few years. So it sounds like their IT staff was just the favored bucket to take money from, which is hardly new thinking these days. It amuses me to no end when companies/governments treat their IT staff like overpaid housekeeping, largely unneccessary drains on budgets, and an unimportant support function and then scream bloody murder when the shit hits the fan.
    • Re: (Score:3, Insightful)

      by Trojan35 (910785)

      Usually, when shit hits the fan (as in, firing 60% of your workforce), your operation is understaffed. The very first thing people start skipping is documentation. And that's true everywhere, not just IT.

    • by DarkOx (621550) on Wednesday July 23, 2008 @12:48PM (#24306673) Journal

      I don't know how you document things but I generally draw some basic diagrams so I know where everything can be found and where the physical cables go, that does not change very often, and is easy to keep up to date. ACLs and things like QOS and priority rules, IP translations etc change all the time. I certainly make notes about anything exotic but I don't document everything. What I did do is put all config file(s) in CVS everytime I make changes, for some switch is some office some place that is the documentation.

      I think this is a perfectly adequate practice in lots of environments.

  • by Bomarc (306716) on Wednesday July 23, 2008 @11:54AM (#24305609) Homepage
    Reading a lot of comments about him being a nut job. My question is - what if he isn't? Is it possible that as a administrator of a SAN/Network, he saw some significant security issues, and when he presented them to his supervisors was slammed for reporting the problem -- including being fired? I know from experience the feeling: Management does not like to know that they've screwed up, and will fight kicking and screaming rather that admit that they've done something wrong. For me -- most recently this includes bogus Business Requirements, and critical Business Requirements that are not being met. I've found significant security holes in the where I currently work. Presented the problems to management. The response - don't call use, we'll call you.
    • Re: (Score:3, Interesting)

      by mccabem (44513)

      Is it possible that as a administrator of a SAN/Network, he saw some significant security issues, and when he presented them to his supervisors was slammed for reporting the problem -- including being fired? I know from experience the feeling: Management does not like to know that they've screwed up, and will fight kicking and screaming rather that admit that they've done something wrong.

      Not that you need it, but I'll second this from my own experience. Still job hunting for that matter. Grr.

      -Matt

    • Re: (Score:3, Interesting)

      by dubl-u (51156) *

      Reading a lot of comments about him being a nut job. My question is - what if he isn't? Is it possible that as a administrator of a SAN/Network [...]

      Wait, you lost me there.

      As a recovering sysadmin who ran a Cisco network covering 8 time zones myself, I feel comfortable saying that admins of my acquaintance range from a little crazy to a lot crazy.

      You have to be at least somewhat obsessive to worry about all those niggling details and tiny inconsistencies that can bring things crashing down. A big helping of paranoia is entirely necessary to stay two steps ahead of anybody wanting to break into your network. And it's hard to say you have a good sense of

  • "Save the network. Save the world."

  • by Illbay (700081) on Wednesday July 23, 2008 @12:00PM (#24305707) Journal

    ...Couldn't the guy have just MARRIED the computer system, then claimed that it couldn't testify against him under Spousal Privilege [wikipedia.org]?

  • The Fountainhead (Score:4, Interesting)

    by slashkitty (21637) on Wednesday July 23, 2008 @12:04PM (#24305761) Homepage
    The more I read about this story, the more it reminds me of "The Fountainhead". This lone, brilliant man fighting the mediocrity of committees and less achieved managers. The government is NO place for a person like this. He'd be much better off running his own company with no bosses.
  • was there a crime (Score:3, Insightful)

    by TRRosen (720617) on Wednesday July 23, 2008 @12:05PM (#24305801)
    I'm not sure they can even establish any kind of crime in this case. Its been made clear that there was no intent to shut out the city, It was really just a case of an over protective geek securing his network. Until a new security guy (who I bet is pretty dumb and just as arrogant) got pissed when he realized the security on the network wasn't just to lock out those of lesser authority (that might purposely cause damage) but those of lesser knowledge (that might accidentally cause damage)and the later included him.

    you really can't claim the his knowledge of the password as property of the city and access to the network was never blocked (only to changing his configurations). City could have rebooted an used a new configuration at any time.

    lets face it there really is no precedent for charging someone for not giving up a password.

  • by nobodynoone (940116) on Wednesday July 23, 2008 @12:38PM (#24306479)
    1....2....3....4....5.
  • by cluge (114877) on Wednesday July 23, 2008 @12:42PM (#24306557) Homepage

    Knowing how many government IT departments act (blame EVERY failure on the guy that was just fired or left) - his actions could be considered a protective act, of not just the network, but his reputation. As odd as it sounds, he just guaranteed his exit interview was with the mayor, not some HR peeon that has no clue what means when the network fails. In doing so he has protected his network (which ran flawlessly without other folks getting in), his reputation will have to wait until his day in court. The city of SF may wish to avoid that . . .

    cluge

  • by erroneus (253617) on Wednesday July 23, 2008 @12:43PM (#24306597) Homepage

    To have someone ELSE give the "key to the city" to the mayor?

  • by mpapet (761907) on Wednesday July 23, 2008 @01:12PM (#24307105) Homepage

    Another chapter in a very cautionary tale regarding workplace politics. This is how playing a good political game from the bottom always ends badly. Very, very badly.

    SFPD .... that would allow unauthorized connections to the FiberWAN ...

    This factoid, bereft of any detail whatsoever permanently casts the Admin as the Black Hat. He manages a WAN so of course there will be undocumented, but approved (by someone somewhere) devices accessing the WAN. But the admin has no method of getting his case heard by the court of public opinion. None.

    It fact has yet to be established that the WAN was being held ransom or otherwise. The admin has yet to be heard from!

    I'm not arguing for this Admin, because it seems like he committed quite a few wrongs along the way. But this is how fragile one's system admin career actually is.

  • Integrity (Score:5, Insightful)

    by Pointy_Hair (133077) on Wednesday July 23, 2008 @01:14PM (#24307141)

    I agree with many others that point out the gaps in the headlines. The so called "rest" of the story. This circumstance didn't just develop in a week. This case is a classic story of I/T service immaturity - which could be caused by dastardly BOFH's or equally by incompetent management failing to initiate/fund a proper plan. Or both.

    Once you strip away the glorius certifications and acronyms that give you credibility, all that's left is your integrity. Terry Childs has gone to jail to keep his intact. So he's either really stupid or really right.

    Within the linked article is a link to the original InfoWorld "scoop" that contains copy from a confidential source. That copy contains statements that back Childs as having proposed and promoted an I/T security policy, which would be a first step toward process maturity (having a process in the first place).

    My guess is when the dust settles, the story will be as follows:

    1. For years Childs unsuccessfully tries to bring I/T service maturity to the city
    2. Childs continues keeping things running while excercising some CYA
    3. Management finally gets interested or worried enough to hire a "security manager" who then
    4. Crashes down on Childs to relinquish control unconditionally and without any explanation or bothering to include him in the process (typical PHB approach)
    5. Terry bows up and says "you ain't getting shit from me"
    6. New security manager calls the police while preparing a wooden cross and some 20d nails for Childs' hands and feet.
    7. Hello this is InfoWorld, how can we help?
  • by PPH (736903) on Wednesday July 23, 2008 @01:27PM (#24307317)

    He should have written up his side of the story and handed it over to the local papers.

    Then, offer to hand the admin passwords over to the city as a position code based upon the text of his story as printed.

Receiving a million dollars tax free will make you feel better than being flat broke and having a stomach ache. -- Dolph Sharp, "I'm O.K., You're Not So Hot"

Working...