Slashdot Log In
SF Admin Gives Up Keys To Hijacked City Network
Posted by
timothy
on Wed Jul 23, 2008 10:25 AM
from the please-let-this-be-the-end- dept.
from the please-let-this-be-the-end- dept.
snydeq writes "Jailed IT admin Terry Childs relinquished his hold over San Francisco's multimillion-dollar FiberWAN, handing his administrative passwords over to San Francisco Mayor Gavin Newsom, who was 'the only person he felt he could trust.' Childs is still being held on $5 million bail for his lockout of the city's FiberWAN, a case that has been called into question since an insider came forward with details about both the network and Childs himself. The case hinges on No Service Password Recovery commands Childs allegedly configured onto several Cisco devices, as well as dial-up and DSL modems the SFPD has discovered that would allow unauthorized connections to the FiberWAN. Childs intends to 'expose the utter mismanagement, negligence, and corruption at DTIS, which if left unchecked, will in fact place the City of San Francisco in danger,' according to his motion. The Department of Telecom and IS has cut 200 of its 350 IT positions since 2000 — pressure that may have contributed to Childs' actions, according to interviews with current and former DTIS staffers. Newsom secured the passwords without first telling the DTIS that he was meeting with Childs."
Related Stories
[+]
News: Disgruntled Engineer Hijacks San Francisco's Computer System 1082 comments
ceswiedler writes "A disgruntled software engineer has hijacked San Francisco's new multimillion-dollar municipal computer system. When the Department of Technology tried to fire him, he disabled all administrative passwords other than his own. He was taken into custody but has so far refused to provide the password, and the department has yet to regain admin access on their own. They're worried that he or an associate might be able to destroy hundreds of thousands of sensitive documents, including emails, payroll information, and law enforcement documents."
[+]
News: The Inside Story On the San Francisco Network Hijacking 471 comments
snydeq writes "A source with direct knowledge of San Francisco's IT infrastructure has tipped off Paul Venezia to the real story behind Terry Childs' lockout of San Francisco's network, providing a detailed account of the city's FiberWAN, interdepartmental politics, and Terry Childs himself. Childs pleaded not guilty to charges of tampering yesterday and is being held on $5 million bail. According to the source, Childs' purview was limited to the city's FiberWAN — a network he himself built and, believing no one competent enough to touch the network but himself, guarded religiously, sharing details with no one, including routing configuration and log-in information. Childs was so concerned about the network's security that he refused even to write router and switch configurations to flash. But what may prove difficult for the prosecution in its case against Childs is that his restricted access to the network was widely known and accepted among managers and the city's other network engineers. Venezia, who has been suspicious of the official story from the start, suspects that the Childs case may be that 'of an overprotective admin who believed he was protecting the network — and by extension, the city — from other administrators whom he considered inferior, and perhaps even dangerous.' Further evidence is that fact that the network, from what Venezia understands, has been running smoothly since Childs' arrest."
[+]
Entertainment: San Francisco DA Discloses City's Passwords 333 comments
snydeq writes "The office of San Francisco District Attorney Kamala Harris has made public close to 150 usernames and passwords used by various departments to connect to the city's VPN. The passwords were filed this week as Exhibit A in a court document arguing against a reduction in $5 million bail in the case against Terry Childs. Though they placed the passwords in the public record, city prosecutors do seem to think that they are sensitive. InfoWorld's Paul Venezia, who has been following the case closely, provides further analysis of the technical details in the city's case. 'By themselves, [the passwords] would not be enough to allow anyone to access the network via VPN,' Venezia writes, 'but the fact that the city entered them into evidence is quite shocking. At the very least, they'll have to shut down their VPN access for awhile until they've changed them all and modified the configurations of some large number of VPN clients.'"
This discussion has been archived.
No new comments can be posted.
The Fine Print: The following comments are owned by whoever posted them. We are not responsible for them in any way.
Full
Abbreviated
Hidden
Loading... please wait.
'the only person he felt he could trust.' (Score:5, Interesting)
Re:'the only person he felt he could trust.' (Score:5, Insightful)
No, instead, he's a paranoid monomaniacal prima donna. If it is was me, I'd rather be a white cat-stroking schemer bent on world domination, because the former demonstrates a sick mind.
Parent
Re:'the only person he felt he could trust.' (Score:5, Funny)
Has anyone checked on his wife?
I am just saying...
Parent
Re:'the only person he felt he could trust.' (Score:5, Funny)
Yeah, I was hoping to get a Reiser out of someone.
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
I'd say the guy is probably suffering from mental illness.
there are a lot of people who simply never get diagnosed, because they seem to be able to function normally without medication... myself i've had lesser symptoms dating all the way to childhood, but until i had a 'severe' hospitalization requireing symptoms people just didn't think that i was bad off enough.
Parent
Re:'the only person he felt he could trust.' (Score:5, Interesting)
"Your mental illness is not real."
Mine is very real. I doubt you've ever spent 6 months in a hospital trying to tell people that human beings are being infected by a computer virus. (note: the computer virus was real, and i was the only person who could actually get it off the machines, because it was infecting the BIOS and had 'symptoms' like going to the desktop in the middle of a full screen video game, that other people dismissed as being 'real')
To this date, with medication I still am hazy on if computer viruses can infect human beings, on a bad day, i look for malware in every OS on every computer i have, with every tool available to me, including many useful FOSS tools like dban which allows complete erasure of the drive...
"You are the product of a society that is unable to deal with stress and disappointment."
I'll give you that, but you've never gone 6 days unwilling to eat food or drink tap water because it's poisoned, luckily this symptom has been dealt with with medication.
you've probably never hidden in a basement with aluminum covering you to block the mind control waves either. related to this is using a FM radio from the 1980's and 3 cell phones, wrapped in aluminum foil to see if they really block broadcaster waves. While you're still slightly concerned about the type of high energy particles that can go through entire mountains...
"Have you ever looked at mental illness in other countries. It is tiny compared to the USA."
That may be, in a rural environment, telling people about stuff i was worried about every day and shit my family would likely instead of taking me to a doctor, that they couldn't afford would just humor me, and try to keep me eating foods and drinking water. Also, I would likely die at a much younger age, because of the lack of medical treatment overall. Not being treated by doctors is not the same as 'not having mental illness.'
"You embody the problem with the world today. A lack of conviction and discipline that has spread like wildfire."
Except my mental illness is real, my doctor even increased my medicines last week, because he though i was having too many symptoms with just 1 medication.
"Go on with your drug induced normality. You will not be mourned."
If only the drugs actually caused normality. Mine do not, i still have paranoid thoughts ever single day, they're random and unpredictable, and medication only does so much. I don't hear voices, i don't 'see things' that other people don't see, i don't walk around calling people names that i don't recognize, as if i was in a dream, and i don't wind up in a hospital writing notes about everything that i'm worried about thinking that magically if i write it a system administrator of the universe (it's all just a simulation in a computer after all) would be able to deal with the problems if i simply wrote enough notes...
I have 4 boxes of various paperwork including my 'note' writing phase.
you sir, have never been mentally ill so you know not what it is like. you've never been convinced, with you were in a hospital that another mental patient could read your mind, and control you for not looking at the pictures of their family when they asked you nicely to look at the photos.
Parent
Re:'the only person he felt he could trust.' (Score:5, Interesting)
Anyone having spent that much effort creating a network - and succeeding - would become paranoid and protective of it. I challenge anyone to invest so much in any project and then happily see it messed up by people who are less competent.
However the situation is still messed up, the City should never have allowed one person to take on so much responsibility, and at the first sign that he was becoming indispensable, they should have moved him to another project.
If someone is essential for a project, replace him as soon as you can...
In fact the whole story is a good case study for outsourcing - a small, competent network firm would have done as good a job, and treated the incompetent managers simply as clients, not bosses.
The blame lies squarely with the City, not Childs.
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
Parent
Re:'the only person he felt he could trust.' (Score:5, Funny)
I'm not sure we want to know what you do with your family at 2 AM in the morning. Wife/girlfriend, sure .. but not the rest of your family
Parent
Re:'the only person he felt he could trust.' (Score:5, Funny)
Obviously, you have never met the Aristocrats.
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
Being indispensable in one role will prevent you from being promoted. I was up for a coveted project but it was assigned to a less qualified person because I was too indispensable in my current project. Lest you think management was just letting me down softly, they had me train the person who was assigned the coveted project. That's right. They had me train the person who took the good job, and had me stay on my less-good project because I was really good at my current project.
I'm now trying to become a manager on the other project. They'll probably say that I don't have any experience in the field and promote the person who has it now, but we'll see.
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
I thought that becoming indispensable meant I was a valuable employee, and I had job security.
When I was similarly obsessed with handling every problem myself, a friend said to me, "The graveyards are full of indispensable people."
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
Replace them? No. Distribute their responsibilities and knowledge? Yes. You still want the brainchild around to give input and support; it's just that you need backup in case they get hit by a bus (or paralyzing delusions of grandeur).
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
Even more so when it's one's job to make sure it runs correctly. IOW, if he does provide access to others, and they screw up, it's his fault.
While paranoia may have something to do with it, I've found that a reasonable dose of it is healthy. I think, however, that it's more of a case of normal admin paranoia (which is a good thing), coupled with "damned if you do, damned if you don't" workplace policy (e.g. "give him access and you will suffer if he screws up"), that likely led to this paranoia growing to the degree it has.
Sane, logical, people are often driven "crazy" when forced to work in an illogical environment (where no matter what one does, one is "wrong", and the "correct" choice is the one which has the lowest product of "consequences" multiplied by "likelihood of discovery").
I recently took the "high road", and paid a hefty price for it: I had been hired to be subcontracted to a large "three letter" telecom firm. The project manager at this firm wanted my SSN. They had no legitimate reason for it: I was being paid (and had SS withholdings taken) by my employer, not them. The manager claimed that it was necessary for me to get an access badge. I responded that I didn't mind signing in every day and getting a temp badge. The manager then claimed it was necessary for a "background check". I responded that I would happily provide my SSN and any other identifying data (date of birth, drivers' license, etc.) to any reputable, well known, background-checking company of their choice. I was reported as "difficult" and reqested that I be removed. While my employer strongly defended my position, in the end, it was not reasonable that I be paid to be idle, we parted ways amicably, and I quicky found (much better!) work elsewhere.
Turns out the manager in question was allegedly driving those working "for" her as slaves and threatening them with derogatory credit reports from bogus lenders if they did not comply, using one instance of prior theft to justify such "background checks" requiring the SSN.
It was easy, (though expensive), to walk away from that job. But, in this case, lives may very well hang in the balance no matter what choice Childs makes: refusing raises the possibility of the network "going down". Complying, where he has reasonable belief that others will have a good chance of making the network go down, causing havoc in the city, could border on criminal negligence. As far as criminal charges, what could stick? "Theft?" (of access codes).
Should he be fired? Perhaps, for insubordination. But, if the management of the city is so bad that backup systems aren't in place, and properly trained IT staff to run networks, this might be the only way to raise awareness of the problem that could really impact lives of the city's residence. He may have very well done a very good deed -- as the story unfolds we'll know more. At this point, like with all accused, he should receive the benefit of the doubt.
But, regardless of whether he did the "right thing": No good deed goes unpunished, and he should realize this.
Parent
Re:'the only person he felt he could trust.' (Score:5, Funny)
Parent
Re:'the only person he felt he could trust.' (Score:5, Interesting)
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
Hypothetical situation. My job is to keep an eye on a nuclear reactor. It begins to meltdown, and my manager (who isn't trained with the system) instructs me to cool it down. I refuse for [insert reason here].
That's one of many "not doing your job = crime" situations.
He was basically blackmailing / extorting the city of San Francisco - keep me on board or you lose access to the server completely.
Parent
What a bad analogy. (Score:5, Interesting)
Let's try this one instead:
You're responsible for maintaining a nuclear reactor. Your manager, who has no idea how to actually runs the reactor comes in and demands to be given all of the necessary keys and passwords to the reactor. The reactor is currently working flawlessly, and there is no obvious reason for your manager to need access to the system.
Do you:
A. realize that this could be very bad for the company, and protect the company by refusing to turn over access to an unqualified person?
B. turn over access to the access to an unqualified person, and just hope that they don't do anything which results in anyone's death, or your working 16hr shifts for the next 3 months straight.
I would argue that choosing "B" could be criminally negligent, and that A is the better choice, however, he should also immediately go to HR and explain why he's violating the order.
In this particular case, he might've saved the city of San Francisco millions of dollars in lost productivity from someone getting access who had no clue what they were doing.
Parent
Re:What a bad analogy. (Score:5, Interesting)
"In this particular case, he might've saved the city of San Francisco millions of dollars in lost productivity from someone getting access who had no clue what they were doing."
I've worked with this type before. Damn, I've *BEEN* this type before.
Maybe, maybe not. Sounds like this admin was convinced that the rest of the crew were dangerous idiots. Maybe he's unusually paranoid. I vote for paranoid. Just as dangerous as being right, for different reasons. Imagine serious problems occurring while he is in jail. His propable response might be "See? You need to let me out of here so I can fix this and prevent disaster". Suuuuure... I imagine the authorities will swing the door open and let him out to 'fix' things.
If for no other reason, this poor admin is incompetent in a novel, or NOT novel way. He has no competent backup. A kidney stone, myocardial infarct, or even a knee replacement would leave him out of commission, and SF exposed to loss of network. Sheesh. You back up data, you should back up staff as well, wherever possible, and this is clearly in the realm of 'possible'. Even 'essential'.
Of course, the funniest part of this to me is where he claims he can only trust Gavin Newsome. That's F-U-N-N-Y!
Parent
Re:'the only person he felt he could trust.' (Score:5, Funny)
It's not wreckless if you crash.
(Laugh, its a joke!)
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
Hey dreamchaser, this is your boss. I need write access to the email archives. The SEC has been poking around and, well, you know how it goes.
PS - get back to work.
Parent
Re:'the only person he felt he could trust.' (Score:5, Insightful)
Hmmm...under what Statutes?
While employed he was authorized to access those systems. He didn't access them after his employment was terminated, so it isn't Computer Tresspass or anything similar.
The system works, so he didn't break it.
While they can certainly fire him for insubordination, I'm not exactly sure what he could really be charged with.
Parent
The more I read the less I know... (Score:5, Insightful)
This story has a real obvious 'bad guy' in Childs.
Arrogant, supposedly unstable, egotistical.
But there are odd, contrary, little pieces of this tale that intrigue me.
I'd like to see some comprehensive treatment of this tragicomedy written a year from now, when the dust has settled, and Childs' side of the story can be heard as well.
Re:The more I read the less I know... (Score:5, Funny)
Agreed. But only if it's in the form of a Broadway musical.
Parent
Re:The more I read the less I know... (Score:5, Funny)
And written by Joss Whedon, starring Neil Patrick Harris.
We can call it Admin Horrible.
Parent
Re:The more I read the less I know... (Score:5, Funny)
Instead you will get a made-for-TV movie with oodles and oodles of computers running 12 screens each and a funny OS that only have warnings in 100 point sans-serif fonts and backgrounds which look suspiciously like an FBI badge.
But the hero will be a down-on-his-luck gay single parent who obviously uses a Mac Book Pro to compute the primes needed to crack the passwords (while drinking a triple grande latte and eating a scone).
Oh, and explosions. It will have lots of explosions.
Almost forgot the half-naked teenage girls^Wboys (forgot, this was SF).
Parent
Re:The more I read the less I know... (Score:5, Interesting)
I agree completely.
There seems to be a lot more going on here than what we see.
The conspiracy side of me thinks that there's something fishy going on in the department. He found out and got fired because of it. Except he acted fast and hijacked the network. Hence why he only gave the password to the mayor...
Parent
Did anyone else... (Score:5, Funny)
Re:Did anyone else... (Score:5, Funny)
Mayor: For your act of bravery we give you the key to the city's network!
Citizen: Uh thanks, I guess I'll use that to download porn, and umm, upload torrents, and stuff...
Mayor: Give use the key back.
Parent
Actually ... (Score:5, Funny)
Re:Actually ... (Score:5, Funny)
Worst....Apple product.....EVER!
Parent
Re:Actually ... (Score:5, Funny)
Hey, it's possible. I'll never forget the first time we ran l0phtcrack on our 3000-user domain, back before any password policy was in place. It cracked 60% or 70% of the passwords in the first 30 minutes, and the list was full of good blackmail material.
I remember scanning down the password list and coming across 'nosexforme'. Then I looked at the user name and collapsed laughing. The guy was someone everyone in the department knew, and he was a friendly, personable guy. Everyone ALSO knew his wife - the Ice Queen, who worked on the financial side and ruled over our department with an iron fist.
Every person who came over to the console to see what the laughter was about did the same thing - looked at the password, followed the line over to the username, figured out who it was, realized the implication, and collapsed laughing.
Ok, so it was maybe a little unprofessional of us, but we couldn't help it. And we all liked the guy, so it really engendered more sympathy than ridicule... and reinforced what we all thought of his wife anyway.
Parent
Re:Actually ... (Score:5, Funny)
*sigh* you just targeted the wrong audience.
He was just too embarrassed by the password - ibonkedmydad
Fixed!
Parent
Expose mismanagement (Score:5, Insightful)
"Childs intends to 'expose the utter mismanagement, negligence, and corruption at DTIS, which if left unchecked, will in fact place the City of San Francisco in danger,' according to his motion."
The fact that one employee had complete control over the network should be enough of a sign. Of course this is management, so they're all likely still confused on what's going on and need to have another meeting.
Miserable Slashdot (Score:5, Insightful)
1. The problems between IT and Management are so bad across the board that there is a famous cartoon relating these problems. This famous cartoon spawned the "PHB" reference. So...to listen to an IT guy complain of incompetent management shouldn't be a surprise at all. Please everyone, raise your hand if you have been handed complete and utter bullshit requirements or policies that some "PHB" without a technical clue has demanded that you implement. Now...raise your hand if you were stupid enough to EVER give them administrative rights over ANYTHING.
2. The media has a fucking field day with "evil hackers". This is so bad that the world "hacker" now means criminal and hordes of geeks wimpering and moaning about how the media stole the word. So...the media reporting on yet another "evil hacker holding city hostage" should be taken with a grain of salt. Sensationalist crap reported by people that have less than 0 IT understanding to the masses who also have less than 0 IT understanding. Million to one odds says that if they actually reported the more technical facts of this case the ratings would be near 0 and this story would have never gotten to be so high profile.
3. He did give the password to the person at the top of the chain of responsibility for this. Which to me sounds like the most appropriate thing to do. If you are so concerned that everyone is an incompetent fool then your only option is to go straight to the top. Imagine how much trouble this guy would be in if he gave out these passwords to a bunch of corrupt and incompetent folks who did bring the city down? At least this way everything continued functioning.
Finally...and most concerning to me is a quote from the article.
But without access to either Childs' passwords or the backup configuration files, administrators would have to essentially re-configure their entire network, an error-prone and time-consuming possibility, Chase said. "It's basically like playing 3D chess," he said. "In that situation, you're stuck interviewing everybody at every site getting anecdotal stories of who's connected to what. And then you're guaranteed to miss something."
Really...so basically these people didn't document ANYTHING. Because config files or not, rebulding your network if you bothered to document things isn't all that hard, it's just time consuming. But straight from their man there they would be stuck interviewing people for anecdotal stories becaues they were too incompetent to bother documenting the network. Nevermind that they seem to have cut their IT staff from 350 to 100 over the last few years. So it sounds like their IT staff was just the favored bucket to take money from, which is hardly new thinking these days. It amuses me to no end when companies/governments treat their IT staff like overpaid housekeeping, largely unneccessary drains on budgets, and an unimportant support function and then scream bloody murder when the shit hits the fan.
Just out of curiosity... what if he isn't? (Score:5, Interesting)
Sounds like a Heroes episode (Score:5, Funny)
"Save the network. Save the world."
It Being San Francisco... (Score:5, Funny)
...Couldn't the guy have just MARRIED the computer system, then claimed that it couldn't testify against him under Spousal Privilege [wikipedia.org]?
Is this irony? (Score:5, Funny)
To have someone ELSE give the "key to the city" to the mayor?
Re:Falling Down (Score:5, Funny)
Parent
Re:Falling Down (Score:5, Funny)
I guess Newsom is an MCSE/CCNA and therefore is trusted.
It's actually Newsom's perfect hair that generates a trust enhancement field. Terry Childs saw through this, but recognized the hair as a superintelligent alien symbiont that is on our planet to save us from ourselves, so he gave the passwords directly to the hair.
Parent
Re:Falling Down (Score:5, Funny)
If he believes that the Mayor is going to be reconfiguring the routers he certainly is a nutjob!
Parent
Re:Falling Down (Score:5, Insightful)
unconstitutional state law.
We should be able to work this out. Maybe we can just agree that you get to keep your handguns and I get to get married.
Parent
Re:Falling Down (Score:5, Insightful)
Martin Luther King once said, "An individual who breaks a law that conscience tells him is unjust, and who willingly accepts the penalty of imprisonment in order to arouse the conscience of the community over its injustice, is in reality expressing the highest respect for the law."
It was also a law that the California Supreme Court later declared unconstitutional, so it seems like in retrospect it was a pretty good call.
Parent
Re:Self-defeating (Score:5, Interesting)
He's probably hoping for whistleblower protection, and intends to show that he was being terminated wrongfully for threatening to blow the whistle.
It may be a desperation move, but until the facts come out, we don't know. If it turns out that he was being terminated wrongfully, it's possible that the city of SF could be forced to keep him on their payroll... on the other hand, I'd speculate that he's grasping at straws.
I've read some about the "situation", and all I think all we know for certain is that we don't know anything for certain yet.
Parent
Re:Not too bad... (Score:5, Funny)
Parent
Re:Do I understand this correctly? (Score:5, Insightful)
Mr. Paranoid Admin with a God complex had big freakin' huge vulnerabilities on his precious network?
Attaching old-fashioned modems to the console ports of routers and switches is sometimes done in order to allow the administrator to remotely access the equipment during a major network failure.
It's not an egregious "vulnerability", assuming the console it password protected. That statement was spun to make it sound like they were back doors, when in reality this was likely done for no other reason than to facilitate emergency maintenance.
Please note I am not defending Childs generally. I'm just saying that the way they've minced words in some of these allegations gives me pause.
Parent
Re:Do I understand this correctly? (Score:5, Informative)
Most folks aren't familiar with WAN management, so they probably still don't get what you're saying.
People: Installing backdoors in a WAN saves you a 1+ (sometimes much more than 1+) hour trip somewhere to check a stat or reset a device. Installing backdoors in a LAN is lazy. In other words, the difference is geography. As a WAN manager if you don't have what's called an "out of band" management plan, you're an idiot. (Or you have a micro-sized WAN.) It's also not something that's left secretly, it's planned and secured like any other WAN exposure.
Good luck!
-Matt
Parent
Re:End of the days (Score:5, Interesting)
What was the point of holding back for so long now. Now he just lost the last hope for his negotiation.
Or, he wasn't holding back in order to negotiate, but because he wanted to get the opportunity to tell all of his grievances to the one person who he thought might have the power and wherewithal to "fix" the situation. From reading about the motions that his lawyers have filed in court, it seems that Childs is willing to risk going to jail just to be able to publicize the hard time he's been having at work for the past couple of years. In fact, he might have willingly accepted or even pursued the prospect of prosecution because he knew that he would then have a public forum to air his views, and possibly embarrass his bosses (which, despite their best efforts, he has).
Parent
Re:LOL, omg the net (Score:5, Insightful)
Although I find your delivery crude, I agree with your message.
I would not be surprised in even the slightest if the now-branded "paranoid" admin is hailed as a hero in the future for exposing precisely what he has set out to expose.
200 people in eight years?
Enough security risks to compel him to likely ruin his life for what he believes is a good cause?
Why is it so silly to give the benefit of the doubt to someone who, up until his last action, has been trusted with some of the most valuable information the city has to offer?
Parent