Searching For Backdoors From Rogue IT Staff 328
WHiTe VaMPiRe writes "When IT staff are terminated under duress, there is often justification for a complete infrastructure audit to reduce future risk to a company. Here is an exploration of the steps necessary to maintain security." Of course the first piece of advice is to basically assume you've been rooted. Ouch.
Multiple Backdoors (Score:5, Interesting)
Re:I'd say treat it like a DR drill (Score:4, Interesting)
If you're seriously considering this as a possibility, I'd say treat it like a DR drill. Burn everything down to bare metal and restore only the data. It's the only way to be sure...
That seems a bit risky. I cannot see any manager worth his salt giving authorization to purposely destroying data "to see if the backup works".
That's because the order of operations is out of whack.
Rebuild, then cut over. Same result, less risk.
Sorry for glossing that over.
Re:Three words (Score:3, Interesting)
This.
I've worked in a highly stressful environment before where I didn't know if I was going to still have a job the next day or not. I had everything set up sufficiently complex but still for good reasons, that if they had fired me getting someone else to fix it would have been a nightmare and cost them a fortune, which they would find out as soon as they tried to get someone else to go in and fix it.
Since I left on good terms I overhauled everything before I left and took out most of the non bog standard bits I had implemented. They ended up with a slightly worse but fixable in a pinch system.
Had the work environment been less stressful I wouldn't have felt it necessary to go through all of the trouble, but they decided to make it that way, so I decided to build some security into my job that was otherwise nonexistant.
Re:the work involved.. (Score:5, Interesting)
It's fairly impossible to audit all systems to the extent needed. You can easily burn enormous amounts of money and time doing that, and the remedies can disrupt production more than the damage the disgruntled employee would do.
There are so many ways to hide what you're doing that even rebuilding all systems isn't enough. Dangers can hide not only in backdoors, but dead man switches built in to compilers, stored procedures in databases, backups, or the Boss' PC, for that matter.
So instead of sending good money after bad, it can be immensely sensible to let things be and instead try to ensure that the employees don't leave disgruntled.
logic bombs on a timer (Score:5, Interesting)
My accidental SSH backdoor... (Score:4, Interesting)
I had to administer a system when the vendor's software would fail on the rollover for the day. So it would fail at 5 am, and I would have to be the one to come in to fix it. As it happens at least once every two weeks I started to SSH in to fix it rather than rush to work and have to work an extra three hours that day (and not be compensated for it). The policy that I fought to implement at work was to do a quick audit, change any passwords/keys for any remote entry and to actually create passwords for many of the accounts that did not have passwords. So done and done I thought.
To continue: I had many problems with upper management, one of which was their wanting me to 'tweak' time sheet accounting so that new entry level minimum wage employees were paid for as little as 75% of their legitimate hours worked. I thought this was particularly dickish as they fired employees on a project basis and anyone was usually fired within two weeks. So I quit and tried to get myself as good as a parachute as I could.
Well two weeks after I left I found out the newbie replacement didn't perform the audit when I accidentally clicked on a bookmark at home (Putty) and I was suddenly in a server from my old job. I logged out and didn't feel particularly compelled to tell them that my keys were still trusted. About a month later I made the same mistake. The hole was no longer there. I thought to myself, "Good for him. I guess he's not so incompetent at all."
But curiousity a la Facebook and Twitter revealed that a server had actually gone down that day. Apparently there was a 'rm -rf' oopsy!!!
The story continues, but the end result is that he managed to destroy three servers within a month of my leaving. If I had been malicious I don't think I could have caused that much destruction...
Re:terminated under duress (Score:2, Interesting)
Reminds me of a speech Ian Angell gave at Defcon. I guess a CEO of a bank there terminated and outsourced the entire IT department. A couple days later, it surfaced that he had all kinds of pr0n on his computer.
So what is the advice (Score:5, Interesting)
Re:terminated under duress (Score:5, Interesting)
Relatively current events counterexample A: Terry Childs
He may have bucked the chain of command, but if his employer had sat him down, said, "look, Terry, we think you'd be better off somewhere else - we're going to keep you on until you find a better opportunity, and we're going to help you do that," he would have probably said, "yeah, but you have nobody else here who can handle this thing. You're going to need to hire a firm to manage this or get some better talent on staff," which seemed to be his motivating concern. And so they probably would have done that, and nobody would have gone to jail.
Instead it seemed like a "give us the passwords and um, no you don't need to clean out your desk, why?" kind of scenario. I'm not meaning to absolve Childs of incorrect behavior, but a little Golden Rule would have gone a long way there. I think this is what the GP meant by not disgruntling the employees.
Re:the work involved.. (Score:5, Interesting)
If the back door is as well hidden as the one Ken Thompson [foldoc.org] hid in an early version of Unix, a complete audit of the source code and complete recompile of everything won't be enough to get rid of it. Of course, not many people are capable of pulling that kind of stunt off.
Why the nastiness ? (Score:3, Interesting)
Nastiness is usually a sign of guilt: "It is human nature to hate those we have wronged [sic]" Tacitus.
If the corp is nasty, it will attract further nasties and have to cope with the results. The nice people leave.
If a nice corp has to fire someone for gross malfeasance and such yet cannot charge them, then perhaps send in a trusted senior specialist to check things out quietly. A big investigative purge will just tell everyone there you don't trust them. Then why should they trust you? Thieves have the best locks. Lots of moves in this chessgame.
Let's just say this depends on the former sysad (Score:2, Interesting)
This is why companies need to have an IT-savvy IT manager and know their employees well, and have multiple IT workers watching each other, much like accountants and finance officers are supposed to watch each other and have separated powers.
Know your employees, their abilities, and their personality. Without knowing the person, it's difficult to assess the risk as to whether or not they might or might not do or attempt to do certain things. And what things are even possible for them to attempt.
The easiest way to avoid running around in circles is to know what they are capable of exactly. If their personality is psychopathic super-programmer, you might have good reason to look for hand-coded hidden kernel drivers, or little binary blobs in a proprietary tampered-with program, containing custom logic bombs, and exploits for bugs in other programs (automated privilege elevation and exploitation).
If not, well, more mundane audits should be fine.
If the person is familiar with scripting, then, well, you'll have to check all the scripts extra carefully. Even if not, they might have found something on the web, and it doesn't take rocket science to cron "rm -rf". Which should not be that much a concern if you have solid frequent backups and take additional precautions to secure those.
(Probably) the worst case scenario is they are conspiring with skilled outside hackers, who are providing expertise and assistance.
Once the outsiders have enough information, they may get the IT admin to "run some code" from somewhere obscure, which will lay the playing field, and then later the outsiders will infiltrate the network.
However, that implies premeditation. If an IT admin is going to forcibly lose their job for serious disciplinary reasons, and anything is suspected to be a risk, they should be escorted by security and not allowed to touch any computers until they are gone for good.
Make them stay on premise during working hours, and have them use pen and paper to fill out some paperwork and answer questions.
This way they will not have a lot of "free time" until all your new IT admins' audits and password changes are established.
Re:Two words (Score:2, Interesting)
Here's a fun little story, and only slightly relevant, too!
My company's HR head also controlled access to the network. We outsource our IT and the head of HR was the point of contact (the head of HR was also assistant to the CEO. It was one of the few positions in the company that wasn't procedurally isolated from other responsibilities).
Anyway, the CEO and President decide to clean house. The head of HR is fired, and the CEO goes in and changes passwords. The CEO, however, didn't have direct day-to-day focus on network issues (passwords, accounts, etc), and as a result didn't get every single account the ex-head of HR had access to. A day or so later, "someone" accesses the network using the one account that would have still be accessible the ex-employee. Using this account, "someone" went in and deleted a bunch of data from our servers, including data we were maintaining for over a year on a lawsuit that we were very tangentially connected to.
As far as anyone knows, no action or investigation will take place. I suspect the decision-makers just want this one to go away.
Corporations DEFENSELESS against determined foes (Score:2, Interesting)
I was CIO for a fairly large company. I reported to the CFO. The first few years were great. Then the good CFO retires and is replaced by the bad CFO. This guy was looking for excuses to get rid of me from day one. It took him several years, but eventually he fired me.
It seems he hired some type of security consultant to lock down the place just before I was fired. Some of my staff members were forced into cooperating with this little adventure. Evidently, the bad CFO thought I would launch some kind of high-tech retaliation. This was actually a fair prediction, based on how I was treated. If you treat people badly, you should EXPECT trouble.
While I am in the process of getting the bad news from HR, it occurs to me that I really want revenge. But a high tech attack would be illegal, unethical, and they're probably expecting it. Therefore I will NEVER attempt anything related to IT. I'm not sure exactly how to proceed, but I decide to wait and give it some thought. My staff members knew from day one that I would do something and whatever it was, it was going to be big. The funny part is that the company's security consultant did everything recommended in TFA and then some. And yet he STILL left a gaping hole that I helpfully reported to the company after the fact. So much for the security audit.
Good things come to those who wait. I stumbled across an idea that had nothing to do with IT. I REALLY REALLY want to write about what I did, but details of the operation must remain classified. Everything was 100% legal and ethical. The results were absolutely spectacular. I might turn this into a movie script someday. It was that good. The amount of pain I inflicted exceeds anything that I could have done with computers.
Three important lessons here: (1) Security audits are seldom 100% effective and a determined opponent is going to get in anyway. (2) A really determined foe is not limited to computers. (3) Treating people poorly leads to unintended consequences; see (1) and (2).
Let me correct that (Score:3, Interesting)
The assumption should be that you have been rooted by somebody who knows exactly what things are logged in your systems, possibly with continuous influence on what is being logged and how long, maybe even with the power to alter log files. IMHO one of the important things is to use several servers just for logs, to whom only a single admin has access. If one of them is going in a bad way, then you have at least the logs on the other machine. If you are paranoid, transfer the md5 checksums of the files on your servers to these machines and use git on the etc directory, backing the etc directories up on these machines. and force the it staff to make builds of custom SW automated.
This means you have
a) logs of what has happened (at least you know what you know)
b) a possibility to determine which files changed
c) a documentation about which configuration changes have been done for which purpose.
d) a backup of the configuration, enabling you to reinstall the machine
e) a way to rebuild programs added to the system easily.
Re:Three words (Score:3, Interesting)
Amen to this.
I still have my job, and have never bothered to install back doors. But I am think about moving to a different position/geographical location, and am trying to get rid of all my hacks and cludges so that my replacement can have an easier time. Let me tell, with both of us working on this 2-3 hours each week, we are nowhere close to getting rid of all the crap.
Just a simple example, of which we got rid last week. In 1997 when I had been just hired, my company was in the process of changing its ERP software. The problem was that they had a front end to it that had been written by an outside contractor whom they had fired. He did not put backdoors or anything, but no one had realized that the front end would not work with the new SQL based solution.
Because the problem was dropped in my lap, I ended up hacking together a really ugly, brute force solution - watching the front end server process for disk access requests, putting it to sleep, and creating the old style file on the fly. Thirteen years later, the company owner and two of his close friends who head two of our 50 warehouses refuse to use any other front end. So until last week, I had a compiled program with full access to the main ERP database, to the payroll's server physical disk and to a modem. Good luck finding that.
And yes, I realize that it was a terrible thing to leave active for more than a decade. But seriously, who remembers to go back and work on something like this unless it breaks? The only reason it's gone is that I am trying to tidy things up before I move... and if I was not moving within the company, I doubt I would be so nice.
Re:So what is the advice (Score:3, Interesting)
I wonder how you could know the credentials weren't deleted?
My Boeing e-mail address was on a number of mailing lists. It took a few years for messages to begin bouncing. People would tell me that my address worked one month but not the next nd I had a pretty good idea when my account was dropped.
Boeing's computing security isn't too bright. They shouldn't be bouncing bad e-mail addresses. It lets spys probe the organizational structure. One can also send a message to a valid employee using the first.last@boeing.com format with a return receipt request and examine the headers to see where it was delivered and the internal domain name structure (which tracks the organizational structure).
Re:Three words (Score:3, Interesting)