Forgot your password?
typodupeerror
IT

What To Do After You Fire a Bad Sysadmin Or Developer 245

Posted by samzenpus
from the cleaning-up-the-mess dept.
Esther Schindler writes "The job of dealing with an under-performing employee doesn't end when the culprit is shown the door. Everyone focuses on security tasks, after you fire the idiot, such as changing passwords, but that's just one part of the To Do list. More important, in the long run, is the cleanup job that needs to be done after you fire the turkey, looking for the hidden messes and security flaws the ex-employee may have left behind. Otherwise, you'll still be cleaning up the problems six months later."
This discussion has been archived. No new comments can be posted.

What To Do After You Fire a Bad Sysadmin Or Developer

Comments Filter:
  • Re:Blame them! (Score:5, Interesting)

    by TheGratefulNet (143330) on Friday November 09, 2012 @01:41AM (#41929057)

    its been my experience that people are generally pretty good, some better than others, but I rarely run into an evil person.

    companies, otoh, ...

  • by HellYeahAutomaton (815542) on Friday November 09, 2012 @01:53AM (#41929121)

    You hired this employee. Chances are you started off with a relationship of mis-trust:
      - You did a criminal check on the hire
      - You did a drug check.
      - You did a credit check.
      - You did personality test.
      - You used Shockley style brain-teasers to see if they could do things other than what their jobs entail because you don't know how to measure skill, intelligence, or talent.
      - You interviewed in a style of hazing akin to a gang-bang. .. And you still were too stupid to figure out whether or not you had someone who could do the job right.

    Sorry, but the tone of the summary makes you look like an asshole, and you deserve whatever you get. This is your wake-up call.

  • Check your wallet!!! (Score:4, Interesting)

    by dminor14 (2723933) on Friday November 09, 2012 @02:10AM (#41929239)
    I hope he reads this. After a bunch of expensive equipment disappeared under his watch we fired him. The day after, standing around the coffee room I mentioned. "Too bad they fired him, he owed me 50". Three other people suddenly said, "He owed us 50 also." It turned out the same story for everyone. He borrowed 100 and returned 50. (note: some of my best friends are sysadmins so don't get me wrong)
  • Re:Here be Dragons (Score:5, Interesting)

    by Anonymous Coward on Friday November 09, 2012 @02:21AM (#41929281)

    If it's bad enough you should treat it as a bad virus outbreak and build a completely new system in parallel with the old and move the business information to that system and cut off the damaged system from the net. It's a dirty and tedious job but someone needs to do it.

    5 years ago I would not have believed you. That's ludicrous and there is no way a system no matter how touched by an incompetent is so bad it needs to be completely replaced (losing all maturity and buy in etc).

    And then it happened. In the form of a manpower resource management tool designed for internal use. Won't go too detailed, but at the worst end of it, after burning the entire budget with the thing only partially finished it was rolled into production. And very soon after it was discovered that for a relatively important part the idiot was actually going in and changing fields in the database himself every day to give the appearance that the thing worked.

    The guy who was tasked with fixing it basically came back and said "impossible" .. manager didn't doubt it, thing was scrapped, and a quick and dirty one was built on a shoestring budget to replace it that ended up doing far more than the partially "finished" one that had cost much much more.

  • by cstdenis (1118589) on Friday November 09, 2012 @03:00AM (#41929489)

    There isn't really any practical way to be completely sure, but one thing that can help is to not give him reason to want to attack the company.

    Lay him off and pay him out a good severance pay and he is much less likely to leave disgruntled. There may also be other parting perks besides pay that can generate good will depending on the person.

    This also give the added benefit of when something breaks in the old obscure undocumented part of the system only one person knows, that one person may be more willing to help. Tho how beneficial this is depends on how useless he is.

    As for the technical stuff, only way to be sure with sysadmin is rebuild all the servers from scratch (an extremely time consuming task of course).

    For programmer, the whole team should be doing regular code reviews anyway looking for any security bugs. Maybe an extra code audit would be a good idea.

  • Re:idiot? (Score:5, Interesting)

    by Anonymous Coward on Friday November 09, 2012 @03:05AM (#41929527)

    As the same time, I've worked with some people who it took way too long to get rid of.

    Ah, haven't we all.
    I've been in the unenviable position of having to cover for several 'idiots' higher up the sysadmin foodchain who should have been 'let go', I got so pissed off with the nonsense (e.g. why the fuck was someone who hadn't a clue about Linux managing a whole bunch of Linux servers on paper, when I was doing it on a daily basis as an adjunct to the servers I was looking after) and left myself eventually for pastures new..and left them to deal with it. No doubt I'm now the idiot (à la OPs comment) as far as that lot are concerned (and, no doubt so was my replacement who only stuck it out for about 10 months). .

    Whilst I'm at it, here's another true story. One job I had, I set up a Hard disk based backup server, as a backup to our main backup server (a networked tape library) for one of our Linux servers. Everyone was informed, location of server and UPS in one of the comms rooms flagged on network maps (and, it had a big fucking label on the front along the lines of 'Secondary Backup Server - Don't touch'.)
    Six months or so after I leave that job, get a phone call, the third hand HDs on the Linux server failed (they were warned in writingthat fitting these disks wasn't a good idea, but hey, that's another story), the tape backups didn't have all the data, so where was the backup server I set up?. I name the comms room, the server name/IP number, there was a couple of minutes silence at the other end of the phone, 'oh, the network manager removed that machine from the network three months ago.'
    So again, I'm probably OP's 'idiot' (and the writer of the article pointed to's 'turkey') for that, and probably got the blame for the disks failing.

    Ex employees are such wonderfully useful scapegoats to cover up the inadequacies of those still employed.

    Finally, OP and the writer of the article pointed to are both idiots and boors.

  • Re:Here be Dragons (Score:5, Interesting)

    by sgunhouse (1050564) on Friday November 09, 2012 @03:26AM (#41929653)

    That's fine for the "or developer" part.

    There was a village near here who fired their IT person. She tried to hold the system hostage after they fired her, which obviously didn't go too well for either her or the village council - I forget all the details as it's been a couple of years ago now but it was all over the news at the time. Talk about your nightmare scenaios ...

    Z00L00K above is right in general terms - in effect you have a virus or worm where someone has total control of your system. In a worst case, back up the essential data if you can, then do a system rebuild and import you data. No other way to be sure. And of course, make sure they aren't selling your data to your competitor or the Russians or whoever.

  • by Serious Callers Only (1022605) on Friday November 09, 2012 @03:36AM (#41929701)

    Yep. The submission raises more questions about the submitter than the person who just left for me. People who rate others as incompetent with no redeeming features are often incompetent themselves in my experience. The level of paranoia in the submission is also remarkable, but I guess all this checking and for 'hidden messes and security flaws' might be a good excuse for not doing anything useful for the business. Any problems for the next few months can just be blamed on the recent turkey without introspection as to how they might have ended up with this employee or how they might have created such a mess with no-one esle knowing.

    If you have decent processes in place, hidden messes and security flaws would not be possible without extreme malice and intelligence (not possible for an 'idiot' and a 'turkey'), if you don't and cannot change the processes, leave, as you should recognise the workplace is dysfunctional (and that starts right at the top of the department and goes all the way down).

  • by Kelerei (2619511) on Friday November 09, 2012 @04:05AM (#41929813) Homepage
    One of my previous employers, a while back, employed an individual who I will henceforth refer to as the Office Freak From Hell (it had various freaky habits: no personal hygiene, odd behavioural patterns, that kind of thing). I kind of ignored it at first (except to avoid it as much as possible), until it was moved over to my team. It didn't take me long to realise how useless it was -- his code was often delivered late, and was always of a poor quality (example: using strings as every variable type -- really, what the FUCK?). Between my manager and myself, we tried to mentor him, correct him and all of that -- we couldn't fire him straight away as South Africa has really fucking stupid labour laws which makes firing a tedious and difficult process at best (and you'd better not slip up, otherwise the fucktard can successfully sue for damages and the old position back). Meanwhile, I was searching for alternative employment (although mainly because software development in Durban is a dead-end industry, the OFFH was a major contributing factor), received an offer that I couldn't refuse from a company in Cape Town, and put in my resignation. I still had to work a calendar month's notice period though (Americans, things work differently over here!).

    That's when things got interesting.

    My manager and I started the process of handing over all my projects -- most to the rest of my team, but a few went to the OFFH. It didn't take long for the OFFH to piss off one of my soon to be ex-clients to the extent where top level management got involved, the OFFH was finally pulled into a disciplinary hearing (wasn't fired, but received a final written warning), and I had to step back in and clean out the mess. The next day, the OFFH put in for leave on the Friday coming up, went away... and never came back. It was formally dismissed for absconding shortly afterwards.

    That's when we found what was really going on. To summarise:
    • - The code that would be pushed through to production was often not the same code checked into the source code repository, and the production code was riddled with security holes, backdoors, and that kind of thing. (Since I used the code in the repos for code review purposes, I never picked this up.) A few months after I'd worked my notice period and left, I heard that they ended up writing new, parallel systems and chucking everything he'd worked on, while doing their best to maintain it until the parallel system was complete. (Side note: I left on friendly terms, and I still keep in contact with those guys.)
    • - When we went to try to get source code from his machine (see point above regarding the source repos), we discovered a whole lot of background services constantly maxxing out the CPU. We never found out exactly what they did, but given other discoveries, this pretty much resulted in the network team dropping everything and performing a full security audit of absolutely everything.
    • - He would often tag in after hours and during weekends. I remain convinced that he was up to absolutely no good during this time, particularly as I am in possession of an IRC log detailing an intrusion he was involved with on the South African XBox 360 fansite around mid-2009.

    So, while we thought we were dealing with mere incompetence, in truth, the OFFH was a malevolent fucktard.

    All of us involved has learned our lessons -- personally, I'm far more security conscious, and the folks I worked with are far stricter regarding who they hire, development practices and policies, and that kind of thing. As for the OFFH, it seems to have vanished into thin air...

  • Re:Here be Dragons (Score:4, Interesting)

    by Anonymous Coward on Friday November 09, 2012 @04:55AM (#41929979)

    Similar story here - my company was called in to fix the mess when a client fired their internal VB developer. Guy had been working for tham for two years; first year he built an internal admin system for managing their advertising inventory, while for most of the second he'd been attempting to redesign it to allow the advertisers direct acres to book their own ads. The damned thing was so shoddy we're basically rebuilding from scratch. Every one of around 200 asp pages had at least one sql injection flaw, including the login page (password '" or 1=1' let anyone in...). Several publicly accessible pages seemed to exist only for the purposes of hopelessly corrupting the database. The system needed manual maintenance three times a week to prevent an unindexed table of advert impressions served from growing so large that the site's home page timed out during the log in process (the page was O(n^2) on the number of entries). Large numbers of pages on the site consisted of nearly-but-not-quite identical cut and pastes of standard code with filenames and remote urls changed. Among other issues.

      Our choice was fix it over about 6 months, and never be entirely confident we'd hit all the problems, or replace it in about 3. The client having approved replacement, wer're just about to start work on it.

  • Re:Here be Dragons (Score:5, Interesting)

    by Bogtha (906264) on Friday November 09, 2012 @08:51AM (#41930825)

    Wow, it's like this t-shirt [thinkgeek.com] in real life. I have also replaced somebody with a very small shell-script, I felt like I should have gotten an award or something.

Never appeal to a man's "better nature." He may not have one. Invoking his self-interest gives you more leverage. -- Lazarus Long

Working...