Become a fan of Slashdot on Facebook

 



Forgot your password?
typodupeerror
×
IT

Should Developers Have Access To Production? 402

WHiTe VaMPiRe writes "Kyle Brandt recently wrote an editorial exploring the implications of providing developers access to the production servers of a Web site. He explores the risk introduced by providing higher level access as well as potential compromise solutions."
This discussion has been archived. No new comments can be posted.

Should Developers Have Access To Production?

Comments Filter:
  • For me (Score:5, Insightful)

    by enderjsv ( 1128541 ) on Wednesday August 25, 2010 @11:29AM (#33370324)

    Whenever an error occurs that I can't replicate in a dev environment, I'm always SO tempted to hop into prod and start adding in some output statements.

    Yeah, it's probably a good thing I don't have access to prod.

    • by xaxa ( 988988 )

      Round here, it needs to be the other way round.

      My manager likes to stick to the processes, so we (the developers) fill in forms before doing anything (or asking to have it done).

      The sysadmins hate processes, so they just change whatever they feel like on the production servers and make some excuse for why it didn't need any discussion. Unfortunately, since we work quite closely with the users, we get the complaints when it all breaks :-(.

      • Re: (Score:3, Interesting)

        by x2A ( 858210 )

        Eugh yeah I hate that. So what I try to do is code in such a way that if a bug should occur, the whole thing stops working, that way there's no point in my /not/ fixing it on the production server! I'm a freakin genius! No of course I'm joking, but a recent project has hit some problems where I've been able to explain and the client has actually been able to understand the challenges of trying to reproduce an intermittent undiagnosed problem without touching the production code (ie, is just not worth the ti

        • Comment removed (Score:5, Insightful)

          by account_deleted ( 4530225 ) on Wednesday August 25, 2010 @12:51PM (#33371628)
          Comment removed based on user account deletion
    • Re:For me (Score:5, Insightful)

      by IICV ( 652597 ) on Wednesday August 25, 2010 @11:39AM (#33370482)

      That's why you shouldn't have access to prod, but you should be able to either A. get a clone of prod made fairly quickly or B. already have one running so you can mutilate it however you want.

      Seriously, hardware is cheap and people are expensive. Minimizing person-time is worth a bit of hardware gluttony.

      • Re:For me (Score:5, Insightful)

        by FatSean ( 18753 ) on Wednesday August 25, 2010 @11:46AM (#33370618) Homepage Journal

        Test environments are essential, but they do require people-time to keep them matching production.

        • Re:For me (Score:4, Insightful)

          by Americano ( 920576 ) on Wednesday August 25, 2010 @12:25PM (#33371210)

          Deploying to an extra "prod-clone" test server should require pretty trivial amounts of people-time if it's set up properly - it's just another test server, if it takes ridiculous amounts of manual effort to deploy to a single extra test server, you're probably doing something remarkably inefficient.

          • Re: (Score:3, Insightful)

            by Anonymous Coward

            Yes, we call that staging.

            We work on dev.

            We promote changes to stage, which is a dup of the prod environment.

            When stage is approved we promote to prod.

            • Re:For me (Score:5, Interesting)

              by mwvdlee ( 775178 ) on Wednesday August 25, 2010 @02:11PM (#33372444) Homepage

              In the mainframe shop we used to have 5 stages; (production, shadow (with similar load to production), functional acceptance, system integration and development), next to that 2 well secured "emergency" stages linking to prod and shadow and a single "free for all" development area outside the control of the basic stages.

              Mainframe shops tend to be much more closed and mature than more modern environments, and hence much less goes wrong.

              I've also worked in a Java shop at the same company, where they had 3 stages and a locally for dev, but the stages were much less controlled and you could easily skip straight to production. Obviously only the most experienced of programmers did this and only when they were absolutely certain. Obviously quite some more fixes went wrong on production.

              Currently working in an environment without stages, I try to work on test copies as much as possible but the temptation of bugfixing directly in production is quite large.

        • Re:For me (Score:5, Informative)

          by TheRaven64 ( 641858 ) on Wednesday August 25, 2010 @12:30PM (#33371312) Journal
          Depends on how your deployment system is designed. Mainline Xen now has support for brining up complete live duplicates of a running VM and keeping them in sync (or letting them diverge), and this support has been in high-availability hypervisors for decades. If you're using a system like this, it's just a couple of commands to clone the production VM and get something that you can break without impacting users.
        • Re: (Score:3, Interesting)

          Use an automated process that rebuilds your test environments nightly from production backups. Test environment synchronization and backup verification rolled into one.
      • Re: (Score:3, Funny)

        Unfortunately, copying prod to test does not copy the users, which are often an integral part of the error.

    • Re:For me (Score:5, Funny)

      by stillpixel ( 1575443 ) on Wednesday August 25, 2010 @11:42AM (#33370546) Homepage Journal
      User: There is an error on page X
      I tweak that page code on the production server after looking at the error log. Me back to User: An error really? Have you tried pressing F5?
      User: Oh.. hmmm I guess I must have done something wrong. Sorry for bugging you!
      Me: Hey, no problem.
    • With some fore-thought and some discipline an application can be developed with very robust logging techniques. It takes development time, but there is nothing cooler than asking the production guys to turn the logging detail up for a few packages and seeing tons of data in the logs. It's not perfect as you can't log every variable at every moment but it certainly does help.

      I understand some shops can't or won't modify the logging levels on production servers.

    • by gorzek ( 647352 )

      I make changes directly in production on a pharmacy benefit management system. If I screw up, people can't get their pills.

      It's fun to live dangerously. :-p

      • Re:For me (Score:4, Insightful)

        by idontgno ( 624372 ) on Wednesday August 25, 2010 @11:58AM (#33370796) Journal

        If I screw up, people can't get the correct pills.
        It's fun to make other people live dangerously. :-p

        FTFY. Well, for certain values of "pharmacy benefit management system". If your production hacking can botch scrip fulfillment, please say what company you're working for so I can try to avoid it like the plague it is.

        • by Bourdain ( 683477 ) on Wednesday August 25, 2010 @01:10PM (#33371838)

          If I screw up, people can't get the correct pills. It's fun to make other people live dangerously. :-p

          FTFY. Well, for certain values of "pharmacy benefit management system". If your production hacking can botch scrip fulfillment, please say what company you're working for so I can try to avoid it like the plague it is.

          I don't know if Blue Cross Blue Shield has fixed this but, as of a few weeks ago (and this probably has existed for a while), living in EST has made it impossible for scrips to be fulfilled via insurance between midnight and 3AM. This is because, according to the late night pharmacist who is familiar with the issue, the servers are in PST and won't allow fulfillment from the anything but the "current day" regardless of time zone. Too bad the devs there don't understand time zones adjustments / UTC/GMT. Yet again, non-profit environments don't tend to attract the swiftest of folk in general.

      • Re: (Score:3, Insightful)

        by tomhudson ( 43916 )
        And it's sometimes necessary.

        The article is crap. Sample quote:

        Account privileges, file permissions, web server configuration are often not what developers have experience in or are very interested in

        Retarded. Absolutely retarded. Anyone who can write that hasn't got a clue.

        • Re: (Score:3, Interesting)

          by gorzek ( 647352 )

          The whole article is absurdly vague, anyway. Sometimes developers need access to production--such as on a critical system--and sometimes they don't. Had the article between written toward a narrower domain, something more specific than just "Web sites," it might actually be useful. As it is, it's too light and fluffy to have much real-world impact.

    • by rwven ( 663186 )

      While it's not always a good idea to allow devs unrestricted access to prod, sometimes it IS needed in order to solve problems.

      The fact is that most Infrastructure teams that generally manage the production environment simply don't have the technical chops to debug and ferret out issues that are only appearing in production.

      If the infrastructure teams were "team players" it'd be one thing, but they tend to be overbearing, territorial types that insist on debugging by proxy which takes 100x as long and often

    • Re:For me (Score:4, Insightful)

      by Gerzel ( 240421 ) <brollyferret@UUU ... inus threevowels> on Wednesday August 25, 2010 @12:47PM (#33371566) Journal

      This article also assumes that the developers (or often developer) is not also the server admin. Many shops have one or a few IT people wearing many hats.

    • Re: (Score:3, Funny)

      by es330td ( 964170 )
      I did this once and hosed a production Informix system. We had over 100 external users call in before it was realized and fixed. Fortunately, I did it because my manager told me to so I kept my job and she was reprimanded for A) changing production and B) asking someone else to do it and not doing it herself. I learned my lesson and in the subsequent 10 years have never modified a production system without thorough dev testing first.
  • Short answer (Score:5, Insightful)

    by Issarlk ( 1429361 ) on Wednesday August 25, 2010 @11:30AM (#33370346)
    LOL! No.
    • Long Answer (Score:3, Insightful)

      If I found out a developer changed something in a system I tested without it going through the proper process...

      Let's just say I would be very interested to hear why they shouldn't go back and rerun everything again on THEIR dime. (at the very least) In fact, we DID do just that to someone who let a revision slip into their UUT because a developer felt it would fix something and make it perform better.

      It wasn't too expensive of a mistake, just $250,000 to rerun that portion of the test. Although that wa

  • by Score Whore ( 32328 ) on Wednesday August 25, 2010 @11:31AM (#33370360)

    No. It just encourages sloppy development practices.

    Would you want to drive over a bridge that wasn't actually designed and engineered, but rather they just piled some stuff up and will fix it if it collapses? Or have a surgeon chopping you open with the idea that they'll figure it out as they go? So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

    • by jameson71 ( 540713 ) on Wednesday August 25, 2010 @11:43AM (#33370560)
      On the other hand, I wouldn't want the surgeon to have to give instructions to a trained monkey on how to do the the surgery because the surgeon does not have access to the production patient.
      • by jellomizer ( 103300 ) on Wednesday August 25, 2010 @11:57AM (#33370774)

        Bad analogy.

        It is more like having a Cardiologist diagnosing the problem then telling the heart surgeon what to fix.
        We don't give our fixes to a trained monkey we give them to System Administers or implementation specialists.

        I would say that I am a good developer. However I am only an OK System Administrator and I often when my code is done and working well. Deploying the code offers a new set of issues to work threw.

        However If I give to people who are good at taking my code and implementing it, the process runs much smoother without much problems.

      • Re: (Score:3, Insightful)

        by rtfa-troll ( 1340807 )
        Ahh. Well then I don't advise you to visit any medical colleges or hospitals. 'cos, whilst the doctors that will be treating you aren't going to be exactly trained monkeys, they definitely won't be the ones that developed the procedure. In fact, most of the time you will find that they are pretty much following the documentation.
    • by dkleinsc ( 563838 ) on Wednesday August 25, 2010 @11:51AM (#33370706) Homepage

      So why would we want developers to work with the expectation that they get to intervene at the last instant to resolve their failures?

      Because if there's a problem, there will be an expectation that they need to intervene to resolve their failures.

      To play Devil's Advocate here, there are some semi-legit reasons why developers might get production access:

      • If there's a serious production failure, developers are often called upon to assist the admins, because while they aren't admin experts they generally have some administration skills.
      • If there's a bug that makes it to production, the time it would take to fix the bug using proper procedures may cost more than doing a quick-and-dirty fix now and cleaning up using proper procedures later.
      • Diagnosing production-only bugs, which frequently require read-only access. For instance, developers may need read-only access to determine that their software didn't deploy correctly.
      • Helping admins properly configure their software.

      Now, none of this should be done willy-nilly. The basic rule at my workplace is that a developer can do nothing that could potentially alter behavior without managerial approval and admin approval where appropriate. At the same time, the primary enforcement of that rule is trusting our devs, so very little of that is actually enforced technologically.

      • by SatanicPuppy ( 611928 ) * <Satanicpuppy@nosPAm.gmail.com> on Wednesday August 25, 2010 @12:04PM (#33370910) Journal

        I work for a big company where every significant business unit has it's own devs and own processes. And while I'm primarily an admin, I still deploy some dev-level stuff a few times a year.

        And nothing, nothing annoys me more than going to a shop where the admin doesn't understand the tech, doesn't read the project requirements, and will not, under any circumstances, let me see his production hardware before he tries to deploy.

        Because inevitably it fails and then he tries to throw me under a bus, and I have to get on a conference call and try to use my magic mind powers to debug a system I'm still not allowed to see.

        Back when I was primarily a java dev, I used to have to configure Tomcat all the time, and it was the sort of thing were you needed to have some experience. And time after time I'd end up dealing with some windows admin who knew absolutely nothing, but was utterly convinced that I could never know more than him about a server technology.

        In short, common sense and a reasonable respect for experience beats a hard and fast rule any day.

    • Re: (Score:3, Insightful)

      by mini me ( 132455 )

      I do prefer to drive over a bridge that is well designed, but if that bridge is found to have design flaws, I prefer to have them fixed right away. I do not want to wait for the crew to build a second identical bridge next door in order to test their patches.

      • Re: (Score:3, Insightful)

        by John Hasler ( 414242 )

        I do prefer to drive over a bridge that is well designed, but if that bridge is found to have design flaws, I prefer to have them fixed right away. I do not want to wait for the crew to build a second identical bridge next door in order to test their patches.

        How about if, having built the bridge, they could build a second one in a few minutes for a dollar or so, store it out of the way in hyperspace, and swap the two at will? Would you still not want them to work on the offline one? Read I35 Bridge [wikipedia.org] be

    • Re: (Score:3, Interesting)

      by recharged95 ( 782975 )
      Then again, when I interviewed with Google for youtube:

      "We develop on the main servers, it's typical here, but now, aside from that and more importantly, I have a question : can you show the result of inserting the following values into an empty AVL Tree ... in a Python context"

      Being more of a software engineer than a pure CS wonk (couldn't answer the question completely), and having worked on spacecraft control software........ just say not working there was mutual.
  • no. (Score:2, Insightful)

    by pdp1144 ( 599396 )
    It is my experience that giving development access to production gives you a production environment that looks like it has been vandalized. Although meaning well and trying to make the best application as possible; they need their own development lab, and their own staging / production lab.
  • No.

    If needed there should be a mechanism to automate bug reports in a meaningful way, as most professional software has.

  • by Junior J. Junior III ( 192702 ) on Wednesday August 25, 2010 @11:32AM (#33370394) Homepage

    If you want to have control over your production code, you need to have assurance that it is not changing in an uncontrolled fashion. Allowing developers to have access to production locations makes it all too easy for this to happen. Read-only access allows developers to see the running code and perform file comparisons which can be useful in troubleshooting. They should never need more than this.

    And in some cases, even read access can be risky -- I've seen production web sites with resources linking back to development server URIs. It's a good idea to firewall your production servers in such a way that it is not possible for them to reach resources on development servers. This shouldn't prevent developers from being able to read the files on the production server, though.

    • by SatanicPuppy ( 611928 ) * <Satanicpuppy@nosPAm.gmail.com> on Wednesday August 25, 2010 @11:45AM (#33370602) Journal

      See, to me this is more an issue of devs not obeying the rules. They damn well shouldn't be changing production code, and they damn well shouldn't be linking code from other servers.

      Either your devs are a bunch of barely trained lunatics, or they're breaking the rules in a vain attempt to get things done in a timely manner.

      Most times, when I see devs screwing with production it's either a "hero" coder who is way too good to use best practices, or a situation in which the environment is so hostile that the "best" solution seems to be breaking the rules.

      I once did some contract work for a company where the Q&A and testing process took a minimum of two weeks for the most trivial changes, and where the admins on the production servers refused to deploy things like security patches without a testing period that ran close to a month. The devs there had a hundred tricks for sneaking their code into production, and linking production code to the development servers in an attempt to meet their productivity goals.

      Fucking nightmare. Once we ironed out the Q&A thing, and split the admins into two groups (one who maintained, and the other who upgraded and approved changes) the whole process evened out and the devs stopped screwing around on production.

      • by GigsVT ( 208848 )

        I once did some contract work for a company where the Q&A and testing process took a minimum of two weeks for the most trivial changes, and where the admins on the production servers refused to deploy things like security patches without a testing period that ran close to a month. The devs there had a hundred tricks for sneaking their code into production, and linking production code to the development servers in an attempt to meet their productivity goals.

        Fucking nightmare.

        Wow you worked for Linden Lab?

    • by hsmith ( 818216 )
      My favorite is, developer writes the install instructions.

      Works great in dev. test. uat. production staging.

      Doesn't work in production, because something is configured differently.

      Of course, it is the developers fault to not account for it!
  • Everyone agrees... (Score:5, Insightful)

    by SatanicPuppy ( 611928 ) * <Satanicpuppy@nosPAm.gmail.com> on Wednesday August 25, 2010 @11:33AM (#33370414) Journal

    Everyone agrees that developers should never have access to production...Unless they're the developer, in which case it's different.

    Its a good practice to keep them separated, but in the end its just a pissing contest. The server admins don't want some filthy dev messing with their stuff, and I can appreciate that.

    However, admins often lack appreciation of some dev-specific issues, and their ignorance can lead to problems down the line.

    In the end, its the best practice to have everyone work together sensibly, than throw down inflexible rules that cause more trouble than they prevent.

    • Re: (Score:3, Insightful)

      by Aceticon ( 140883 )

      Everyone agrees that developers should never have access to production...Unless they're the developer, in which case it's different.

      Developers should not have access to Production and I say this as a developer.

      Having a way to check logs in Production, maybe read the databases yes, more than that, no.

      Two reasons, one "good" and one bad:
      - If people have access to Production willy-nilly, sooner or later they will break it. This can even be true for Support people only they're the ones getting calls at 5 AM to

    • Re: (Score:3, Insightful)

      by avandesande ( 143899 )

      I don't think developers want access either- if something goes wrong, do you want to be responsible? I never want access to anything I don't need access to.

  • by dmomo ( 256005 ) on Wednesday August 25, 2010 @11:34AM (#33370418)

    So, yes.

  • Uhm, no. (Score:4, Insightful)

    by HerculesMO ( 693085 ) on Wednesday August 25, 2010 @11:34AM (#33370422)

    This is why there is a change control process, and a testing environment.

    If you're doing it wrong, you're asking for trouble.

  • No correct answer (Score:3, Insightful)

    by Motard ( 1553251 ) on Wednesday August 25, 2010 @11:34AM (#33370430)

    There's no correct answer to this question. It depends on the size of the organization and the nature of the system. I've worked in different companies that have been on either sides of where I thought the line should be. The line is drawn in a very different place for a 20 employee company than where it is in a 20,000 employee company.

    • Re: (Score:3, Interesting)

      Bingo, and I don't think the line is drawn at the size of the company but how mission critical the application is. Granted, in a 20 person firm versus a 20,000 person firm, the developer's probably also the administrator. But OTOH, if your business is 20 people big and one of them is a developer, it's easy to assume it's probably IT based and as such, some sort of administrative control is probably a good idea in general. Think about it. Well, sure, 20 people but how many machines? Half rack? rack? 5

    • More than that it also depends upon how it's dealt with. It's a completely different situation if you've got proper auditing, revision tracking and revision control to if you're allowing the files to just be copied without such measures being in place. Not to mention how the accountability when a mistake is made gets handled and how clear the delegation of authority is.
  • Jesus no (Score:3, Informative)

    by BigJClark ( 1226554 ) on Wednesday August 25, 2010 @11:34AM (#33370432)

    The day developers can write code that compiles the first time, then yes, otherwise, jesus, no.

    I work as an Oracle DBA for a mid-size company, and I provide a day-old cleaned copy of production in a different environment/box, and it does the trick.
    • Re: (Score:2, Troll)

      by unformed ( 225214 )

      Yep because code that compiles is guaranteed to work properly.

      Stick to what you know best, the database. Let us do what we do.

    • by rednip ( 186217 )

      The day developers can write code that compiles the first time, then yes...

      I use an IDE (Eclipse), and the builds I do always compile, so, like, thanks for the access! What I think you mean is that the day when developers can guarantee the expected results...

      Personally, I like the QA process, as it gives me 'cover' if something expected happens.

    • by klui ( 457783 )
      Getting stuff to compile first time is easy. Getting it to work properly the first time, that's something else.
    • As a developer, I agree 100% with this. However, my company doesn't provide a day-old cleaned copy of production, so I still need read-access to debug what piece of data is causing catastrophic failure in the customer billing process.

      When I have all the tools I need to do my job, then no, I don't need any access to production.
      When I don't have all the tools I need to do my job, I need to have the ability to look at production.
      I should never be given the ability to touch production.

    • überdev (Score:3, Funny)

      by Anonymous Coward

      I am one of the few people who can run correct code the first time round. I am also proficient enough in OS matters to be able to circumvent access to locked down resources. So I don't care what this post says, I'm doing it myyyyy waaaaay.

  • Hmm (Score:3, Insightful)

    by mark72005 ( 1233572 ) on Wednesday August 25, 2010 @11:35AM (#33370436)
    I think it's helpful in analyzing real-world data and getting an idea about real system loads, testing issues to see if they are in the wild today, etc. For a good developer, it makes life much easier.

    In a very healthy development ecosystem all this data is replicated and there is never any need for a developer to touch prod. In the development ecosystems that exist in the real world though, most are very unhealthy, frustrated by ham-fisted security, process flaws, red-tape, inconsistency, and incompetence ranging from scattered to mostly cloudy.

    The answer is, do you have the class of developer that knows what not to do and desires to play nice, or do you have the usual.
  • As a developer I can tell you that it's impossible to test programs properly and thoroughly without access to production data. However, developers should NOT be granted access to production logins/sites - production data should be copied into development work areas so that developers have an appropriate "sandbox" in which to work/test.
  • by mr_stinky_britches ( 926212 ) on Wednesday August 25, 2010 @11:36AM (#33370450) Homepage Journal

    And the concensus is ... NO

    Who let this question through? It doesn't even seem controversial. I am not aware of any good reason to routinely give developers access to production.

  • I dislike blog postings on Slashdot as a rule - they can get a Slashbox like everybody else - but the arguments made in the article are well-reasoned if somewhat short on detail. How do developers troubleshoot in a production environment? The article acknowledges that troubleshooting in production is necessary and mentions the installing of software, but installing software alone changes the environment (generally a bit of a no-no for debugging, due to Heisenbugs) and debugger hooks can pose a potential sec

  • by i.r.id10t ( 595143 ) on Wednesday August 25, 2010 @11:39AM (#33370490)

    Biggest issue my cow-orkers and I have is that the sysadmin *claims* that the dev box and production box have the same packages, configuration, etc. but in reality, they don't. Most often we find out when we ask for production stuff to be copied over to the dev site to test errors, etc. and just loading it - which works on the live site - generates errors on the dev site.

    • Re: (Score:3, Interesting)

      by PPH ( 736903 )

      Good point. Lots of people are jumping in with remarks about developers tweaking production code. But there are other sorts of access. As a 'developer' I've had very good experiences with having shell access to production systems to read logs, inspect packages and even run little test suites to verify the configuration of the production system.

      For example, at one outfit, I was one of the few non administrative users with shell access to production servers. One day, everything came to a screeching halt. Not

  • Yes, certainly developers should have access to their production machines.

    No, they shouldn't be allowed to do anything they want with them.

    Troubleshooting application breakdowns are much easier for the developer to do. Thus, the access should be limited to logging data, etc. Unless the admin worked on the application itself, diagnosing those kinds of issues through someone else can be extremely difficult at best.

  • by jaymz2k4 ( 790806 ) <jaymz.jaymz@eu> on Wednesday August 25, 2010 @11:44AM (#33370582) Homepage
    If you are a small software shop then I can see reasons for allowing your small technical staff to have access to production. It's all well and good saying that only the admin of that server should have access and there's a full rollout procedure in place to be followed only on certain days, certain times; but even when I've seen that sort of structure in place there are times when it's useful for the developers to have access to production. Nothing is perfect and we'd all love to have multitude's of staging servers, replicating the typical load and uses of production but for a hell of a lot of (non critical I'd add) systems that just doesn't happen.

    There simply is no one rule fits all. Sometimes I wish we had extremely rigorous rules & regulations in place - I'd probably get to go home a hell of a lot earlier. I'm not suggesting you start chucking exceptions all over your checkout code on live but I think you should asses your own situation (and staff for that matter).
  • I work in an environment where the devs fix bugs before adding features, so the code is stable almost all the time. I have less than 1 callout a week that's caused by something a dev has done to the code.
    We hire the best devs, and work in an environment where fixing bugs is more important than adding features. The result is that our devs get full access to production, and even offer to provide support in order to ensure that they're the ones that are woken up if something they've broken falls over OOH.
    I've

  • If you are running into any issue that requires the developers to have access to production then you have much bigger problems than access control. Developers should need access to development servers only (which really should just be there local box or a set up identical to the supported configuration if you need to test things like clustering or different platforms). Developers should not even require access to testing environments. If you have valid contracts and adequate testing then the only issues th
  • Is that font illegible to anyone else? I had to turn Readability [arc90.com] on, it was so bad. Who the heck thought it was a good idea?

  • Developers should have read-only access to production. In this way, they can investigate what is happening but should on no account have any ability to alter anything.
  • by Todd Knarr ( 15451 ) on Wednesday August 25, 2010 @11:50AM (#33370688) Homepage

    Speaking as a developer, I want/need read-only access in production. All too often I need to dig out information while troubleshooting, and most commonly I don't know what all bits I'll need when I start. If it were easy to identify exactly what I'd need to find the problem, I usually already know what the problem is. The hard ones are the ones I can't replicate in development and I only have a starting point, something that won't identify the problem but might help me narrow down where to look next. In those cases the only place I can look is production (since I can't make it happen in a controlled development environment) and I can't give the admins a list of what I'll need (because I need to dig through logs and config files before I'll know what I need to look for next). And if we've gotten to this point, it's probably a priority problem impacting production so it needs to get fixed Right Bloody Now.

    OTOH, while I may need to look at production, I don't need and don't want the ability to modify production except by going through the admins. This, of course, also requires admins who can follow basic instructions like "Look at config file FOO. Find the line in section X that starts with Y. It's value should be XYZZY followed by the number 1. Change that 1 to a unique number for that machine/instance. Repeat this for every machine/instance.". But all too often the response is "That's too complicated. Can you just give us config files to install?". And of course when I ask for the current config files, so I can be sure I'm not overwriting any other modifications to them (which may have happened since the admins control them and do modify them), I get "We can't do that, they've got production passwords in them.". Now all I can do is throw up my hands and go "Whatever.".

    • Re: (Score:3, Funny)

      by EricWright ( 16803 )

      You gotta know how to talk to admins. Tell them they also can be replaced by a very small shell script.

  • by OldeTimeGeek ( 725417 ) on Wednesday August 25, 2010 @11:56AM (#33370764)
    It's not necessarily a case of the admins versus the developers, its more of practicing good data governance.

    Our developers used to have direct access to all of the production databases. This was bad enough, but because of this the organization permitted them to directly "clean up" databases (meaning they wrote to tables directly), we had data that was being changed without the ability to really know who did it. The DBAs hated it and the developers were extremely uncomfortable doing it but it happened anyway. We eventually had a real process audit and the auditors had a field day.

    Needless to say we changed. I hope.
  • Before I would have said "at least read access", since in my experience the bug reports are usually very inadequate and you need to know exactly what the user was seeing and any settings/configuration made in production. Write access was already rather iffy before, and now with most servers being virtualized the best way would be a fast track to create a new clone of production for the ugly cases. We used virtualization heavily at a client I was at, they originally had two environments, test and production.

  • The author/owner of an application should be on the hook for keeping it running and for it's failures. To separate these responsibilities creates perverse incentives and encourages fire and forget development with no thought to future maintenance and troubleshooting. At the same time, to discourage the practice of 'keeping things going by kicking them', access should result in a detailed audit trail, which would be necessary anyways for regulatory compliance.

    This doesn't work very well without other arrange

  • Super simple question: yes, they should have read-only access.

    Unless you are concerned about privacy issues, but then you probably solved those for your sysadmins too, so no biggie.

  • In my experience, programmers with production access cause more visible problems but have substantially higher productivity (6x to 8x).

    My company has previously had unbelievably tight controls put in place as a result of SOX which added a 45 day overhead to any change (except emergency changes) regardless of size (which means small projects were no longer approved- only home runs).

    Now we are going to SAP. All that is gone for now-- productivity is off the charts. I'm sure they will start locking it down a

  • I work with world class developers and an equally competent team of operations folks. The amount of disconnect between the 2 sets of folks is amazing. The developers black box stuff out of their consideration (e.g. setting up load balancers, with or with out affinity, not littering certificates all over the place, the amount of privileges a service needs etc.). The operations folks ignore other aspects (a cache that's hard to build could be lost after a process recycle, not version controlling their ad-hoc

  • Not only no, but hell no.

  • I worked at a small company where I was the sole developer, and had access to the production system. I was able to make changes and roll them out quickly, and only once or twice did I screw something up (and I was able to fix it right away). The problem is that users started coming to me instead of the sysadmin when they had problems. Then the sysadmin/tech support guy got all butt-hurt about it and declared that he would no longer support anything I wrote. As a result, I ended up having to spend way to
  • by TarPitt ( 217247 ) on Wednesday August 25, 2010 @12:09PM (#33370974)

    I worked as an IT auditor for a very big public accounting firm. Reviewing IT controls was a key part of the financial audit (and more so now with Sarbannes Oxley).

    If I found developers had access to production, it was automatically a "no reliance" finding.

    This means the financial applications are inherently untrustworthy that the financial auditors would have to review original source documents for validation.

    "No reliance" meant the audit became much more expensive as a result.

    Also - if the auditors can't rely on the financial reports, should management?

  • No (Score:3, Insightful)

    by istartedi ( 132515 ) on Wednesday August 25, 2010 @12:14PM (#33371048) Journal

    Even when the suggestion of "would you like root on this internal box?" was put to me, my answer was always "No". I write code. Others test it. Admins deploy it.

    People specialize for a reason. If you want half-assed administration, give root to a developer. If you want half-assed code, let admins write software. If you want half-assed testing, have admins and/or developers do it.

  • With Great Power... (Score:3, Interesting)

    by Sandor at the Zoo ( 98013 ) on Wednesday August 25, 2010 @12:22PM (#33371158)

    Of course developers should have some level of access to the production environment. No matter how good your test environment is, it's not going to match the live server in load, or what's in cache, or the concurrent access to some resource, etc.

    Our process was to have one person with access, investigating whatever problem via the SQL command line, or the Rails console (let the RoR jokes commence), with another person watching, to make sure they were doing select * and not update or delete. Even then we'd execute stuff in a transaction or sandbox so that we weren't making any permanent changes, although changes to memcache generally can't be rolled back so easily.

    I've seen admins, who are adamant that dev not be allowed to change anything, change psql configurations at a whim, crippling DB performance. And then blame dev for poor response times. That's so not cool.

  • by ChronoFish ( 948067 ) on Wednesday August 25, 2010 @01:16PM (#33371896) Journal
    For a one man show the answer is self evident.

    For a small web company developing "brochure-ware" - probably more efficient.

    For a small team it's ideal to have individual sandboxes - with one sandbox listed as "staging". Assign the lead developer to turnover code to production. Individual developers have access but are told not to touch anything. They will typically sift through live environment making sure it matches what is in their sandbox, looking at logs, etc.

    For a mid-size team you need one person for maintenance (which includes monitoring nightly builds, responding to code turnover requests, managing automated testing). Even more critical if the code you write is compiled, fragile, or highly sensitive. - Individual developers don't have access to the live box - maybe the team lead will.

    For large teams or small team "units" part of a large production shop : Several layers of "staging and testing" will exist. Code turnovers are mostly automated. Developers don't have access. Automated rollbacks are possible from a robust code management system.

    The key is discipline. If you find yourself modifying live code - you're not disciplined. It means you're not willing to insert logging code and would rather pollute the production environment. There should never be a need to copy from production back to a sandbox (that is what version control software is for!) And version control files should never live on the production server (i.e. in Subversion you never do a checkout of code on the production server - you do an export instead).

    Even with controls in place, there may be a tendency to "develop on production by proxy". Which means instead of re-creating the problem in development, the developer is saying "here try this, here try this, here try this". The team lead should recognize this and put a stop to it.

    -CF
  • by galego ( 110613 ) <jsnsotheracct@gm ... m minus language> on Wednesday August 25, 2010 @02:12PM (#33372464)
    ... in a split decision, vi wins. Oh wait ... wrong holy war!
  • by Zenin ( 266666 ) on Wednesday August 25, 2010 @04:03PM (#33373832) Homepage

    The more developers work in production, the more they can ONLY work in production.

    I'm all for read access (the more eyeballs the better), but actual access to change anything is a train wreck. The devs will forget to check the changes in to the source repo, or they'll check them in differently (bad copy/paste), or they'll check them into the wrong branch/tag. Regardless the next release that goes out silently adds the bug back into production.

    And if developers think it's difficult to fully clone a prod environment configuration into dev now, wait until they try to do it after developers have been hacking on it directly for a while.

    Pretty soon every release is a train wreck requiring tons of post-release tweaking and hammering to get it in place. Every release is a stressful mess as you're all crossing your fingers because you really have no idea what you are actually changing and no way to find out.

    Just don't do it. Hire a good build engineer/release manager/software configuration manager that can sort out, automate, and track environment management well enough that yes, you can reliably clone an accurate representation of production in a matter of minutes. He'll cost you about as much as a good sr developer, but the savings across the board will easily dwarf his salary.

  • by Lord Kano ( 13027 ) on Wednesday August 25, 2010 @04:23PM (#33374144) Homepage Journal

    Long answer, it depends on the situation.

    I (and a couple of my coworkers) have access to production servers, but we don't develop on prod. End of story. We have other devs who do not have access to prod. Dev is for dev, prod is for prod and don't let anyone without the discipline to keep that rule have access to prod.

    LK

  • No. Do not give developers access to the production machine ever, except me...just this once..

Genius is ten percent inspiration and fifty percent capital gains.

Working...