Forgot your password?
typodupeerror
Security Databases IT

Cache On Delivery — Memcached Opens an Accidental Security Hole 149

Posted by timothy
from the everything-has-a-downside dept.
jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."
This discussion has been archived. No new comments can be posted.

Cache On Delivery — Memcached Opens an Accidental Security Hole

Comments Filter:
  • Firewall? (Score:4, Insightful)

    by chx1975 (625070) on Saturday August 07, 2010 @02:08AM (#33171862)
    I run my memcacheds behind firewall. I thought that the basic server security rule was that you firewall everything opening ports very cautiously as necessary.
    • Re:Firewall? (Score:5, Insightful)

      by MikeFM (12491) on Saturday August 07, 2010 @02:13AM (#33171880) Homepage Journal

      My memcached server is on the private network only accessible to other servers and is firewalled to everything but the servers that need access. Not exactly rocket science.

      • by tsm_sf (545316)
        My memcached server[...]

        Since you run memcached maybe you can answer this. From what I've read it seems like an alerted (or just savvy) admin could easily detect this kind of futzing and (if you're lucky just) inject honey directly onto your machine. Attempting to maliciously write to memcached seems equally problematic. You'd never know if you were successful or just seeing what someone wanted you to see. How off base am I?
        • Re:Firewall? (Score:5, Insightful)

          by PIBM (588930) on Saturday August 07, 2010 @07:10AM (#33172494) Homepage

          Ive been running memcached since it's out, even sent some patches in.

          The thing is, why aren't they running this on a private network ?? Memcached is designed to be fast AND non-secure, to be run on your local network. Running it on a server farm with thousands of people having access to your computers and ips is not a private network.

          I had heard about people running it on the local interface and still getting problems before (somebody else with the same computer ran it too and forgot to pick the good port and finally used the same key ...) but that's because IT'S NOT BUILT TO BE USED ON AN UNSECURED NETWORK.

          Nothing new, bad admins get bad things done to them, move along.

          • Memcached is designed to be fast AND non-secure, to be run on your local network

            My entire training on memcached is a 45-minute LUG presentation I went to a couple years back, and even then this was readily apparent.

            The affected websites need to review their security personnel and procedures.

            • by allenw (33234)
              They likely don't have security personnel. I wouldn't be surprised if there were some "just get in the way" or "security can be added later" or "who cares" or any number of other statements made that clueless developers later on regret.
        • by MikeFM (12491)

          I doubt most admins look and if memcached was open it's virtually a feast of access. You wouldn't necessarily find it easy to jump from memcached to the rest of the system but you could gather a lot of information and alter data going to users and internal systems. Depending how memcache is being used it could be a pretty big exploit. The hardest part would be finding the right data keys. All mine are md5 hashes of a per site secret token combined with the given key so pretty difficult but I'm sure the kind

    • Re:Firewall? (Score:5, Interesting)

      by IICV (652597) on Saturday August 07, 2010 @02:16AM (#33171886)

      Yeah, slide 52 (paraphrased) is as follows:

      Fixes?

      1. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW. FW....
      2. .....
      3. Also, FW

      I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?

  • by Anonymous Coward

    DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
    Version 3, August 2010

      Copyright (C) 2010 Anonymous Coward

    Everyone is permitted to copy and distribute verbatim or modified copies of this license document, and changing it is allowed as long as the name is changed.

    DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
    TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION

        0. You just DO WHAT THE FUCK YOU WANT TO.

    • Re: (Score:1, Offtopic)

      by blue trane (110704)

      http://tunes.org/legalese/bugroff.html [tunes.org] [tunes.org]

      Richard Stallman of the Free Software Foundation devised, in addition to some marvelous software, the GNU General Public License (GPL for short). Or the CopyLeft it is sometimes called.

      It is quite a revolutionary document, using the "copyright" tool to to protect your right to use free software.

      Unfortunately using copyright to protect free software is a lot like using a Jackal to guard the hens.

      In fact, various inconveniences relating to this have resulted i

      • Re: (Score:1, Offtopic)

        by flyingfsck (986395)
        Sorry to pop your bubble, but there already is a Do What The Fuck You Want To license.
  • by OverlordQ (264228) on Saturday August 07, 2010 @02:09AM (#33171868) Journal

    Much less 'memcached' being at fault. They say it themselves:

    Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users.

    All this is is stupid admins doing stupid things story and those are dime a dozen.

    • Re: (Score:2, Insightful)

      by Etylowy (1283284)

      This.

      That's what you get for being cheap when hiring security team. Setting up memcached on a public IP or without a firewall is as bad as having your session directory fully writable on a public ftp (and quite similar too).
      Memcached is good software and does exactly what it has been designed to do - provides fast key cache - but as with every tool if you are dumb you will hurt yourself. If those admins were carpenters they'd have an average of 4 fingers ;-)

      • Re: (Score:2, Insightful)

        by Anonymous Coward

        You don't need a security team, just a clue.

    • by TheRaven64 (641858) on Saturday August 07, 2010 @06:19AM (#33172372) Journal
      A good tool is designed in such a way that the correct use is the easiest. Does memcached, for example, default to only accepting connections from the local host and require other IPs to be explicitly added? That would be trivial to implement and, if done, would require the admin to implement something like the correct security policy. In contrast, defaulting to accepting connections from anywhere means that the admin can use it incorrectly without needing to do any thinking, but needs to think before using it correctly.
      • by MikeFM (12491) on Saturday August 07, 2010 @07:35AM (#33172560) Homepage Journal

        It defaults to not being installed and running. Memcached is meant to be ran from one or more caching servers (not really on the web server itself). It isn't really meant to be ran on localhost under ideal usage.

        • by nebular (76369) on Saturday August 07, 2010 @08:24AM (#33172710) Homepage

          I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

          • Re: (Score:3, Insightful)

            by apoc.famine (621563)
            That's like saying knives should be designed to be dull, because users are idiots, and will cut themselves.

            It's not in any way the programmer's responsibility to hold the hands of idiots so they don't hurt themselves. The people who wrote this wrote a solid programs, and IN THE DOCS noted that if you set this up like this, you were an idiot.

            Come on... we have far too much nanny-state as it is. My coffee has a warning on it that it's hot, my keyboard has a warning on it about RSI, my girlfriends hair strai
            • by enoz (1181117)

              I forget who said it, but some comedian said we should remove the warnings from everything, and let nature sort it out.

              I have a hunch it was George Carlin, maybe someone can find the video evidence.

          • I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!

            Perhaps utilities like rm should be rewritten to use -i (or similar) by default then? Or how about we ought to remove them outright since they might be used incorrectly?

            Honestly, it's not up to the developer to make any such assump

            • "Any piece of software can be used "insecurely" if configured incorrectly"

              Not that you don't have a point, but you are doing nothing but supporting the argument you tried to debunk.

              Note that your point stands on the "if configured incorrectly" while the parent's is "it should be distrubuted on such a state that the sysadmin would be forced to do *any* kind of configuration prior for it to work" (thus giving the option to even "configure" it, either correctly or incorrectly).

              I think the parent's point of vie

              • Re: (Score:3, Insightful)

                by Zancarius (414244)

                Note that your point stands on the "if configured incorrectly" while the parent's is "it should be distrubuted on such a state that the sysadmin would be forced to do *any* kind of configuration prior for it to work" (thus giving the option to even "configure" it, either correctly or incorrectly).

                No, you're just splitting hairs. My point is that software can be configured incorrectly and that it isn't the developers' fault if that happens. I believe the parent's point of view (and the general view espoused

                • I think that in your rush to rebuke the parent with scathing sarcasm, you have misinterpreted their point. I don't see anyone suggesting the software should install in a secure, perfectly functional manner, with no configuration whatsoever. The only quote I saw was saying that it's preferable for the software to be installed without a working configuration, so the administrator must explicitly configure it for their purposes. If they fail to configure it securely in the process, that's their own failing. Wh

                  • Seriously? Your point is that Gentoo doesn't do enough hand-holding? This is a distribution that has built its reputation around NOT doing excessive hand-holding.

                    That having been said, Gentoo does leave services that have security implications unconfigured, but I'm having trouble understanding how memcached is such a service. This isn't a system configuration issue, but a network layout issue.

                    Finally, there is this:

                    The admin may not be aware that memcached is not designed to be exposed directly to the internet.

                    Huh? You're administering an internet-facing server, and installing an internet-facing se

                    • True, it's an unusual network configuration to be "exposed by default"; however I still haven't seen any rational explanation why "listen on all interfaces" is a better default than "listen only on loopback".

                      If you're that oblivious, how the hell is making you fill out a config file going to help you?

                      Because the config file section that needs to be modified has comments explaining the risk? Not everyone is going to go and read external documentation on a piece of software before installing it. If you're going to do that, why even use a distribution in the first place? Any documentation you read is l

                    • by Zancarius (414244)

                      True, it's an unusual network configuration to be "exposed by default"; however I still haven't seen any rational explanation why "listen on all interfaces" is a better default than "listen only on loopback".

                      I can think of a number of network-facing software packages in the *nix world that are usually configured, by default, to listen on all interfaces, because that's usually what the end user wants. There are some particular issues I'd like to point out, but if you're feeling exceptionally annoyed you may

                  • by Zancarius (414244)

                    The Gentoo example appears to install a functional but insecure configuration by default. If the user simply starts the service (or possibly reboots the server without doing further configuration and it auto-starts at the next boot), then tests their application, they will be overjoyed that it's working and move on to their next task. The admin may not be aware that memcached is not designed to be exposed directly to the internet.

                    Insecure is in the eyes of the beholder. If the machine is on a firewalled net

          • by MikeFM (12491)

            It's pointless to design for the biggest idiots you can find because they keep designing better idiots. Memcached isn't an app for average users. It's a programmers tool. If people that should know better are still idiots then let the damage be on their heads. Why waste resources trying to save idiots from themselves.

            I say we let people self destruct early so that we can stop wasting resources. Let bad developers get themselves fired, let drug users overdose, etc. Eliminate the chaff instead of coddling eve

        • by TheRaven64 (641858) on Saturday August 07, 2010 @08:32AM (#33172750) Journal

          Which is exactly the point. The default install should never be working-and-insecure. It should be secure, and ideally it should be working. If it is not possible for the default install to be both useful and secure, as appears to be the case with memcached, then it should install only listening on localhost and require explicit intervention by the user to accept connections from other hosts.

          If you can install it and have it work by default, then there is no reason for the user to bother reading the manual, so they won't learn that it needs to be specially configured to be secure. If the default is secure but not particularly useful, then the user needs to explicitly adjust the setting that makes it insecure, and in so doing needs to read the documentation explaining that this will make it insecure and how to mitigate it.

          • by Bigjeff5 (1143585)

            You secure memcached the same way you secure any cache: by restricting direct access. All you have to do is put a proxy between the memcached server/instance and the internet and the problem is solved. If you don't have command-level access to the memcached instance you can't mine it, period.

            Many very large websites don't bother to do this, which is very bad security practice, as the slideshow clearly demonstrates.

            The weakness isn't really with memcached. To say memcache is insecure is like saying your d

            • "You secure memcached the same way you secure any cache: by restricting direct access."

              Which is exactly the parent's point. That's why he says it should default to listening only on 127.0.0.1: in order for the default install to restrict direct access.

              • by Unoti (731964)
                You and the some other ancestors are not understanding the distributed/shared nature of memcached, and how it needs to be accessed from more than one host except in the most trivial use cases.
          • With your philosophy, MySQL wouldn't be where it is.

          • by MikeFM (12491)

            Maybe the problem is that Linux has gotten to easy to use. We should go back to requiring everyone download source files and compile everything themselves. Call it an intelligence test.

        • More than that, once installed, it defaults to not being used at all.

          You have to actively develop and deploy software which uses and trusts memcached before this is a security hazard.

          In other words, while it might be nice if it was secure by default, you now need two groups of people to fuck up at the same time -- system administrators and software developers -- both of whom, as part of their job, are supposed to think about security.

      • Re: (Score:3, Informative)

        by bjourne (1034822)
        Yes memcached defaults to only accepting connections from the local address. From memcached.conf:

        # Specify which IP address to listen on. The default is to listen on all IP addresses # This parameter is one of the only security measures that memcached has, so make sure # it's listening on a firewalled interface. -l 127.0.0.1

        • by Rudolf (43885)

          Yes memcached defaults to only accepting connections from the local address. From memcached.conf

          What you posted does not say the default is localhost. It says the opposite:
          "# Specify which IP address to listen on. The default is to listen on all IP addresses "

          • by bjourne (1034822)
            If you have not specified otherwise (e.g. if /etc/memcached.conf is missing), memcached listens on all interfaces yes. If you install memcached from a Debian-derived distros package repository, then it will create a default /etc/memcached.conf file with the above, default, configuration specified.
    • by Ash Vince (602485)

      All this is is stupid admins doing stupid things story and those are dime a dozen.

      I'll say. I have been looking at using memcached in conjunction with the mod_auth memcache module. I know it exists and have done very cursory investigation. I do not have a working prototype yet but I do know my predecessor built one a year or so ago. My very cursory glance already made it startlingly clear that this would be an issue, any competent sysadmin really should have spotted the same thing.

  • A few clarifications (Score:5, Informative)

    by marcoslaviero (1873058) on Saturday August 07, 2010 @03:06AM (#33171996)
    In terms of the vendors identified, Bit.ly, GoWalla and Pbs were notified. Bit.ly and GoWalla repaired the flaws within minutes. I am not aware of Pbs repairing the issue. This talk seems to have struck a chord which I can't really explain (suggestions welcome). Yes, exposing your memcached's is bad (the talk shows just how bad), but it's not a clever find to discover them. [fd: that's my name on the slides]
  • Admin or distro? (Score:5, Interesting)

    by shish (588640) on Saturday August 07, 2010 @03:12AM (#33172012) Homepage

    Debian's default config says:

    # Specify which IP address to listen on. The default is to listen on all IP addresses
    # This parameter is one of the only security measures that memcached has, so make sure
    # it's listening on a firewalled interface.
    -l 127.0.0.1

    Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)

    • Re: (Score:3, Interesting)

      by Paul Jakma (2677)

      So what if you want to run memcached on a multi-user machine?

      It's slightly mad that software like this, which is designed without security, would use TCP per default, instead of local Unix sockets (access to which can be controlled with standard Unix filesystem permissions on the containing directory (careful about relying on permissions on the socket itself have any effect - not portable)). Indeed, it doesn't even seem to support Unix sockets (would be a trivial patch though).

      • by MikeFM (12491)

        Don't run it on a multi-user machine. That's why we invented server farms and virtualization. Nothing should be on a multi-user machine.

        • by Culture20 (968837)

          Nothing should be on a multi-user machine.

          Then Unix shouldn't be multi-user. Somehow I think there are practical issues with that sentiment.

  • Yeah like... all these open a socket to listen to, and man, if somebody logs into it, they're like, owning your server! Geeze, what where mysql/ssh/postgres/ftp/etc. thinking?!!!
  • Read my blog about it here:

    http://blog.codesherpas.com/on_the_path/2010/08/securing-memcache-in-2-minutes.html [codesherpas.com]

    This isn't a security problem - this is operating by design. If you are a memcache user and this is news to you, you need to read more about the tools you are using. I bet you have security problems beyond this one.

  • This whole conversation is stupid.

A CONS is an object which cares. -- Bernie Greenberg.

Working...