Cache On Delivery — Memcached Opens an Accidental Security Hole 149
jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."
Firewall? (Score:4, Insightful)
Re:Firewall? (Score:5, Insightful)
My memcached server is on the private network only accessible to other servers and is firewalled to everything but the servers that need access. Not exactly rocket science.
Re: (Score:2)
Since you run memcached maybe you can answer this. From what I've read it seems like an alerted (or just savvy) admin could easily detect this kind of futzing and (if you're lucky just) inject honey directly onto your machine. Attempting to maliciously write to memcached seems equally problematic. You'd never know if you were successful or just seeing what someone wanted you to see. How off base am I?
Re:Firewall? (Score:5, Insightful)
Ive been running memcached since it's out, even sent some patches in.
The thing is, why aren't they running this on a private network ?? Memcached is designed to be fast AND non-secure, to be run on your local network. Running it on a server farm with thousands of people having access to your computers and ips is not a private network.
I had heard about people running it on the local interface and still getting problems before (somebody else with the same computer ran it too and forgot to pick the good port and finally used the same key ...) but that's because IT'S NOT BUILT TO BE USED ON AN UNSECURED NETWORK.
Nothing new, bad admins get bad things done to them, move along.
Re: (Score:2)
Memcached is designed to be fast AND non-secure, to be run on your local network
My entire training on memcached is a 45-minute LUG presentation I went to a couple years back, and even then this was readily apparent.
The affected websites need to review their security personnel and procedures.
Re: (Score:2)
Re: (Score:2)
I doubt most admins look and if memcached was open it's virtually a feast of access. You wouldn't necessarily find it easy to jump from memcached to the rest of the system but you could gather a lot of information and alter data going to users and internal systems. Depending how memcache is being used it could be a pretty big exploit. The hardest part would be finding the right data keys. All mine are md5 hashes of a per site secret token combined with the given key so pretty difficult but I'm sure the kind
Re:Firewall? (Score:5, Interesting)
Yeah, slide 52 (paraphrased) is as follows:
I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?
Re: (Score:2, Insightful)
I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?
Re: (Score:2)
All the cool kids are using 'pf' these days: https://cipitunk.wordpress.com/2007/07/07/ipfw-vs-pf/ [wordpress.com]
Mastodoniasis (Score:2)
Re: (Score:1)
that said, i'll stick to "firewall" next time.
Re: (Score:2)
* ar
* cp
* df
* ls
* mv
* ps
* rm
* sh
* tr
M'kay?
Re: (Score:2)
that was due to the slow rate of terminals at ~8 chars/second back in the day. copy would have taken you 2 times as long to type, and don't forget each char had to be sent to the computer and then sent back to your display. meaning typing copy took at minimum 1 second at 8 baud.
Re: (Score:2)
Re: (Score:2)
I have mod points, but the +1 troll modifier seems to be missing.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE (Score:1, Funny)
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
Version 3, August 2010
Copyright (C) 2010 Anonymous Coward
Everyone is permitted to copy and distribute verbatim or modified copies of this license document, and changing it is allowed as long as the name is changed.
DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE
TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION
0. You just DO WHAT THE FUCK YOU WANT TO.
Re: (Score:1, Offtopic)
http://tunes.org/legalese/bugroff.html [tunes.org] [tunes.org]
Richard Stallman of the Free Software Foundation devised, in addition to some marvelous software, the GNU General Public License (GPL for short). Or the CopyLeft it is sometimes called.
It is quite a revolutionary document, using the "copyright" tool to to protect your right to use free software.
Unfortunately using copyright to protect free software is a lot like using a Jackal to guard the hens.
In fact, various inconveniences relating to this have resulted i
Re: (Score:1, Offtopic)
I fail to see why this is news (Score:5, Insightful)
Much less 'memcached' being at fault. They say it themselves:
All this is is stupid admins doing stupid things story and those are dime a dozen.
Re: (Score:2, Insightful)
This.
That's what you get for being cheap when hiring security team. Setting up memcached on a public IP or without a firewall is as bad as having your session directory fully writable on a public ftp (and quite similar too). ;-)
Memcached is good software and does exactly what it has been designed to do - provides fast key cache - but as with every tool if you are dumb you will hurt yourself. If those admins were carpenters they'd have an average of 4 fingers
Re: (Score:2, Insightful)
You don't need a security team, just a clue.
Re:I fail to see why this is news (Score:4, Insightful)
Re:I fail to see why this is news (Score:5, Insightful)
It defaults to not being installed and running. Memcached is meant to be ran from one or more caching servers (not really on the web server itself). It isn't really meant to be ran on localhost under ideal usage.
Re:I fail to see why this is news (Score:4, Insightful)
I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!
Re: (Score:3, Insightful)
It's not in any way the programmer's responsibility to hold the hands of idiots so they don't hurt themselves. The people who wrote this wrote a solid programs, and IN THE DOCS noted that if you set this up like this, you were an idiot.
Come on... we have far too much nanny-state as it is. My coffee has a warning on it that it's hot, my keyboard has a warning on it about RSI, my girlfriends hair strai
Re: (Score:2)
I forget who said it, but some comedian said we should remove the warnings from everything, and let nature sort it out.
I have a hunch it was George Carlin, maybe someone can find the video evidence.
In that case... (Score:2)
Perhaps utilities like rm should be rewritten to use -i (or similar) by default then? Or how about we ought to remove them outright since they might be used incorrectly?
Honestly, it's not up to the developer to make any such assump
Re: (Score:2)
"Any piece of software can be used "insecurely" if configured incorrectly"
Not that you don't have a point, but you are doing nothing but supporting the argument you tried to debunk.
Note that your point stands on the "if configured incorrectly" while the parent's is "it should be distrubuted on such a state that the sysadmin would be forced to do *any* kind of configuration prior for it to work" (thus giving the option to even "configure" it, either correctly or incorrectly).
I think the parent's point of vie
Re: (Score:3, Insightful)
No, you're just splitting hairs. My point is that software can be configured incorrectly and that it isn't the developers' fault if that happens. I believe the parent's point of view (and the general view espoused
Re: (Score:2)
I think that in your rush to rebuke the parent with scathing sarcasm, you have misinterpreted their point. I don't see anyone suggesting the software should install in a secure, perfectly functional manner, with no configuration whatsoever. The only quote I saw was saying that it's preferable for the software to be installed without a working configuration, so the administrator must explicitly configure it for their purposes. If they fail to configure it securely in the process, that's their own failing. Wh
Re: (Score:2)
Seriously? Your point is that Gentoo doesn't do enough hand-holding? This is a distribution that has built its reputation around NOT doing excessive hand-holding.
That having been said, Gentoo does leave services that have security implications unconfigured, but I'm having trouble understanding how memcached is such a service. This isn't a system configuration issue, but a network layout issue.
Finally, there is this:
The admin may not be aware that memcached is not designed to be exposed directly to the internet.
Huh? You're administering an internet-facing server, and installing an internet-facing se
Re: (Score:2)
True, it's an unusual network configuration to be "exposed by default"; however I still haven't seen any rational explanation why "listen on all interfaces" is a better default than "listen only on loopback".
If you're that oblivious, how the hell is making you fill out a config file going to help you?
Because the config file section that needs to be modified has comments explaining the risk? Not everyone is going to go and read external documentation on a piece of software before installing it. If you're going to do that, why even use a distribution in the first place? Any documentation you read is l
Re: (Score:2)
I can think of a number of network-facing software packages in the *nix world that are usually configured, by default, to listen on all interfaces, because that's usually what the end user wants. There are some particular issues I'd like to point out, but if you're feeling exceptionally annoyed you may
Re: (Score:2)
Insecure is in the eyes of the beholder. If the machine is on a firewalled net
Re: (Score:2)
It's pointless to design for the biggest idiots you can find because they keep designing better idiots. Memcached isn't an app for average users. It's a programmers tool. If people that should know better are still idiots then let the damage be on their heads. Why waste resources trying to save idiots from themselves.
I say we let people self destruct early so that we can stop wasting resources. Let bad developers get themselves fired, let drug users overdose, etc. Eliminate the chaff instead of coddling eve
Re:I fail to see why this is news (Score:5, Insightful)
Which is exactly the point. The default install should never be working-and-insecure. It should be secure, and ideally it should be working. If it is not possible for the default install to be both useful and secure, as appears to be the case with memcached, then it should install only listening on localhost and require explicit intervention by the user to accept connections from other hosts.
If you can install it and have it work by default, then there is no reason for the user to bother reading the manual, so they won't learn that it needs to be specially configured to be secure. If the default is secure but not particularly useful, then the user needs to explicitly adjust the setting that makes it insecure, and in so doing needs to read the documentation explaining that this will make it insecure and how to mitigate it.
Re: (Score:2)
You secure memcached the same way you secure any cache: by restricting direct access. All you have to do is put a proxy between the memcached server/instance and the internet and the problem is solved. If you don't have command-level access to the memcached instance you can't mine it, period.
Many very large websites don't bother to do this, which is very bad security practice, as the slideshow clearly demonstrates.
The weakness isn't really with memcached. To say memcache is insecure is like saying your d
Re: (Score:2)
"You secure memcached the same way you secure any cache: by restricting direct access."
Which is exactly the parent's point. That's why he says it should default to listening only on 127.0.0.1: in order for the default install to restrict direct access.
Re: (Score:2)
Re: (Score:2)
With your philosophy, MySQL wouldn't be where it is.
Re: (Score:2)
Maybe the problem is that Linux has gotten to easy to use. We should go back to requiring everyone download source files and compile everything themselves. Call it an intelligence test.
Re: (Score:2)
More than that, once installed, it defaults to not being used at all.
You have to actively develop and deploy software which uses and trusts memcached before this is a security hazard.
In other words, while it might be nice if it was secure by default, you now need two groups of people to fuck up at the same time -- system administrators and software developers -- both of whom, as part of their job, are supposed to think about security.
Re: (Score:3, Informative)
# Specify which IP address to listen on. The default is to listen on all IP addresses # This parameter is one of the only security measures that memcached has, so make sure # it's listening on a firewalled interface. -l 127.0.0.1
Re: (Score:2)
Yes memcached defaults to only accepting connections from the local address. From memcached.conf
What you posted does not say the default is localhost. It says the opposite:
"# Specify which IP address to listen on. The default is to listen on all IP addresses "
Re: (Score:2)
Re: (Score:2)
For that distribution. Some distributions default to all available interfaces, thus the Memcached default.
This is also why it is important to verify software configurations before running them.
Re: (Score:2)
All this is is stupid admins doing stupid things story and those are dime a dozen.
I'll say. I have been looking at using memcached in conjunction with the mod_auth memcache module. I know it exists and have done very cursory investigation. I do not have a working prototype yet but I do know my predecessor built one a year or so ago. My very cursory glance already made it startlingly clear that this would be an issue, any competent sysadmin really should have spotted the same thing.
Re:I fail to see why this is news (Score:5, Insightful)
The difference is that in this case a non-retarded admin can secure things. With Microsoft products it often takes an act of God to secure them (the best security feature of a Windows system is a blue screen of death). And memcached isn't meant to be a public service. It's very plainly described as not being secure. Completely different than a service that is meant to be public such as web or email not being secure.
Re: (Score:2)
I'm sorry, but I don't see how a system with absolutely no security can somehow be intrinsically more secure than a system without security.
The same security rules apply no matter what systems you use. If you're too stupid to understand basic security practices, then even Microsoft can't save you (though they do at least try).
Food Analogy (Score:3, Funny)
Think of it like this:
System that is never intended to be secure: plastic apple with a warning label stating "THIS IS NOT FOOD"
System that should be secure, but isn't: apple full of worms
You're not going to have a good experience biting into either apple, but there's definitely a difference in the expectations that someone would have when looking at them.
Re: (Score:3, Interesting)
netstat -lpn seems simple enough. I tend to run it every time I change something in a configuration file of a network-enabled service, just to be sure. It would be irresponsible to do otherwise.
Re: (Score:2)
"netstat -lpn" lists a lot of stuff that is not exported, you can get closer with "netstat -tulpen" or whatever, but even then you only have a very very rough overview about what you are exporting. For example you could have samba (or apache or whatever) running, which you want, but have it export all of '/' instead of just the directory you want. There is no easy way to find that out without digging through a lot of config files.
Re: (Score:2)
extremely crappy in communicating what they export to the world,
If you're an admin -- particularly one for, say, PBS -- and you're setting up a server, don't you think it'd be worth asking how that server does its communication? And it's not like this is undocumented -- from the wiki page on setting up a server [google.com]:
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.
And maybe I am just "clever", but this was pretty much obvious to me within about two minutes of learning of memcached -- I'm amazed anyone actually ran it this way. Sorry, but this isn't a case of poor UI design -- if you're employed as a system administrator, i
Re: (Score:3, Informative)
http://code.google.com/p/memcached/wiki/NewConfiguringServer
Networking
By default memcached listens on TCP and UDP ports, both 11211. -l allows you to bind to specific interfaces or IP addresses. Memcached does not spend much, if any, effort in ensuring its defensibility from random internet connections. So you must not expose memcached directly to the internet, or otherwise any untrusted users. Using SASL authentication here helps, but should not be totally trusted.
From their wiki page detailing how to configure a new server. Surely the part they highlight in bold should have raised a flag to even the dumbest administrator.
Here's an idea that won't impact performance: At startup, issue a big multi-line warning if the IP addresses that are getting bound aren't on a Private Internet [faqs.org]:
The Internet Assigned Numbers Authority (IANA) has reserved the
following three blocks of the IP address space for private internets:
10.0.0.0 - 10.255.255.255 (10/8 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)
Re: (Score:2)
Re: (Score:2)
The MS way usually is to pretend that it's easy. The docs consist of a leaflet that explains how to open your cd drive and place the cd the right side up, and how to handle the mouse to make the cursor move to the "Ok" button.
Can you link to an example of this ?
A few clarifications (Score:5, Informative)
Re:A few clarifications (Score:5, Interesting)
Man pages help, but when the defaults don't aid developers we need to a rethink both of the software (memcached) and the systems were it's not running securely (cloud platforms).
Re: (Score:2)
The deeper issue that's exposed is that your mythical developers are ignoring the fact that most attacks come from inside. Mostly through rouge employees or compromised machines. They are operating on the fallacy that you can have one big happy subnet and everybody attached it is a team player.
I don't use memcached but it sounds like one should have your memcached server machines on their own LAN and multiple network cards in the client machines.
It's really not that difficult, it's just :
1 developers are no
Re:A few clarifications (Score:5, Funny)
Luckily, they often get caught red-handed.
Re: (Score:2)
I refuse to give even the boss access. I've even designed systems that wouldn't give me access except under very specific conditions such as a second admin also entering their credentials from a known system during business hours. Some data needs to be treated with extreme care.
Re: (Score:2)
Terry? Is that you?
Admin or distro? (Score:5, Interesting)
Debian's default config says:
# Specify which IP address to listen on. The default is to listen on all IP addresses
# This parameter is one of the only security measures that memcached has, so make sure
# it's listening on a firewalled interface.
-l 127.0.0.1
Are there any distros that don't have it locked down by default? I would hope not, but if something has it insecure out of the box with no warning that might explain it... (though a good sysadmin would firewall all internal services, whether the documentation tells them to or not)
Re: (Score:3, Interesting)
So what if you want to run memcached on a multi-user machine?
It's slightly mad that software like this, which is designed without security, would use TCP per default, instead of local Unix sockets (access to which can be controlled with standard Unix filesystem permissions on the containing directory (careful about relying on permissions on the socket itself have any effect - not portable)). Indeed, it doesn't even seem to support Unix sockets (would be a trivial patch though).
Re: (Score:2)
Don't run it on a multi-user machine. That's why we invented server farms and virtualization. Nothing should be on a multi-user machine.
Re: (Score:2)
Nothing should be on a multi-user machine.
Then Unix shouldn't be multi-user. Somehow I think there are practical issues with that sentiment.
Re: (Score:2)
From http://linux.die.net/man/1/memcached [die.net] :
-l
Listen on ; default to INDRR_ANY. This is an important option to consider as there is no other way to secure the installation. Binding to an internal or firewalled network interface is suggested.
yeah ... it's hard to blame the memcached guys
Re:Admin or distro? (Score:5, Insightful)
default to INDRR_ANY
And this is why they're to blame. Default should be the loopback, and enabling external access should require explicit configuration.
Re: (Score:2)
Sorry, I don't agree with that.
Memcached is not meant for single-server configurations. It's meant to combine the memory of all connected servers, so that if you connect 10 servers with 4GB each, you get 40GB of cache. This won't work if you only allow loopback connections.
The admin should block memcached connections on the outer interface. And set the default firewall policy to DENY to start with.
Re:Admin or distro? (Score:4, Insightful)
Memcached is not meant for single-server configurations
That's silly, it's a generic object store. There's no reason not to use it to cache expensive local operations. Of course it shines across a farm of caches, but the server mapping hash will work just fine with one machine.
If you're a startup with just one webserver and starting to hit performance problems, memcached will likely buy you a few more months.
Going from one server to two is hard, three is a bit more work, and after three it's roughly all the same until you start adding more data centers and then it's all the same until you're Facebook. Taking on that 'hard' expense too early would be a poor allocation of resources.
Re: (Score:2)
That's why you create your own packages and have a system to add your mods to the public versions of the packages. I don't custom compile everything but the components most key to what I'm doing are often custom compiled because the public packaged lack the functionality I need.
mysql/ssh/postgres/ftp/etc. opens security hole (Score:2, Insightful)
Re: (Score:2)
..and presumably a developer would notice when at some point they write connect_to_memcached() and don't need to pass in a username or password..
Re: (Score:2)
Yeah, if it was a web developer, they'd notice and would then think, "Shit, I wish PostgreSQL was like that. It's such a pain in the ass having to give it a username and password."
Right... because if someone exploits a security bug in your app (I know... your apps have no bugs) and is able to 'drop table users;' because the DB had no security mechanism, that would really rock. Instead, we have to suffer the tedium of only granting the privileges that the app needs and locking down all other functionality.
Secure mamcached with the '-l' option on startup (Score:2)
Read my blog about it here:
http://blog.codesherpas.com/on_the_path/2010/08/securing-memcache-in-2-minutes.html [codesherpas.com]
This isn't a security problem - this is operating by design. If you are a memcache user and this is news to you, you need to read more about the tools you are using. I bet you have security problems beyond this one.
Let me be the first to say (Score:2)
This whole conversation is stupid.
Re:Let me see if I understand this (Score:5, Informative)
Memcache's one purpose in life is to be as fast as possible. It makes perfect sense for it to drop the overhead of authentication and leave it on the server operator's head to not make it publicly accessible. It's not rare to strip out MySQL's authentication layer (and presumably the same for other DBs) for a speedup when your DB server is sitting behind a firewall.
Re: (Score:2, Insightful)
I've dealt with a lot of large production databases, Oracle, SQL Server, MySQL, PostgreSQL, and similar. I've never seen the authentication layer striped out. These were pretty performance hungry applications as well, though perhaps not that many quick connections, but still the overhead is so small, that when you're communicating internally, it's generally not a problem. For companies like Google I could see this being a problem, but for MOST instances of these servers, I wouldn't suggest people do it.
At t
Re: (Score:3, Insightful)
It's open source, if you want to add a layer of protection, go away. For the rest of us, we are using memcached for maximum speed, no such thing in.
Lets say you add a layer of protection to your memcache, it now requires a password to get in. The hacker is in your network, so he can sniff those packets. Where's your protection ? Ok, lets say you add in some crypto to prevent sniffing attack. He's in your network, on your computers, reading your files. He then parse your code and grab the password.
So, what h
Re: (Score:2)
I don't allow random systems to communicate with my database. It is only accessed directly from localhost. Any other system has to go through a predefined RPC function that sanitizes inputs and outputs and does a very specific task. Even that access is only granted to a few systems and never over the public network. I usually still use the built-in security too but I would consider removing it as I consider it almost useless.
Re: (Score:2)
Re: (Score:2, Funny)
Re: (Score:2)
Yes, totally true. But it would still have been better to have authentication enabled by default. The user who actually knows what he is doing can then disable it.
Re: (Score:2)
Because of MySQL injection attacks and other problems, it is nice to have different access rights for different parts of your site. So I would not recommend stripping out the auth layer, especially if you have more than 1 DB running.
Even the best programmers and practices can create bugs.
Re: (Score:2)
I wouldn't recommend stripping out the authentication layer either, I merely said it's been done. That being said, if your app can suffer from SQL injection, you've got bigger problems. If your app is to the point of needing to strip out DB authentication for some extra performance (and if it is, you should probably be taking a different approach... such as memcache, the point of this whole discussion), you damn well better have all sorts of tests in place to make sure that you don't introduce problems tha
More Boiled and Distilled. (Score:5, Insightful)
Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?
Anyone can write on your underwear too, if you are stupid enough to wear it outside your pants.
Is that an underwear design flaw?
Re:More Boiled and Distilled. (Score:5, Insightful)
Best. Analogy. Ever.
Re:More Boiled and Distilled. (Score:4, Funny)
No. Car. Was. Involved.
Re:More Boiled and Distilled. (Score:5, Funny)
That's. Why.
Re: (Score:2)
No, they obviously have funny cartoon animals [tumblr.com].
Re:More Boiled and Distilled. (Score:5, Funny)
As spokesman for the Justice League, I say yes.
Re:More Boiled and Distilled. (Score:5, Funny)
That's actually more of a feature.
Re: (Score:1, Redundant)
Is that an underwear design flaw?
Well, kinda. That's why I don't use them.
LK
Re: (Score:3, Funny)
Boiled and distilled underwear....Ewwwww.
Re: (Score:2)
Tell that superman face to face and lets see how strong your point is :)
Re: (Score:2, Funny)
Jesus Tapdancing Christ, they explicitly say that in the doc(s) where they discuss design decisions. They can't use stunnel or blah or blah2 or blah3?
"It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?"
Yeah, they saw it and they saw their site and their systems and saw that they did not require that feature for themselves - they weren't creating memcache for charity to donate to the world at large - ffs. Say what you want about livejournal b
Re: (Score:2)
Flash for a freaking slide show? This is too easily doable without Flash. Even Javascript is overkill. And it requires Flash version 9 as if no earlier Flash version could do a slide show.
Re: (Score:2)
I don't understand, how is writing a JavaScript slideshow easier than hitting "Upload" and then "Embed" at SlideShare?
Seriously, you've got way too much time on your hands.
Just upgrade your fucking flash already, nobody else cares.
Re: (Score:2)
Sure, I used bugmenot:
http://s3.amazonaws.com/ppt-download/slaviero2010memcache-100803232447-phpapp02.pdf?Signature=CatPvvPgJL3nxkECUrqgQ%2ByH4OE%3D&Expires=1281288118&AWSAccessKeyId=AKIAJLJT267DEGKZDHEQ [amazonaws.com]
PS: I also have no Flash installed and felt your pain as well.