Cache On Delivery — Memcached Opens an Accidental Security Hole 149
jamie spotted this eye-opening presentation (here's a longer explanation) about how easy it is to access sensitive data on many sites using memcached, writing "If you already know what memcached is, skim to slide #17. The jaw-drop will happen around slide #33. Turns out many websites expose their totally-non-protected memcached interface to the Internet, including gowalla, bit.ly, and PBS."
Firewall? (Score:4, Insightful)
I fail to see why this is news (Score:5, Insightful)
Much less 'memcached' being at fault. They say it themselves:
All this is is stupid admins doing stupid things story and those are dime a dozen.
Re:Firewall? (Score:5, Insightful)
My memcached server is on the private network only accessible to other servers and is firewalled to everything but the servers that need access. Not exactly rocket science.
More Boiled and Distilled. (Score:5, Insightful)
Memcache allows anyone to overwrite a cache instance. Seriously? It does not authenticate a write to the cache? And they didn't see this as a problem when desgining memcache? Really?
Anyone can write on your underwear too, if you are stupid enough to wear it outside your pants.
Is that an underwear design flaw?
Re:More Boiled and Distilled. (Score:5, Insightful)
Best. Analogy. Ever.
Re:Firewall? (Score:2, Insightful)
I assume he means "firewalls" by "FW". Seriously, you can't even bother to spell out "firewall" in a presentation?
Re:I fail to see why this is news (Score:2, Insightful)
This.
That's what you get for being cheap when hiring security team. Setting up memcached on a public IP or without a firewall is as bad as having your session directory fully writable on a public ftp (and quite similar too). ;-)
Memcached is good software and does exactly what it has been designed to do - provides fast key cache - but as with every tool if you are dumb you will hurt yourself. If those admins were carpenters they'd have an average of 4 fingers
Re:Let me see if I understand this (Score:2, Insightful)
I've dealt with a lot of large production databases, Oracle, SQL Server, MySQL, PostgreSQL, and similar. I've never seen the authentication layer striped out. These were pretty performance hungry applications as well, though perhaps not that many quick connections, but still the overhead is so small, that when you're communicating internally, it's generally not a problem. For companies like Google I could see this being a problem, but for MOST instances of these servers, I wouldn't suggest people do it.
At the very least, for that extra micro-second of overhead, it means if some cracker gets into your network, you don't have to worry about that as much (not that you don't have to worry, hell, some crackers in your network!).
Also, wouldn't you at least want to restrict it from employees that shouldn't be fucking with it. Granted companies whose purpose it is to develop this application, likely want almost everyone having access, but most instances you wouldn't want this.
None the less, this is off topic, and the article is retarded.
mysql/ssh/postgres/ftp/etc. opens security hole (Score:2, Insightful)
Re:I fail to see why this is news (Score:0, Insightful)
Are there double standards on this site, depending on whether Microsoft are involved or not? You blame the admins here, but if it were Windows, I bet you'd blame Microsoft and not stupid users.
Re:I fail to see why this is news (Score:4, Insightful)
Re:Admin or distro? (Score:5, Insightful)
default to INDRR_ANY
And this is why they're to blame. Default should be the loopback, and enabling external access should require explicit configuration.
Re:I fail to see why this is news (Score:2, Insightful)
You don't need a security team, just a clue.
Re:Firewall? (Score:5, Insightful)
Ive been running memcached since it's out, even sent some patches in.
The thing is, why aren't they running this on a private network ?? Memcached is designed to be fast AND non-secure, to be run on your local network. Running it on a server farm with thousands of people having access to your computers and ips is not a private network.
I had heard about people running it on the local interface and still getting problems before (somebody else with the same computer ran it too and forgot to pick the good port and finally used the same key ...) but that's because IT'S NOT BUILT TO BE USED ON AN UNSECURED NETWORK.
Nothing new, bad admins get bad things done to them, move along.
Re:Let me see if I understand this (Score:3, Insightful)
It's open source, if you want to add a layer of protection, go away. For the rest of us, we are using memcached for maximum speed, no such thing in.
Lets say you add a layer of protection to your memcache, it now requires a password to get in. The hacker is in your network, so he can sniff those packets. Where's your protection ? Ok, lets say you add in some crypto to prevent sniffing attack. He's in your network, on your computers, reading your files. He then parse your code and grab the password.
So, what have you been able to achieve exactly ? Nothing beside slow downs everyone downs. People should just learn to use the tools at their disposal. And yes, you can kill yourself with a car. Should we lock them all down to 10mph ?
For the parent, here's memcached help. Notice that even if it's ADRANY, you have it in your face should you be on an unsecure network. And besides, locking it to your ip won't prevent someone inside your network from getting the content.
memcached -help
memcached 1.2.6
-p TCP port number to listen on (default: 11211)
-U UDP port number to listen on (default: 0, off)
-s unix socket path to listen on (disables network support)
-a access mask for unix socket, in octal (default 0700)
-l interface to listen on, default is INDRR_ANY
-d run as a daemon
-r maximize core file limit
-u assume identity of (only when run as root)
-m max memory to use for items in megabytes, default is 64 MB
-M return error on memory exhausted (rather than removing items)
-c max simultaneous connections, default is 1024
-k lock down all paged memory. Note that there is a
limit on how much memory you may lock. Trying to
allocate more than that would fail, so be sure you
set the limit correctly for the user you started
the daemon with (not for -u user;
under sh this is done with 'ulimit -S -l NUM_KB').
-v verbose (print errors/warnings while in event loop)
-vv very verbose (also print client commands/reponses)
-h print this help and exit
-i print memcached and libevent license
-b run a managed instanced (mnemonic: buckets)
-P save PID in , only used with -d option
-f chunk size growth factor, default 1.25
-n minimum space allocated for key+value+flags, default 48
Re:I fail to see why this is news (Score:5, Insightful)
The difference is that in this case a non-retarded admin can secure things. With Microsoft products it often takes an act of God to secure them (the best security feature of a Windows system is a blue screen of death). And memcached isn't meant to be a public service. It's very plainly described as not being secure. Completely different than a service that is meant to be public such as web or email not being secure.
Re:I fail to see why this is news (Score:5, Insightful)
It defaults to not being installed and running. Memcached is meant to be ran from one or more caching servers (not really on the web server itself). It isn't really meant to be ran on localhost under ideal usage.
Re:I fail to see why this is news (Score:4, Insightful)
I believe the point here is that software designers should assume that in terms of security their users are complete idiots and WILL install and setup the program in an unsecure manner unless they are specifically beat over the head with the notion that what they are doing is BAD!
Re:I fail to see why this is news (Score:5, Insightful)
Which is exactly the point. The default install should never be working-and-insecure. It should be secure, and ideally it should be working. If it is not possible for the default install to be both useful and secure, as appears to be the case with memcached, then it should install only listening on localhost and require explicit intervention by the user to accept connections from other hosts.
If you can install it and have it work by default, then there is no reason for the user to bother reading the manual, so they won't learn that it needs to be specially configured to be secure. If the default is secure but not particularly useful, then the user needs to explicitly adjust the setting that makes it insecure, and in so doing needs to read the documentation explaining that this will make it insecure and how to mitigate it.
Never underestimate the ignorance of Web 2.0 devs. (Score:1, Insightful)
Web 2.0 is an interesting phenomenon. In many ways, it's nothing but embracing ignorance, bad ideas, the worst possible way of doing things, and the stupidest people (both users and developers!) around.
First, let's ignore how Web 2.0 sites cater to the stupidest users, and because of this are willing to abuse private data given to them by their users. That's a whole different story. Let's just focus on the technology here.
The first thing we see is that the use of PHP is prevalent among Web 2.0 sites. If there's one thing that PHP is "good" for, it is quickly developing poorly-performing and insecure web applications, written by programmers with a minimum of technical knowledge. That's why it has blossomed, so to speak, among the Web 2.0 development community. It caters well to their dislike of quality and their lack of technical knowledge. In many cases, the developers are too ignorant to know that there are better options out there, let alone identify the flaws and problems with their own code.
The second thing we see is a lack of knowledge about data storage and scalability. This has manifest itself as the rise of the NoSQL movement. NoSQL is all about throwing away decades of accumulated knowledge concerning relational databases, and replacing it with hash tables. It's about embracing ignorance.
Many of the most popular Web 2.0 sites moved to NoSQL "databases" claiming it was necessary to do so for "scalability reasons", but that's obviously bunk after reading the many articles they publish describing their architectures. We find out that they're clueless about such basic things as indexes, and in some cases even perform joins by pulling ALL of the data back to their PHP code, and manually joining it there.
Worse yet, these Web 2.0 sites are prone to failure. Twitter's "fail whale" has become widely known, and reddit suffers from frequent downtime, including a lengthy incident just yesterday. What's funniest, however, is that the traffic they face is minuscule compared to the activity that many POS and financial systems, or even "Web 1.0" sites, handle with ease because they're developed by knowledgeable developers who know how to properly use relational databases.
Given that the Web 2.0 developers thrive on ignorance and doing things as incorrectly as possible, it's no wonder some of their flagship technologies and sites would have such glaring security holes. That's just what's to be expected from people who don't have a fucking clue about what they're doing.
Re:I fail to see why this is news (Score:1, Insightful)
From the description (and the memcache "NewConfigurinServer" instruction excerpts) that have been posted here, it looks like memcache is a knife in a box that says "do not handle by blade" on the outside. It comes sharp, but there's a warning. If you're smart, you either know not to wrap your fingers around the blade, or you read the warning and you learn that you shouldn't hold it that way. If you're dumb, you ignore the warning and reach into the box, and wrap your (remaining) fingers around what you find most convenient. I suppose they could also blunt the knife and force you to sharpen it first, but I can only have so much sympathy for those who ignore warnings and cry later when it is harder to play piano.
In the case of a big public site, if you're a professional that gets paid to set these things up, knowing what should be on the inside of the firewall and what should be on the outside of the firewall has to be job 1, and Rule #1 that goes with that should be "it goes on the inside." Rule #2 should be "see Rule #1." Rule #3 is, "are you absolutely, positively, stake your reputation, your job, you life on it sure it needs to be exposed to the outside? Still, unless the site fails to operate, see Rule #1." Rules 1 & 2 are also a good for setting up a home network, too.
Re:mysql/ssh/postgres/ftp/etc. opens security hole (Score:1, Insightful)
Yeah, if it was a web developer, they'd notice and would then think, "Shit, I wish PostgreSQL was like that. It's such a pain in the ass having to give it a username and password."
Re:I fail to see why this is news (Score:3, Insightful)
It's not in any way the programmer's responsibility to hold the hands of idiots so they don't hurt themselves. The people who wrote this wrote a solid programs, and IN THE DOCS noted that if you set this up like this, you were an idiot.
Come on... we have far too much nanny-state as it is. My coffee has a warning on it that it's hot, my keyboard has a warning on it about RSI, my girlfriends hair straightener has a warning on it that it can burn you because it's hot....do we really need to extend this to software?
I forget who said it, but some comedian said we should remove the warnings from everything, and let nature sort it out. Same for software. If you're not smart enough to run a website, perhaps you need a painful pointing out of that.
Re:Admin or distro? (Score:4, Insightful)
Memcached is not meant for single-server configurations
That's silly, it's a generic object store. There's no reason not to use it to cache expensive local operations. Of course it shines across a farm of caches, but the server mapping hash will work just fine with one machine.
If you're a startup with just one webserver and starting to hit performance problems, memcached will likely buy you a few more months.
Going from one server to two is hard, three is a bit more work, and after three it's roughly all the same until you start adding more data centers and then it's all the same until you're Facebook. Taking on that 'hard' expense too early would be a poor allocation of resources.
Re:In that case... (Score:3, Insightful)
No, you're just splitting hairs. My point is that software can be configured incorrectly and that it isn't the developers' fault if that happens. I believe the parent's point of view (and the general view espoused further up the comment chain) was that software should be distributed in such a manner that it cannot be configured insecurely. I'm hoping I just misunderstood what you wrote and that you were simply disagreeing for the sake of, well, disagreement. Or trolling. Regardless, I'd encourage you to read up the chain so you may better understand my point (the parent was just convenient fodder since his view was mentioned numerous times, and I got fed up with this notion that it's the responsibility of the developers to ensure no configuration can be abused).
N.B. that if I were to split hairs, I'd point out that your comment that implies (I think? You may have missed a word or two while typing it out) that sysadmins shouldn't be forced to do anything to make a particular software install work. That's a fairly short order--almost all software works out of the box as expected, but whether it works securely across all "default" installs is another matter entirely and one that, again, is the responsibility of the user to verify. But the point goes far beyond that, and I'm somewhat disappointed that you seem to agree with the general sentiment that post-install configuration should almost never happen except in rare outlying circumstances (hint: this isn't likely to happen, particularly with the vast array of requirements, disparate hardware, and so forth). Note, also, that if this notion of "secure by default" is taken to the extreme, you wind up with OpenBSD. OpenBSD is secure by default. You also cannot use it without installing and configuring additional software (thus no longer having a "default" install), because it comes with only OpenSSH installed by default! In other words, we're just encountering a difference of philosophies, and I ascribe to the one that makes the burden of configuration your responsibility as a user, not mine as a developer.
And really, this point it doesn't matter at all, because someone will still find a way to configure it incorrectly, regardless of how well it was written (don't believe me? you'd be surprised what lurks out there in the wild) or how strict the default configuration is. There are people out there who will tinker with things, including things they may not fully understand, and configure it in a manner that is suddenly insecure. That is my fundamental point; the point of the OP and those subsequently up the chain is that the responsibility should rest upon the shoulders of the developers to ensure their software cannot be configured incorrectly--that's the whole point many of the posts here are arguing against Memcached for! There seems to be so much heartache that Memcached (which doesn't require authentication) can be configured in a manner that it exposes sensitive data to the Interwebs. Guess what? If it does, that is the fault of the administrator and not the fault of Memcached or its developers.
I'm not sure how else to state that more clearly. But I'll try:
Responsibility rests on the person who installs the software to ensure it is configured securely for their needs, not the developers nor the package maintainers. The only point in time, IMO, where such responsibility is the developers' is when it becomes an issue of out-of-band behavior; e.g. something that is unexpected and unintended (a bug or security flaw) and not fundamentally something that is the result of a misconfiguration. In short:
Misconfigurations = user's responsibility (even if t