Attack On a Significant Flaw In Apache Released 203
Zerimar points out a significant flaw in Apache that can lead to a fairly trivial DoS attack is in the wild. Apache 1.x, 2.x, dhttpd, GoAhead WebServer, and Squid are confirmed vulnerable, while IIS6.0, IIS7.0, and lighttpd are confirmed not vulnerable. As of this writing, Apache Foundation does not have a patch available. From Rsnake's introduction to the attack tool: "In considering the ramifications of a slow denial of service attack against particular services, rather than flooding networks, a concept emerged that would allow a single machine to take down another machine's web server with minimal bandwidth and side effects on unrelated services and ports. The ideal situation for many denial of service attacks is where all other services remain intact but the webserver itself is completely inaccessible. Slowloris was born from this concept, and is therefore relatively very stealthy compared to most flooding tools."
So slashdot... (Score:5, Funny)
Re:So slashdot... (Score:5, Informative)
We have a hardware load-balancer and a software reverse proxy (varnish) in front of our apache.
I kinda doubt this would work on us.
Note, I am not inviting anyone to try. It might work great for all I know :(
Re: (Score:2)
I was wondering the same thing as well. I run a small site with varnish on the front end. Anyone know how attacks like this work against servers behind a forward proxy?
What about ... (Score:2, Funny)
WTH? This is an absolutely trivial attack (Score:4, Insightful)
It's just holding sockets open; that's the "Hello, world!" of DOS attacks.
I'm finding it hard to believe that Apache is genuinely vulnerable to this. Did nobody see it coming? For real?
Re: (Score:3, Informative)
No, it's not. It's holding an HTTP session open. That is not the same thing as a TCP socket.
Re:WTH? This is an absolutely trivial attack (Score:5, Informative)
Let me make this clearer for those that aren't very technical: It's holding an HTTP session open and Apache has a limited number of simultaneous HTTP sessions.
All someone has to do is send about 100 requests to your website and leave them open without sending any further information. Nobody else will be able to connect to your web server for a long time. The weekend is coming, so I'm expecting lots of downtime for government sites in the next couple of days...
Re:WTH? This is an absolutely trivial attack (Score:4, Informative)
A simple connlimit declaration in IPTables shuts this down fairly easily...
Re:WTH? This is an absolutely trivial attack (Score:4, Informative)
Not really, it just means you need more than one attacker.
Re: (Score:3, Interesting)
A simple connlimit declaration in IPTables shuts this down fairly easily...
Will that work for anyone using load balancers (read: people with sites worth hitting)?
Re:WTH? This is an absolutely trivial attack (Score:4, Interesting)
No, that won't work. Apache will drop connections which aren't making "useful progress".
However, it's definition of "useful progress" is flawed -- you can keep sending HTTP headers, and it will keep the connection open. You only have to send one every few seconds, so it's a very low bandwidth DOS attack.
Attack of a 1000 snails. (Score:2, Informative)
This type is already known as "attack by a 1000 snails" type attack. It is harder to defned against than you would think. A user can be slow, but coders are hesitant to drop users that ar too slow or too fast.
A user kan just keep the TCP/IP alive by sending one byte every x seconds. If this is patched at http header level, you will see you can do the same kind of attack on the application, that can have limited php or perl sessions.
Re: (Score:3, Interesting)
Yes, I agree. I've seen a handful of attacks like this over the years. Maybe not exactly this one, but Apache has been vulnerable to this for years and I thought all webservers were. And I thought people just knew about it already. This one is a tough one to fix, its not like Apache can just patch something. It sounds like an architectural change is needed.
Why not IIS? (Score:4, Interesting)
Why isn't IIS vulnerable? Does it just assume the headers are done after some amount of time? Does it have a limit to the number of headers it accepts?
Can this even be fixed without technically breaking the protocol (since it sounds like what's going on is correct behavior, theoretically)?
Re: (Score:3, Interesting)
Re:Why not IIS? (Score:5, Insightful)
Does the HTTP spec say anything about the server application timing out the connection? Seems like reasonable behaviour to me. I would be surprised if this isn't a configurable option in Apache too.
People love to hate it, but IIS has matured in to a very good web server. It's my choice over Apache.
Re: (Score:2)
Yes, but which one?
Re: (Score:2)
It may be that once they hit 90% of the session limits, that it starts killing off the longest-idle sessions.
Re: (Score:2)
no Apache developer believed that any self respecting attacker would stoop so low.
That's a brilliant approach to security.
Re: (Score:2)
I'm further thinking that since this is such a stupidly trivial attack, that it was all that MS could think to defend against, and that no Apache developer believed that any self respecting attacker would stoop so low.
Or maybe they're relying on the attacker correctly setting the evil bit as per the specification [ietf.org], while Microsoft is flagrantly ignoring the spec as usual.
Re:Why not IIS? (Score:5, Funny)
If the vulnerability is based on correct, standard conform behaviour of the server, I can see why IIS isn't susceptible to it.
Re: (Score:2, Informative)
More likely IIS survives because it uses a worker pool threading model (no thread/process is dedicated to a connection, so a connection only takes up memory for the state, not for the thread).
Apache had, and probably still has, a process/thread-per-connection model.
So with all due respect, it looks like a proper design decision is what is protecting IIS here:
http://www.kegel.com/c10k.html
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/a63ee1c2-04d6-44dc-b4d6-678eb3117bf9.mspx?mfr=
Re: (Score:2)
Is it really correct and standard to hold a session open for a client that isn't sending any data, while excluding other clients in the process? Where in the spec does it say to do that?
(Un)fortunately, lighttpd is not vulnerable (Score:2)
I might subscribe to your theory if it weren't for the fact that lighttpd [lighttpd.net], which is a first-rate open-source web server, is explicitly listed as not vulnerable.
Re:Why not IIS? (Score:5, Informative)
unless you are using Session()'s in asp in IIS then one thread in IIS handles multiple connections.
what this is doing is opening a connection (getting a thread to work it) and holding it open (keeping the thread busy) and just keep asking for new ones.
it is very common (always i think) for Apache and allot of web servers to have a max thread's so that the site under heavy traffic doesn't open more connections than it can handle.
where IIS also has a worker thread limit - there is no limit *(you can set one - but not on by default) on how many concurrent connections can be managed by a thread (and new incoming connections are passed to the thread with the lowest current work load - not always the one with less connections)..
if you do what they are doing here i can see IIS behavior would be to slowly pile all these slow - no work connections into one thread and the others would happily go about doing actual work..
where apache would slowly lose access to workable threads as this keeps them busy.
this isn't an exploit on the http or tcp protocol - it is an exploit based on the behavior of the web server based on it's best practices for managing it.
Re: (Score:2)
Makes sense.
But wouldn't you run into the connection limit of the OS at some point? Or is that just way too high to be a practical problem (say 16 million)?
Re: (Score:2)
well the connection limit for a single host is the number of available ports for outbound traffic (to return data to the client)
by default (from memory here) i want to say IIS is set up to use ~4k ports for outbound - i do know that can be changed to allow ports 1024+ to be used meaning the number of avaliable ports would be ~64.5k
and that is per host ip - and unless you have the site bound to a specific host ip address (instead of using site headers - iis will respond on an alternate ip (if it has one) whe
Re: (Score:2)
How would that be of any use? The client on the far end is directing incoming traffic based on source IP (among other characteristics). TCP packets that arrive from random IPs are discarded.
Re: (Score:2)
the incoming traffic is always going to IP:80
the return traffic from IIS is going coming from avaliable Ips':port
if you bind a website in IIS to an ip instead of "any avaliable ip" then the return tcp connection is forced to be sorced from the ip that is also the reciving
if you have it set to "any avaliable ip" and the box has 2 ip's your client may request data on IPA:80 and get a reply from IPB:port
there for your avaliable ports for reply (limiting the max connection's) would be MaxPorts* Available IP's.
g
Re:Why not IIS? (Score:4, Informative)
If a client sends a SYN to 10.1.1.1:80 and gets an SYN-ACK from 10.5.5.5:80 then the client will not associate the two as being related, and will keep waiting for a response from 10.1.1.1:80 until timing out.
You would need to have some sort of DNS arrangement that encouraged clients to make their requests to your various IPs. You can't just respond from a different IP than the client contacted.
Re: (Score:2)
well the connection limit for a single host is the number of available ports for outbound traffic (to return data to the client)
A TCP connection is identified by the source IP:port and destination IP:port. Your web server typically will listen on only one port (80) and probably on a single IP, and TCP has 64k ports available. So, the theoretical limit is 64k connections from each and every computer with a routable IP address. Which is completely insane, so what really matters are the limits your OS puts on how many open sockets/file descriptors there can be (per process, or across the entire system), and if that's high enough then t
Re: (Score:2)
A TCP connection is identified by the source IP:port and destination IP:port. Your web server typically will listen on only one port (80) and probably on a single IP, and TCP has 64k ports available. So, the theoretical limit is 64k connections from each and every computer with a routable IP address.
Not really. Yes, the server application listens at a specific IP:port - typically 80 in case of HTTP. However, when you accept() a client connection on the socket you're listening on, you get a new socket for that connection, which is associated with a new, unique port on the server. So you can only have as many client connections on the server as you have ports.
Re:Why not IIS? (Score:5, Funny)
My guess is that the DoS attack is so slow that, by the time it would have completed, the server has already crashed for a different reason.
Re: (Score:2)
Boring (Score:5, Insightful)
Re: (Score:2)
Re: (Score:2)
I'd be embarrassed to publish something like this....
Says the Anonymous Coward
Re: (Score:2)
fairly trivial to fix ...
I'd be embarrassed to publish something like this
So why isn't it fixed? Let me guess: it's a case of all of the Apache developers saying "you have access to the code, you fix it, it's trivial."
Re: (Score:2, Interesting)
No, no they wouldn't.
However like what normally happens whenever some Linux exploit comes out. It's something ridiculously trivial to fix and the windows crowd on slashdot lap it up like it is an "end of the world" scenario. Congratulations on continuing the trend.
iptables helps (Score:5, Informative)
You can have perlbal or any reverse proxy on the same machine but listening on a different port and then use iptables to redirect like this
# iptables -t nat -A -PREROUTING -d ! 127.0.0.1 -p tcp -m tcp --dport 8080 -j REDIRECT --to-ports 80
and then you don't need to change your apache configuration - and having apache listen on a different port to what users see can break some scripted sites if they read the port number from the apache config.
Seems to be a general problem. (Score:5, Insightful)
And the only resolution right now that I can see is to have a connection timeout.
At least the problem is a denial of service problem and not a problem with intrusion so the damage is easily rectified - restart the web server. Not that you really want to restart it.
And I suspect that other services can be vulnerable to this type of attack too, not only web servers.
Re: (Score:2)
And you get raped again as soon as it comes back up. How does restarting it help?
Re: (Score:2)
Re: (Score:2)
Fairly slow has been under a minute on most of the servers I tested it on.
Fortunately my servers are configured to be able to run either lighttpd or apache. (in case I have a problem with one.)
So I can pick my poison.
Re: (Score:3, Interesting)
A connection timeout should be fine. Just start the clock upon accept(). Give the client a generous but limited amount of time to send headers. If the timer expires before the empty line is received, close the connection.
Bonus points for not getting the thread pool involved until the header is complete.
Extra credit for a config option to send a flood of junk to the client and THEN close the socket. That could make attackers considerably more visible to their upstream provider.
Re: (Score:3, Interesting)
Nothing like replyoing to yourself.
Double extra credit if the junk you send back looks enough like downloading music that the RIAA accidentally joins the forces of good and comes down on the attacker due to ISP snooping but not enough like downloaded music to get you actually busted.
Re: (Score:2)
the damage is easily rectified - restart the web server
Good idea, mitigate a DoS attack by taking the server offline.
Other Web Servers....Proxies......? (Score:2)
Re:[Sounds Stupid] (Score:3, Interesting)
Re: (Score:2)
How long would you set the timeout?
Let's say it's 3 seconds (if you want to support crappy dialup connections from South Africa, that sounds about right). That means that an attacker can block out your server for 3 seconds... at a time. As soon as you kill his connections, he just recreates them anew. Or, better yet, determines your timeout, and then disconnects just before your server would drop the connection (so that the logs look more benign).
Re: (Score:2, Informative)
Possible work-around (Score:4, Interesting)
Lower Apache's timeout to below 166 seconds.
Re:Possible work-around (Score:5, Funny)
Re: (Score:2)
If you understand "EOSUCKS2BU", you need to get laid.
Re: (Score:2, Funny)
Sex makes people stupid?
Re: (Score:2)
Why would you ever need a timeout that high? When is a 30 second timeout not enough?
OpenBSD's pf has some mitigation features (Score:3, Informative)
OpenBSD's pf [openbsd.org] firewall has some options that can help mitigate the "single attacker, single source IP" version of this attack. Of course if the attackers decide to spread the attack out over multiple source IPs like a DDoS, this becomes much harder to deal with until Apache has a patch.
Filter rules that create state entries can specify various options to control the behavior of the resulting state entry. The following options are available:
number.
If the maximum is reached, packets that would normally create state
fail to match this rule until the number of existing states decreases
below the limit.
source IP address.
The total number of source IP addresses tracked globally can be
controlled via the
src-nodes runtime option [slashdot.org].
max-src-nodes will limit the number of source IP addresses that
can simultaneously create state.
This option can only be used with source-track rule.
max-src-states will limit the number of simultaneous state
entries that can be created per source IP address.
The scope of this limit (i.e., states created by this rule only or
states created by all rules that use source-track) is dependent
on the source-track option specified.
Re: (Score:3, Informative)
Something like:
iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 5 --rttl --name SSH -j DROP
limits to 5 new connections per 60 seconds
Re:OpenBSD's pf has some mitigation features (Score:4, Interesting)
Many browsers will open more connections than this, resulting in "broken" image links on pages. I've tried all kinds of connection limits to protect against simple DOS attacks, but always have problems with corpororations whose standard desktop configs include IE/Firefox set to open way more connections than would be polite.
It becomes politically challenging to cause those users to have a problem, especially since they don't see it with other sites. The perception is always that my server has a problem, and it doesn't matter how clearly I explain what's actually happening and how inappropriate it is to have 1000 PCs all set to open 30 connections to each web site they visit.
Re: (Score:2)
Our system may be safe (Score:3, Interesting)
Re: (Score:2, Informative)
mod_cband has been tested and doesn't have any effect.
Lingering connections handling (Score:4, Insightful)
It should be non-issues on better designed servers that keep an eye on connections anyway. Any single IP spawning lots of unfinished connections gets flagged fast and remembered for the future, so it will get limited access and bandwidth, marked as abuser etc. This is serving 101.
Not a flaw, easily configured around (Score:3, Informative)
http://httpd.apache.org/docs/2.2/mod/core.html#timeout
The issue is that the default configuration waits 5 minutes for the full request, which is painfully to long a period of time. Drop that from 300 to 5, and the "attack" goes away. If you are running the default Apache config in production, you shouldn't be.
Re:Not a flaw, easily configured around (Score:4, Informative)
http://httpd.apache.org/docs/2.2/mod/core.html#timeout
The issue is that the default configuration waits 5 minutes for the full request, which is painfully to long a period of time. Drop that from 300 to 5, and the "attack" goes away. If you are running the default Apache config in production, you shouldn't be.
seem like a potential fix, can anyone confirm?
Re: (Score:2)
Wouldn't this also affect file uploading? I'm not sure, but I think those are sent as part of the HTTP header.
Re: (Score:3, Informative)
I downloaded the Slowloris and was able to take down a default apache install, however with keepalive disabled and a timeout of 5, the attack became inneffective.
This may be a problem for sites with users that do long-running POSTs, but since we don't have any of those, all I can say is "It works here . . . "
For more info: http://httpd.apache.org/docs/trunk/misc/security_tips.html [apache.org]
Re: (Score:3, Insightful)
"...the server will open the connection and wait for the complete header to be received. However, the client (the DoS tool) will not send it and will instead keep sending bogus header lines which will keep the connection allocated."
In other words.. the connection is not allowed to "timeout" as there is (bogus) traffic on the connection.
Re: (Score:3, Insightful)
Re: (Score:2)
Did you read the doc? Seems like they thought of that situation. Here is the info
The total amount of time it takes to receive a GET request.
The amount of time between receipt of TCP packets on a POST or PUT request.
The amount of time between ACKs on transmissions of TCP packets in responses.
Re: (Score:2)
That's one of the most damning things you can say about a package.
HTTP hints at a solution (Score:5, Informative)
HTTP 1.1 [rfc-editor.org] specifies a status code for "Request Timeout" (408) and "Gateway Timeout" (504).
What is needed, therefore, is a timer running for receiving the complete header, and a second one for accepting the body. The timer for the body can be controlled by the type of request and the Content-Length header. (With, of course, a specific cap.)
Currently, Apache 2.2 [apache.org] has a single timeout value for all types of requests, but it is interpreted differently for the different types.
If your server only handles GETs, the obvious thing is to crank that number down. Unfortunately, for PUTs, the TimeOut value affects inter-packet time in the request, not overall request time.
Strangely, the timeout doesn't seem to run in 2.2.10 and 2.2.11 before data is received. Oh dear. That's an even simpler DoS.
Not quite as stealthy, though. At least as above.
Re:HTTP hints at a solution (Score:5, Funny)
BTW, is there a self-mod value for "I'm not sure I should have posted that"?
Re: (Score:2)
No, but there's a "+5, Thanks anyway!". And looks like you're getting it already.
Re: (Score:2)
That's informative, but I thought the point about GET vs. POST and PUT was confusing.
To be clear, the Timeout directive can be set low without affecting the ability of people to upload large files to the server. Timeout only applies to the time between packets, which should be a few hundred milliseconds apart under most circumstances, right?
From the httpd manual:
Re: (Score:2)
Maybe you're on an OS with a dataready or HTTP accept filter. Timeout applies to reading the entire first line of the request.
Why wouldn't the following setting (Score:2)
mitigate it somewhat:
In httpd.conf
#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 120
Unless of course this timeout is for after the header is received only... which I don't think it is... but as they say... assumption is the mother of all f*ckups.
Re: (Score:2)
I can just hear the words... (Score:2)
...of one of the 14 year olds who uses this, as she runs the script.
"Dodge this." ;)
Many othere services are probably vulnerable (Score:2)
Sendmail and other servers are probably vulnerable to this kind of thing. And it is not necessarily the server application itself may not be where the core of the server slowdown occurs. For example, if one were to spread this kind of attack across several different types of TCP-based protocols (SMTP/SMTPS, IMAP(S), HTTP(S), DNS(tcp version), etc then the operating system's TCP engine might start suffer from too many TCP control blocks. (And it isn't just the memory occupied - some silly implementation m
Re: (Score:2)
My mail server (Postfix) actually takes advantage of this. I have it configured to tarpit known spam sources (from RBL) and hold the connections open without sending a response.
Re: (Score:2)
Oh, sendmail isn't probably 'vulnerable', it definitely is. That's why sendmail has FEATURE(`conncontrol', ,`terminate'), to limit simultaneous sessions per client. Spammers have been abusing it for eons. I put vulnerable in quotes because that feature is configurable, but not a default. If your daemon doesn't have a way of dealing with these things (apache does with a module), there's always
iptables -A INPUT -p tcp --syn --dport 80 -m connlimit --connlimit-above 10 -j DROP
Link to the specific article (Score:5, Informative)
If you're going to post links to isc.sans.org, can you please post links to the specific article, and not just the main page?
Here is the link to the specific article: http://isc.sans.org/diary.html?storyid=6601 [sans.org]
Queuing and timeout (Score:2, Informative)
I believe this works because there is a timeout associated with the completion of a request. Sure, it
Re: (Score:2)
On Apache the request timeout directive is TimeOut.
It does not impact the response timeout. In my tests setting it to 2 seconds broke the tool but did not break normal POSTs and GETs. I saw no lag even with Slowloris running against the server.
Setting it to 5 seconds caused noticeable lag for a browser request.
This will probably work with busy servers but the lag times will be much longer. The directive only affects how long the server will wait _after the request stream starts_. Your browser will wait for
Re: (Score:3, Informative)
You may have missed the 'not' in the summary there.
Re: (Score:2)
Wait, you mean the summary on /. is finally correct!?
So... I did see a pig fly past the windows.
Re: (Score:2)
Tjat would have been your screen saver.
Re: (Score:2)
Re: (Score:2)
Wait, you mean the summary on /. is finally correct!?
No. Here [sans.org] is the link to the article. Not sure what to tell ya about the pig, though. Maybe check the dosage on your meds? ;-)
Re: (Score:2)
HTTP, not apache (Score:2)
The article states that this is like a syn flood, but on the HTTP level, rather than the TCP level. So essentially, it sounds like a flaw in previous understanding of HTTP, which will now be rectified by patching servers INCLUDING Apache.
It's a pity the reports of affected servers wasn't more comprehensive though; I'd like to know where Cherokee and nginx stand.
Re: (Score:3, Funny)
Re:Not *such* a big deal (Score:2)
Windows users: You probably will not be able to successfuly execute a Slowloris denial of service from Windows even if you use Cygwin. I have not had any luck getting Slowloris to successfuly deny service from within Windows, because Slowloris requires more than a few hundred sockets to work (sometimes a thousand or more), and Windows limits sockets to around 130, from what I've seen. I highly suggest you use a *NIX operating system to execute Slowloris from for the best results, and not from within a virtual machine, as that could have unexpected results based on the parent operating system.
Re: (Score:3, Informative)
Increase the number of connections in Windows XP post SP2. [speedguide.net]
Re: (Score:2)
Just tested on www.microsoft.com and I sat there in the header negotiating part for over 3 minutes before the pipe broke on remote end (I didn't do anything.) Slight change in that slowloris makes IIS* vulnerable too.
Heck one could even make it so that the header requests go out at 8 bit/s or less and just whallop any server with a lot of these requests... and have the requests FULLY valid.
Its simple for an admin to block the IP so its not that big of an issue unless it is employed as a DDoS.
Re: (Score:3, Interesting)
Keeping idle sessions open on the server for some period of time is not the flaw. The flaw is that some servers (in default configurations) only handles so many of these idle sessions open before reaching a limit and refusing to accept any new sessions, and that this limit is sufficiently low,
Re: (Score:2)
What are you talking about? There is settings in IIS regarding max connections as well and presumably would be effected in the same manner if these are set too low. This has nothing to do with fork (process) or non forked (lightweight process) method of serving.... or are you telling me that IIS is currently completely on Fiber level instead of their "Threads" ?
Either way at some point you still need to do some poll on the master socket to peek if there is some requests coming your way so you can relegate i
Re: (Score:2)
The problem is a combination of a single client being able to reach the maximum connection limit in the default configuration, and this is more like
Re: (Score:2)
And yet all those web servers that don't use forking are stable enough that they haven't been rewritten...