Attack On a Significant Flaw In Apache Released 203
Zerimar points out a significant flaw in Apache that can lead to a fairly trivial DoS attack is in the wild. Apache 1.x, 2.x, dhttpd, GoAhead WebServer, and Squid are confirmed vulnerable, while IIS6.0, IIS7.0, and lighttpd are confirmed not vulnerable. As of this writing, Apache Foundation does not have a patch available. From Rsnake's introduction to the attack tool: "In considering the ramifications of a slow denial of service attack against particular services, rather than flooding networks, a concept emerged that would allow a single machine to take down another machine's web server with minimal bandwidth and side effects on unrelated services and ports. The ideal situation for many denial of service attacks is where all other services remain intact but the webserver itself is completely inaccessible. Slowloris was born from this concept, and is therefore relatively very stealthy compared to most flooding tools."
Re:Well its not just Apache (Score:3, Informative)
You may have missed the 'not' in the summary there.
iptables helps (Score:5, Informative)
You can have perlbal or any reverse proxy on the same machine but listening on a different port and then use iptables to redirect like this
# iptables -t nat -A -PREROUTING -d ! 127.0.0.1 -p tcp -m tcp --dport 8080 -j REDIRECT --to-ports 80
and then you don't need to change your apache configuration - and having apache listen on a different port to what users see can break some scripted sites if they read the port number from the apache config.
OpenBSD's pf has some mitigation features (Score:3, Informative)
OpenBSD's pf [openbsd.org] firewall has some options that can help mitigate the "single attacker, single source IP" version of this attack. Of course if the attackers decide to spread the attack out over multiple source IPs like a DDoS, this becomes much harder to deal with until Apache has a patch.
Filter rules that create state entries can specify various options to control the behavior of the resulting state entry. The following options are available:
number.
If the maximum is reached, packets that would normally create state
fail to match this rule until the number of existing states decreases
below the limit.
source IP address.
The total number of source IP addresses tracked globally can be
controlled via the
src-nodes runtime option [slashdot.org].
max-src-nodes will limit the number of source IP addresses that
can simultaneously create state.
This option can only be used with source-track rule.
max-src-states will limit the number of simultaneous state
entries that can be created per source IP address.
The scope of this limit (i.e., states created by this rule only or
states created by all rules that use source-track) is dependent
on the source-track option specified.
Re:OpenBSD's pf has some mitigation features (Score:3, Informative)
Something like:
iptables -I INPUT -p tcp --dport 80 -m state --state NEW -m recent --update --seconds 60 --hitcount 5 --rttl --name SSH -j DROP
limits to 5 new connections per 60 seconds
Re:Other Web Servers....Proxies......? (Score:2, Informative)
Not a flaw, easily configured around (Score:3, Informative)
http://httpd.apache.org/docs/2.2/mod/core.html#timeout
The issue is that the default configuration waits 5 minutes for the full request, which is painfully to long a period of time. Drop that from 300 to 5, and the "attack" goes away. If you are running the default Apache config in production, you shouldn't be.
HTTP hints at a solution (Score:5, Informative)
HTTP 1.1 [rfc-editor.org] specifies a status code for "Request Timeout" (408) and "Gateway Timeout" (504).
What is needed, therefore, is a timer running for receiving the complete header, and a second one for accepting the body. The timer for the body can be controlled by the type of request and the Content-Length header. (With, of course, a specific cap.)
Currently, Apache 2.2 [apache.org] has a single timeout value for all types of requests, but it is interpreted differently for the different types.
If your server only handles GETs, the obvious thing is to crank that number down. Unfortunately, for PUTs, the TimeOut value affects inter-packet time in the request, not overall request time.
Strangely, the timeout doesn't seem to run in 2.2.10 and 2.2.11 before data is received. Oh dear. That's an even simpler DoS.
Not quite as stealthy, though. At least as above.
Re:Why not IIS? (Score:5, Informative)
unless you are using Session()'s in asp in IIS then one thread in IIS handles multiple connections.
what this is doing is opening a connection (getting a thread to work it) and holding it open (keeping the thread busy) and just keep asking for new ones.
it is very common (always i think) for Apache and allot of web servers to have a max thread's so that the site under heavy traffic doesn't open more connections than it can handle.
where IIS also has a worker thread limit - there is no limit *(you can set one - but not on by default) on how many concurrent connections can be managed by a thread (and new incoming connections are passed to the thread with the lowest current work load - not always the one with less connections)..
if you do what they are doing here i can see IIS behavior would be to slowly pile all these slow - no work connections into one thread and the others would happily go about doing actual work..
where apache would slowly lose access to workable threads as this keeps them busy.
this isn't an exploit on the http or tcp protocol - it is an exploit based on the behavior of the web server based on it's best practices for managing it.
Re:WTH? This is an absolutely trivial attack (Score:3, Informative)
No, it's not. It's holding an HTTP session open. That is not the same thing as a TCP socket.
Re:Why not IIS? (Score:2, Informative)
More likely IIS survives because it uses a worker pool threading model (no thread/process is dedicated to a connection, so a connection only takes up memory for the state, not for the thread).
Apache had, and probably still has, a process/thread-per-connection model.
So with all due respect, it looks like a proper design decision is what is protecting IIS here:
http://www.kegel.com/c10k.html
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/a63ee1c2-04d6-44dc-b4d6-678eb3117bf9.mspx?mfr=true
Re:So slashdot... (Score:5, Informative)
We have a hardware load-balancer and a software reverse proxy (varnish) in front of our apache.
I kinda doubt this would work on us.
Note, I am not inviting anyone to try. It might work great for all I know :(
Re:Not a flaw, easily configured around (Score:4, Informative)
http://httpd.apache.org/docs/2.2/mod/core.html#timeout
The issue is that the default configuration waits 5 minutes for the full request, which is painfully to long a period of time. Drop that from 300 to 5, and the "attack" goes away. If you are running the default Apache config in production, you shouldn't be.
seem like a potential fix, can anyone confirm?
Re:WTH? This is an absolutely trivial attack (Score:5, Informative)
Let me make this clearer for those that aren't very technical: It's holding an HTTP session open and Apache has a limited number of simultaneous HTTP sessions.
All someone has to do is send about 100 requests to your website and leave them open without sending any further information. Nobody else will be able to connect to your web server for a long time. The weekend is coming, so I'm expecting lots of downtime for government sites in the next couple of days...
Attack of a 1000 snails. (Score:2, Informative)
This type is already known as "attack by a 1000 snails" type attack. It is harder to defned against than you would think. A user can be slow, but coders are hesitant to drop users that ar too slow or too fast.
A user kan just keep the TCP/IP alive by sending one byte every x seconds. If this is patched at http header level, you will see you can do the same kind of attack on the application, that can have limited php or perl sessions.
Re:WTH? This is an absolutely trivial attack (Score:4, Informative)
A simple connlimit declaration in IPTables shuts this down fairly easily...
Re:Why not IIS? (Score:4, Informative)
If a client sends a SYN to 10.1.1.1:80 and gets an SYN-ACK from 10.5.5.5:80 then the client will not associate the two as being related, and will keep waiting for a response from 10.1.1.1:80 until timing out.
You would need to have some sort of DNS arrangement that encouraged clients to make their requests to your various IPs. You can't just respond from a different IP than the client contacted.
Re:WTH? This is an absolutely trivial attack (Score:4, Informative)
Not really, it just means you need more than one attacker.
Re:Why not IIS? (Score:1, Informative)
That's great, but others like myself have been burned too many times with hacked IIS boxes to ever return. Thanks but fool me once... or something like that..
Link to the specific article (Score:5, Informative)
If you're going to post links to isc.sans.org, can you please post links to the specific article, and not just the main page?
Here is the link to the specific article: http://isc.sans.org/diary.html?storyid=6601 [sans.org]
Re:Not a flaw, easily configured around (Score:3, Informative)
I downloaded the Slowloris and was able to take down a default apache install, however with keepalive disabled and a timeout of 5, the attack became inneffective.
This may be a problem for sites with users that do long-running POSTs, but since we don't have any of those, all I can say is "It works here . . . "
For more info: http://httpd.apache.org/docs/trunk/misc/security_tips.html [apache.org]
Queuing and timeout (Score:2, Informative)
I believe this works because there is a timeout associated with the completion of a request. Sure, it might be difficult to distinguish a slow DoS from a slow client, but it wouldn't be impossible to set a reasonable time limit on non-POST requests. That would be a relatively easy way to fix the issue in Apache.
As far as POST goes... well, that's a different (and valid) way to perform a slow DoS attack:
Server: What would you like? Ham bacon spam, or spam eggs bacon spam with spam?
Client: I'm actually here to deliver some SPAM!
Server: How much SPAM?
Client: SPAM, SPAM, SPAM.... (3 hours later)
Slowloris can do this too. By default, IIS only reads the up to first 48KB of post data (I see much smaller numbers in practice), at which point the request is sent to an extension/app. Before this, the request doesn't leave the kernel-mode driver (http.sys). The apps can easily ignore the data or read more (on a timeout). I wouldn't be surprised if Lighttpd did the same thing (sans kernel driver).
Re:Not *such* a big deal (Score:3, Informative)
Increase the number of connections in Windows XP post SP2. [speedguide.net]
Re:Our system may be safe (Score:2, Informative)
mod_cband has been tested and doesn't have any effect.
Re:Not a flaw, easily configured around (Score:1, Informative)
That was tested by the Slowloris guys and it looks like it worked -sorta. See http://ha.ckers.org/blog/20090617/slowloris-http-dos/#comment-105386 for more information.
Re:Well its not just Apache (Score:2, Informative)
You can use fcgid to run PHP in a different process, and then safely run apache multi-threaded. Just FYI for those using PHP.
It's also a good deal faster, and more stable to boot.