•

» Apache

5 Quick Tips for Securing Apache2

Posted on October 15, 2009 by Nathan Drier Leave a comment

Apache is a fantastic web server.  It’s easily installable on pretty much every modern operating system, it has gobs and gobs of community support, documentation and howto’s, and is very robust.  What I don’t like about Apache is its kitchen sink approach to functionality.  By default, lots of modules and extra configuration directives are enabled.  Needless to say, the majority of these aren’t needed for a simple web server.  Even with a more advanced web application, it’s best to start light.  Here’s my list of the first 5 things I do when building an Apache web server:

1.  Harden the OS. A rock solid web server wont stay up for long if the underlying operating system hasn’t been hardened.  Make sure you are running a recent, supported operating system.  Be sure that you are on a aggressive patching schedule, pick strong passwords for all accounts on the machine, and use encrypted management transports such as SSH.  Consider placing the web server in a DMZ or behind a firewall, and then only allow HTTP and HTTPS to it from the Internet.

2.  Get the Latest Version of Apache. There are two paths to take when installing Apache.  First, you can head to http://httpd.apache.org/ and grab the latest version to build from source. Second, you can install the version in your Linux distributions package manager.  Most Linux vendors backport security patches to their supported version of Apache, so even though you are getting security fixes – your version information wont necessarily match up with the latest and greatest from apache.org.

3.  Hide Version Information. Hiding version information can help you keep a low profile, but it really doesn’t do much for actually securing your server.  Nevertheless, it’s a simple step to take.  On a Debian system with Apache installed with apt-get, it’s as simple as editing /etc/apache2/conf.d/security and changing the following lines (While you are in there, disable the Trace method as well):

# ServerTokens
# This directive configures what you return as the Server HTTP response
# Header. The default is 'Full' which sends information about the OS-Type
# and compiled in modules.
# Set to one of:  Full | OS | Minimal | Minor | Major | Prod
# where Full conveys the most information, and Prod the least.
#
ServerTokens Prod


# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature Off

If you are running PHP, you can also hide PHP version information by editing /etc/php5/apache2/php.ini and edit the following:

# Decides whether PHP may expose the fact that it is installed on the
# server (e.g. by adding its signature to the Web server header).  It is
# no security threat in any way, but it makes it possible to determine
# whether you use PHP on your server or not.
expose_php = Off

4. Tune Your sites-enabled. Here is where the meat of your configuration lives.  From here, you can set everything from directory browsing to mod_rewrite statements.  This goes much deeper than the focus of this post (and may be good material for a future one).  The Apache Security Tips page and Virtual Host page are great places to start.

5.  Check Those Logs! Even the most secure Apache installation is crippled if you don’t know what is going on under all that fancy CSS.  Make sure Apache is logging errors and access with the following directives in your sites-enabled/vhost file:

CustomLog /var/log/apache2/access.log combined

ErrorLog /var/log/apache2/error.log

Once you have Apache logging, you need to do a couple things.  First, you need to store those logs in a secure, redundant location.  If your server ever crashes and takes all its logs with it, you will be clueless as to what happened.  Second, you need to check the logs on a regular basis.  I really like OSSEC and its integration with Apache.  OSSEC can watch your Apache logs and email you when certain events trigger it.

Well there you have it.  While this isn’t a end-all list of how to secure Apache, it should get you started.  I know I have spent alot of late nights wrestling with Apache, so hopefully this can save you a little time.

Extra Credit:

ModSecurity. ModSecurity is a monster of a web application firewall.  It comes stock with hundreds of rules to protect against common web attacks, and various communities have created their own custom rules as well.  It can be a little tricky to setup and maintain, but the protection it provides is worth every minute spent on it.

Hiawatha. Looking for a lighter, more security focused web server?  Try Hiawatha.  Its got protection for XSS, CSRF, and SQL Injection build in.  It also keeps very clean configuration files.

Have some input?  Leave your favorite Apache hardening tips in the comments.

DoS-ing over Dial-Up

Posted on June 21, 2009 by Nathan Drier 2 Comments

DoS, or Denial of Service attacks, are nothing new.  The main idea behind a DoS attack is to exhaust a devices resources (be it HTTP, some database backend, or any other form of  ‘service’) until it can’t respond to legitimate requests anymore. Typically, this is done from an application or link-saturation aspect, although it can be much more than that. Taking a sledgehammer to the A/C unit that serves a data center is pretty messy – but technically it is still a DoS attack.

The most popular form of DoS (i.e. the ones you read about in the paper) is link saturation. The plan is to send a massive amount of requests to a server, much more than it normally receives, in order to saturate either the server or its link. This requires LOTS of computers – usually sent from a botnet of some sort. When Amazon got hit with a DoS attack back in 2000, they had this to say about the amount of traffic:

“We had 800M bit/sec hit the site, which equals eight times our capacity. On average, our site runs at only 30% capacity, which gives you an idea of how unprecedented this traffic hit was.”

That is a downright amazing amount of traffic. To sustain that type of a DoS attack, you would need your own botnet. You would need thousands and thousands of computers on fast connections. The computers would have to be spread across the internet so they didn’t saturate the connection to the target site. This would have to be a huge, collaborative effort in the works for weeks. Recently, a new method of DoS-ing Apache HTTP servers has emerged that can take down a site with as little as 11.6Kb/s worth of traffic. That’s slow enough to be launched from a dial-up connection.

The idea has been around for a while, but just recently a tool has been developed to launch the attack. The tool works by exhausting Apache processes.  Apache comes configured to only allow a certain number of processes (default install is 256) and not answer any more requests when that limit is hit. What this tool does is send the first part of a request header. While Apache waits for the rest of the header, one of those 256 processes it taken up.  The tool, Slowloris, will send the partial header to the web server a few hundred times to hog the connection pool.  By default, Apache will wait up to 5 minutes for those connections to complete.  Once the tool has taken up all the available connections – the Apache server will not serve any new requests.

Lets dig in.  Here, I ran the tool against a test web server:

acm@stash:~$ perl slowloris.pl -dns webdev

Welcome to Slowloris – the low bandwidth, yet greedy and poisonous HTTP client

Defaulting to port 80.
Defaulting to a 5 second tcp connection timeout.
Defaulting to a 100 second re-try timeout.
Defaulting to 1000 connections.
Multithreading enabled.
Connecting to webdev:80 every 100 seconds with 1000 sockets:
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Building sockets.
Sending data.
Current stats:    Slowloris has now sent 658 packets successfully.
This thread now sleeping for 100 seconds…

Sending data.
Current stats:    Slowloris has now sent 1059 packets successfully.
This thread now sleeping for 100 seconds…

Sending data.
Current stats:    Slowloris has now sent 1283 packets successfully.
This thread now sleeping for 100 seconds…

As you can see, the tool makes 1000 requests every 100 seconds, more than enough to saturate our default Apache settings of 256 connections every 5 minutes.  While this tool is running, the Apache server is completely unresponsive to any new requests.  If someone is already at the site when this attack is launched, their connection will stay active until they request a new process.  At this point, its a race condition between a legitimate browsing session and the Slowloris tool, one that Slowloris will always eventually win.

Another interesting point is that this type of attack doesn’t create any errors in the logs until the attack is over.  Since the connections never complete – Apache never writes any of it to its logs.  Once the attack is over, multiple 404 errors will pop up – but its a little too late at that point.  First, I saw this in Apache’s error logs:

[error] server reached MaxClients setting, consider raising the MaxClients setting

This shows us that Slowloris was able to hog all the connections to the Apache server.  Then, we see hundreds of these entries in access.log:

[error] [client 10.0.0.45] request failed: error reading the headers
[error] [client 10.0.0.45] request failed: error reading the headers
[error] [client 10.0.0.45] request failed: error reading the headers
[error] [client 10.0.0.45] request failed: error reading the headers
[error] [client 10.0.0.45] request failed: error reading the headers
[error] [client 10.0.0.45] request failed: error reading the headers
[error] [client 10.0.0.45] request failed: error reading the headers

This happens because Apache only got the first part of the request headers, and never saw the complete header.  The following is all the 404 errors the tool generates:

“GET / HTTP/1.1″ 400 351 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSOffice 12)”
“GET / HTTP/1.1″ 400 351 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSOffice 12)”
“GET / HTTP/1.1″ 400 351 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSOffice 12)”
“GET / HTTP/1.1″ 400 351 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.503l3; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MSOffice 12)”

This all happens well into the attack.  It looks like you could keep a server down for quite a while before connections get reset and Apache writes anything to the logs.

I’ve seen some discussion about changing the connection timeouts on Apache from 300 seconds (our default 5 minutes) to something a little quicker – like 10 seconds.  This way, Slowloris can only hog a connection for 10 seconds until it resets.  Even so, limiting the timeouts to 10 seconds does little to stop the attack.  Anyone with an average speed cable connection could still DoS a vulnerable Apache site.  I’ve seen reports that with Apache timeouts set as low as 5 seconds, it still only takes less than 50kb/s of traffic to bring a site down.

If you have a good baseline on your web servers and know the limits of expected traffic – there is a quick and dirty fix for this.  This fix assumes that the attack comes from the same source-ip.  The following iptables rule limits the amount of connections from the same source to 20 connections.

iptables -A INPUT -p tcp –dport 80 -m connlimit –connlimit-above 20 -j REJECT –reject-with tcp-reset

With this in place, Slowloris can only take up 20 connections per source ip, leaving the other connections open for legitimate users.  Now we see Slowloris only sends 35 packets to the web server vs the ~1000 it could do before:

Building sockets.
Sending data.
Current stats:    Slowloris has now sent 35 packets successfully.
This thread now sleeping for 100 seconds…

While this is in no way a long-term fix, it takes the edge of a little.  I’ve also heard of people having success with mod_limitipconn, mod_evasive and accf_http, so they might be worth investigating as well.  Good Luck!