Bad Proxies Causing Apache to Reach MaxClients

Recently, I was called to assist with a server that was constantly getting bombarded with HTTP connections and causing Apache to hit MaxClients. It took a couple of minutes to track the IPs with the most connections using this little command:


netstat -tpnC | grep httpd | awk '{print $5}' | cut -f1 -d: | sort | uniq -c
1 65.55.25.149
4 67.141.163.123
1 67.195.111.40
1 71.247.16.159
28 137.244.215.55

This showed a large number of connections from an IP 137.244.215.55.
Since we have ExtendedStatus enabled in httpd.conf, it's a simple matter to find what site is getting hit:


# lynx -dump -width=160 http://localhost/server-status | grep -e '...[1-9].*' | grep -v OPTIONS

Sure enough, there's the IP and the site getting hit, along with the request URI. I like to use this to see when a comment spammer is POSTing to WordPress blog or otherwise trying something malicious. The site getting hit was a Fantasty Football site.

However, I was a little concerned because the IP that had a large number of connections was a .mil IP address.


# host 137.244.215.55
55.215.244.137.in-addr.arpa domain name pointer uhhz-wpa-001.robins.af.mil.

So I simply used ConfigServer Firewall (a nice front-end to iptables) to block the address for an hour.


csf -td 137.244.215.55 3600


9:45:00 AM Other Tech: block the .mil!
9:45:45 AM Me: blocked 'em.
9:45:49 AM Me: just for an hour though.
9:46:39 AM Me: in case it's the airforce cybercommand thingy that's investigating a terror suspect and enemy combatant.
9:47:05 AM Me: because a fantasy sports site is where all the terrorists hang out....

All was well and good, but shortly after blocking it another .mil address hitting the same site with a large number of connections:


netstat -tpnC | grep httpd | awk '{print $5}' | cut -f1 -d: | sort | uniq -c
1 124.115.4.197
1 186.18.102.180
1 38.99.98.113
1 65.55.25.144
1 68.216.159.34
1 68.240.147.150
32 131.51.128.21

After blocking that one, a wellsfargo.com address showed up, then a few other random corporate addresses. This beginning to concern me, since this resembles some sort of DDOS behavior from a bunch of infected PCs (and the idea of .mil and Wells Fargo computers being infected didn't sit well with me). But it wasn't a very effective attack, since only one or two IPs would hit at the same time. And why would someone attack a Fantasy Football site with military and banking computers? Sure there are better better targets than that!

Next step was to watch the logs for a bit (tail -f /path/to/domain/access_log). I noticed some odd behavior. Usually, a browser hits a site, requests a page, and then requests all the supporting files (CSS, javascript, images, media files, etc), usually listing the original referring URL along the way. This was a WordPress blog, so most traffic was fairly normal along these lines. But grepping for the specific IP addresses in the log showed a more unusual pattern: a single request from a generic user-agent string (“Mozilla 4.0 (Compatible)”), followed by a large number of requests for all the links on the page. Something like this:


131.50.151.28 - - [16/Oct/2010:11:39:28 -0400] "GET /baseball/wp-content/themes/SomeTheme/style.css HTTP/1.1" 200 404 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201009 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201007 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200910 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200912 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201006 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200908 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200911 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/xmlrpc.php HTTP/1.1" 200 404 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201010 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200909 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=201008 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200905 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200811 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/xmlrpc.php?rsd HTTP/1.1" 200 408 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200809 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=5 HTTP/1.1" 200 408 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200808 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200810 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200904 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200906 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=1961 HTTP/1.1" 200 411 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200812 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?p=1989 HTTP/1.1" 200 411 "-" "Mozilla/4.0 (compatible;)"
131.50.151.28 - - [16/Oct/2010:11:15:43 -0400] "GET /football/?m=200907 HTTP/1.1" 200 413 "-" "Mozilla/4.0 (compatible;)"

I watched this for a while, scratching my head. The “Mozilla/4.0 (compatible;)” was suspicious. It was immediately obvious that it was some kind of bot or spider. Bad bots like to disguise themselves or try to pass themselves off as real browsers to avoid detection or redirection based on their behavior. So I was beginning to think this was a really bad indexing/search spider. Except that it's hitting this one single site, and there there were also referrer links in some of the lines indicating traffic from other sites. And bots are usually operated from a single IP address – they don't spring up from other IPs when the first one is blocked.

Troubleshooting is often a team effort, and it certainly helps to discuss a problem and brainstorm ideas.


11:08:53 AM Me: i'm beginning to think that our military is not infected with bots, but are goofing off with fantasy football, which scares me even more.
11:10:31 AM Other Tech: The useragent is weird though
11:11:23 AM Me: yeah. maybe they're behind a proxy server that grabs all the linked pages for immediate caching.

Sure enough, a quick Google for “Mozilla/4.0 (compatible;)” yielded some hits of exactly that. Behind the corporate doors of Wells Fargo and various Air Force bases are a bunch of people reading up on Fantasy Football, and their collective proxy servers (probably Blue Coat) are slamming the server with a ton of requests to pre-fetch all other linked URLs from the first page, so that each visitor is hitting the server with dozens of connections.

Since this is a shared server, this is affecting not only the customer site in question, but all other sites hosted on this server. It's obvious we can't block by IP address, and there are too many variations to block entire ranges (which is a bad idea to begin with).

My solution was to redirect on the User-Agent string until the issue died down. I created a single HTML page on the server's main DocumentRoot (outside the customer's virtual host) and added the following lines to the customer's .htaccess file:


RewriteCond %{HTTP_USER_AGENT} ^Mozilla\/4\.0\ \(compatible;\)$
RewriteRule .* http://host.example.com/denied/ [R]

The actual index.html page in the Redirected URL sums it up like this:


Your "web accelerator" proxy is causing problems with our servers and customer sites.
Sorry, but you will not be able to access content here.
Please contact your IT Support department for assistance.


Addendum:
I found the script on this page very helpful when testing the User-Agent string in my .htaccess rule. While I am quick to telnet to a webserver to pass an HTTP request a simulate a browser visit, I don't know all the details of the HTTP protocol (including the format of User-Agent string). Scripting this to quickly connect to localhost, pass the request and the User-Agent and see if I received a 200 or 302 Redirect was extremely helpful.