Mod_Evasive Blocks Googlebot

I discovered an issue with mod_evasive and Googlebot recently. I noticed that after installing mod_evasive the number of impressions in Google Webmaster Tools had been reduced. Additionally, some of my page results in Google where displaying “A description for this result is not available because of this site’s robots.txt – learn more.”

In /var/log/mod_evasive are the logs of all the IPs that have been banned at some point. I found that some of these IPs were from the Googlebot, and mod_evasive blocking Googlebot was likely the cause of the reduced impressions. I’ve now increased the default setting of mod_evasive which are

DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10

and have increased them to the following in order to relax the DOS trigger

DOSHashTableSize 3097
DOSPageCount 10
DOSSiteCount 100
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10

So far Googlebot has not been blocked again.

In mod_evasive it is possible to whitelist IPs and whitelisting Google IPs might be an option. The problem is that it seems that Google IPs are constantly changing. The IP ranges that some other blog posts had recommended whitelisting were not even close to the Google IPs I was getting. So i’ve opted for more relaxed DOS constraints.

Stopping WP-LOGIN Attacks, Hacks and Floods

One of my blogs gets attacked regularly by hackers trying to break into the WordPress admin area by spamming various passwords on the wp-login page (the page you use to login to your WordPress admin area). This increases CPU usage on my server from an normal average of 5% to about 40%, effectively slowing the site down. There are various ways to stop these flood attacks and in this post I show some methods that I used. But first, although this post is about stopping server floods for performance reasons I should mention that if you are using the ‘admin’ user name you should stop right now and change that username to something else. Almost all the bots operating this kind of attack try to brute force their way in using the ‘admin’ username.

HTTP Auth Login

The best method I have found to stock attacks is a to add a second login screen to your WordPress with an HTTP Auth login. An HTTP Auth login will ask you to input a username and password before you can even access the wp-login page. Most wp-login attack bots won’t be programmed to try and crack HTTP Auth logins and most spammers will simply toss your site away instead of trying to waste time brute forcing two login screens. Additionally, if a spammer were to try and brute force the HTTP Auth login anyway, the server would not see a large increase in CPU usage. This is because HTTP Auth is extremely efficient and uses very little resources since it does not need to access the WordPress database. As long as you use a unique username and strong password brute force methods on HTTP Auth should be secure. And if they do get in they still have to get past the standard WP-Login screen. If you’re extra concerned about security you could set up another two factor authentication of the wp-login screen using Google Authenticator or other methods.

To create an HTTP auth login you will need to have access to your .htaccess file and the home folder on your server. First create a .htpasswd file by going to http://www.htaccesstools.com/htpasswd-generator/ and entering in your desired username and password. Upload the generated file to your home directory or wherever you want to put it. Make sure it is not accessible from the web. Or, instead you could install apache2-utils (sudo apt-get install apache2-utils) and then use htpasswd -c /home/user/.htpasswd HTTP_AUTH_USERNAME where you should replace HTTP_AUTH_USERNAME with your own choice of login username. You’ll then be asked to enter your pick of password twice, then the .htpasswd file will be generated.

Next in your .htaccess file copy and paste the following, being sure to change AuthUserFile to the actual location of your .htpasswd file.

# Protect wp-login
<Files wp-login.php>
AuthName "Authorized Only"
AuthType Basic
AuthUserFile /home/USER/.htpasswd
require valid-user
</Files>

Note: You must use the FULL path to the .htpasswd file. I found that even using ~/.htpasswd would not work even though they both pointed to the same location.

Also note that if you have multiple WordPress users or if you allow registrations on your site then this method may not be for you if you don’t want to annoy your users with two login screens. A simple fix for this might be to change the AuthName string to some user instructions like “ANTI-BOT MEASURES: Use the username ‘access’ and password ‘accesspwd’ to access the login screen”, making sure to set the .htpasswd username and password accordingly. It is unlikely that a human spammer would even bother to look at your site and discover the HTTP Auth login credentials.

Look for No-Referrer Requests

When a spam bot tried to access your wp-login page it will usually try and access the file directly, without first going to your site. By using some simple .htaccess code we can detect if the bot was reffered by your URL or not. And if not, block access to the bot. Simply add this code to your .htaccess file where you should replace MY_DOMAIN.com with your domain. This code will also protect the comments submissions form from some spammers.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{REQUEST_URI} .(wp-comments-post|wp-login)\.php*
RewriteCond %{HTTP_REFERER} !.*MY_DOMAIN.com.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule (.*) http://%{REMOTE_ADDR}/$ [R=301,L]
</IfModule>

Unfortunately, modern advanced bots will get around this easily, but there are still some using older methods looking for unsecured sites.

Block Logins from all IP’s other than your own

If you have a static IP address and only access your WordPress site from fixed locations the best method at stopping login attacks is to block access to the wp-login page from ALL IP’s other than your own. This can be easily implemented by adding the following code to your .htaccess file.

<FilesMatch wp-login.php>
Order deny, allow
allow from XXX.XXX.XXX.XXX (replace XXX with YOUR OWN IP address)
allow from XXX.XXX.XXX.XXX (can use multiple IP addresses)
deny from all
</FilesMatch>

I didn’t end up using this method as I don’t have a static IP and I access like to access my blog while travelling.

WP-Security Plugin

Another method i’ve tried adding to my blog is the WP Security-Protection plugin. This plugin works by detecting the presence of Javascript. Most bots will not have Javascript enabled and will be detected as a bot. Then the plugin sends a cookie that tricks the bot into believing that it has gained entry. In most cases this will stop the bot from continuing. However, I decided not to use this plugin as with HTTP Auth the bot should never even get to the login screen.

IP Blocking with Fail2Ban

This method uses Fail2Ban (an IP blocking server tool) to check access logs for wp-login.php floods. The best tutorial I found on this method was here. However, I ended up not using this method because many of the spammers IPs where spoofed and were pointing to Google servers. Not wanting to block Google IPs and probably get a SEO problem I ditched this method in favor of the HTTP Auth and no-referrer request methods. Another danger is that many of these bots will be coming from hacked PCs around the world. Blocking one of these IPs could be potentially blocking a customer or even an entire company as some companies will use one external IP address for all their computers.