I discovered an issue with mod_evasive and Googlebot recently. I noticed that after installing mod_evasive the number of impressions in Google Webmaster Tools had been reduced. Additionally, some of my page results in Google where displaying “A description for this result is not available because of this site’s robots.txt – learn more.”

In /var/log/mod_evasive are the logs of all the IPs that have been banned at some point. I found that some of these IPs were from the Googlebot, and mod_evasive blocking Googlebot was likely the cause of the reduced impressions. I’ve now increased the default setting of mod_evasive which are

DOSHashTableSize 3097
DOSPageCount 2
DOSSiteCount 50
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10

and have increased them to the following in order to relax the DOS trigger

DOSHashTableSize 3097
DOSPageCount 10
DOSSiteCount 100
DOSPageInterval 1
DOSSiteInterval 1
DOSBlockingPeriod 10

So far Googlebot has not been blocked again.

In mod_evasive it is possible to whitelist IPs and whitelisting Google IPs might be an option. The problem is that it seems that Google IPs are constantly changing. The IP ranges that some other blog posts had recommended whitelisting were not even close to the Google IPs I was getting. So i’ve opted for more relaxed DOS constraints.