Scrapers, Bots and Probers Oh My!

The "Web" has become a smorgasboard for data scrapers, AI bots, vulnerability probers, site rippers and those who wish to do your site harm in some way or another. It doesn't matter if it is a simple niche site like The Guitar Gallery, they blindly grab as much as they can as fast as they can. They couldn't care less if their activity buries your site in unwanted traffic or completely skews and pollutes your site analytics. They simply want to make money freeloading off of you, the small site owner or worse commandeer your site for their on nefarious purposes.

It is estimated that nearly 50% of all web traffic is generated by these jerks. Sure, you can deny them in robots.txt but the reality is most do not retrieve it much less obey it. It's not just that "guy in the dark basement chugging Redbull" that does this. Facebook, Amazon, Google Cloud, Huawei Cloud, Tencent, Microsoft, Apple, Akamai and MANY other "big ones" do the same. Using a robots.txt is quite simply an exercise in futility.

Most shared hosting companies do not block or discourage this kind of activity. As a matter of fact the greedier hosting companies WELCOME it! They use it as leverage to push you into more expensive hosting plans such as Managed Hosting or Bare Metal. This is simply not an option for a small site owner, especially if it solely supported by the site owner. They are only concerned with the safety of THEIR hosting platform, NOT your site.

I have tried a number of hosting companies such as Webline, A2 Hosting, Hosting.com and IONOS. Their support was akin to what you'd get when you have a problem with your cheapie Hong Kong Phooey router. Understanding the words coming out of their mouths is another thing entirely. The last I used was the US based IONOS and it was their WordPress specific hosting plan. Twice they did something on the back end that broke something on the site. They have NO helpdesk ticket system or support email address, only a phone number and we're back to trying to translate gibberish into English. Their response both times was to "Contact WordPress" for support. Needless to say I told them AMF.

Some time back I moved a number of my other domains to Hawk Host and I could not be more impressed or say enough good about them. Not only is the hosting top notch but their support is intelligent, knowledgeable and responsive. Several times now they have bent over backwards to help me solve an issue and never once told me to "Go Fish." In addition they have never tried to upsell me using the bots as leverage. Their security is another thing. They actually know how to use Imunify360 and already block a number of the worst of the worst. When I moved to them I noticed a marked decrease in unwanted traffic.

Combating these jerks is not easy but can be done. They are getting smarter and smarter all the time. Simply blocking by user agent or IP by themselves is not enough. Some switch user agents from one hit to the next. Some are now using anonymizng VPNs (Virtual Private Networks) to circumvent countermeasures. Blocking single IPs is pointless as there are so many of them. I block entire CIDRs (groups if IP addresses ranging from just a few IPs to as many as a million IPs) and in some cases I've blocked ALL the CIDRs associated with a hosting/cloud company if I see too many different IP connections from their service. You will NOT stop them all and keeping up with them requires regular monitoring. However you can block the majority with some work and some simple .htaccess file rules.

I use a two pronged approach. First by user agent and the second by denying CIDRs. Here are the two supported by Apache 2.4. I use comments to tell me what hosting/cloud service or other reason I used for the block. Comments begin with a "#" symbol. However I should say this is and will be an ongoing process and requires regular monitoring to be even partially effective. Here are a few examples:

I also block VPNs known to be used by bad actors however there are some legitimate users who use these VPNs for privacy reasons. I cannot specify by single IP address and if it is an anonymizing VPN the IP can change quickly. As a result I block ALL VPNs known to be used by bad actors. There is ZERO need to use a VPN here, this site collects no data other than the Apache/Litespeed access logs which EVERY SITE DOES. No other data is collected so a VPN is pointless here. I do specify the reasons should a connection get blocked in my 403.shtml file (the page shown when an access is denied) to inform legitimate visitors as to why they were blocked allowing them to turn off the VPN for this site. Unfortunately I cannot allow all VPN connections for the sake of one or a few. For every legitimate VPN connection there are hundreds attempting malicious activity.

Another thing I block is outdated browser versions. Not only because script kiddie bots often report old browser versions but keeping a browser up to date with the latest security fixes is indispensable. Make no mistake, the web is a dangerous place and you are NOT paranoid if you think they are out to get you. SOME ARE. Using a fork of say Firefox or Chrome that does NOT keep its core up to the latest stable patch level is not wise, I don't care what the "benefits" are. I allow two versions back (usually no more than two months old) but it stops there.

In conclusion bots are getting smarter and smarter every day and with the increasing number of AI bots blocking them has become time consuming work. It takes daily monitoring of accesses and adjustments made daily to be effective. Owning and managing a site is work, there's no way around that. Doing so effectively can be a lot of work.