What is Bot Traffic? | How to Detect Bot Traffic?
The latest statistics show that in 2019, overall bot traffic grew by 24% compared to 2018. This article will explain what bot traffic means, how to detect and prevent it with useful tips.
What is Bot Traffic?
Bot traffic is non-human traffic generated by robots to websites and apps. Mostly this kind of traffic gives negative associations; however, it’s not always as bad as it looks. There are two kinds of bot traffic–the good and the bad.
The good bot traffic:
- bots that scan your content (web crawlers, also known as Googlebots);
- chatbots, for example Jivochat (these bots include already prepared answers when a visitor asks something on your site);
- search engine bots (allow to index the content of your website).
The bad bot traffic:
- bots that are providing spammy comments and including some malicious links;
- bots that affect your site and make it inaccessible for users;
- unauthorized web crawlers that can affect your website’s stats and generate click fraud;
- click bots that make ‘fake’ automatic clicks on your ads.
Robot Network (aka Botnet), is usually responsible for bad bot traffic. It is a set of computers infected by malware under the control of a single attacking party, known as the «bot-herder.»
Usually, bots are created to replace human activity that repeats repetitive tasks, such as copying and posting comments, ad clicking, and other activities associated with malvertising attacks.
Related Article: Digital Ad Fraud | Everything Publishers Should Keep in Mind
How to Detect Bot Traffic?
To detect it, you should check out your Google Analytics account or any other web analytics tool that analyzes website traffic.
If your site has experienced bot traffic, your reports might show:
- an unexpected increase in traffic (usually during odd hours, for example, at 11 PM);
- more pageviews than usual;
- increase in bounce rate;
- low average session duration time;
- traffic from unusual geo-locations;
- a lot of low-quality conversions (spam bots can cause many filled-up sign-up forms with fake email addresses, names, numbers, etc.).
The Easiest Way How to Filter Bot Traffic on Google Analytics
*In the example below, we use Setupad data to show how we identified the unexpected bot traffic on our site.
1. Login to your Google Analytics Account.
2. Click on section ‘Acquisition’, then ‘All Traffic’ and choose section ‘Channels’.
3. By looking at the image above, we can see that there has been an unexpected increase in traffic for one specific period.
4. Then, filter out that particular period of time. Scroll down and check out from which channels most of your traffic came from–Organic Search, Direct, Referral, etc.
The image below shows that the most traffic came from Referral and Organic Search channels with super high bounce rates of almost 100% and average session time only several seconds compared to other channels.
5. Click on each channel to see the source URL from where the traffic came from. In the image below, we can clearly see that the site has been affected by bots and the same source appeared on the channel Organic Search.
This is one way to understand if your site has been affected by the bots and from what source it really came from. In our case, someone has bought the bot traffic for Setupad site.
Since the bot traffic amount wasn’t that big and happened only once, the only drawback was that it messed up our reports a bit. However, to avoid bad bot traffic in the future, we have explained 3 useful tips in the next paragraph.
But, before we continue with the tips, there is a way to remove bot traffic stats from your Google Analytics reports.
How to Remove Bot Traffic from Google Anlytics?
❗Remember, excluding bot traffic from your reports doesn’t mean you will be safe from the effects of invalid traffic.
First, you have to log in to your Google Analytics account, then go to the ‘Admin’ section and ‘View Settings’.
Then, find the section ‘Bot filtering’ and tick it, press ‘Save’. This will help you to keep bot traffic out of your Google Analytics reports.
3 Tips to Avoid Bot Traffic
- Use reCAPTCHA–It works like a firewall to protect your site from bot attacks. It’s a more advanced technology compared to classical CAPTCHA. Regarding this, you can find help from bot filtering services like Cloudflare. See an example of reCAPTCHA and classical CAPTCHA below..
- robots.txt file–This file helps to block bot traffic while it’s crawling through your site. According to Google: «This is used mainly to avoid overloading your site with requests». Here you can learn more about the robot.txt file, what kind of bot activities it blocks, and its limitations.
- Use Bot Mitigation solution–It’s usually provided as a tool by companies like Cloudflare, Imperva. They use machine learning across many Internet properties to understand if the traffic coming to the site is from a bot. These solutions can differentiate the good bots, like search engine bots, from bad ones, like spam bots.
It’s worth keeping in mind that now bot developers are discovering new ways to circumvent standard security solutions using machine learning and artificial intelligence to imitate human-like actions on websites. However, companies that try to block and stop the bot traffic give the hope of beating the bad guys and creating a healthy and secure web ecosystem.
Setupad Expert Opinion:
Povilas Goberis, COO at Setupad: «If a publisher’s site that displays programmatic ads has experienced fraudulent bot traffic attack, I would suggest using Cloudflare solution with traffic verification turned on to avoid the bots entering a site. There is also another method, but it’s more hash- blocking all the IPs from foreign countries.»
Final Thoughts
Don’t panic if your site has experienced unexpected and anomalous bot traffic; however, make sure you have done your job of investigating and understanding what kind of bot traffic it is and from where it came from.
After that, if your site has experienced fraudulent activity by bots, make sure to avoid it from happening again in the future. Check out the tips we mentioned above; they will give you some insights on how to secure your site.
