A user agent is a string of text that a web browser or other client software sends to a web server along with each request to identify itself and provide information about its capabilities.
The user agent string can contain information such as the browser name and version, the operating system, and other details.
Site owners can block user agents easily using CloudFlare Firewall Rules known to be associated with malicious activity, such as bots and scrapers
In this post, you will learn all you need to know about blocking user agents for security purposes.
Table of Contents
What User Agents to Block?
Some examples of user agents that may be associated with malicious activity and should be blocked for security purposes include:
| python | Go-http-client |
| curl | github |
| Apache | Scrapy |
| ruby | wp_is_mobile |
I check my Firewall logs regularly and I don’t see that many malicious user agents since most malicious automated traffic identify itself as legitimate traffic:
These are some examples of user agents used by bad bots:
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Firefox/102.0
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0 Safari/537.36
Where to get a List of User Agents to Block?
I know I recommended 8 user agents to block but I don’t really believe in copying someone’s huge list of malicious user agents and building a firewall with it.
Automated traffic with a specific user agent that might be bothering someone’s website and never come close to your site.
So it is a better idea to check your Firewall logs and spot the ones that are trying to attack your site.
Blocking SEO Tools and the Other Guys
Blocking certain types of scrapers or SEO tools can be done by blocking their user agent.
You can block non-malicious bots like Ahrefs, Moz or Search Engines like Yandex
The good thing about blocking these types of services is that these types of bots identify themselves as bots.
Blocking User Agents is not the Ultimate Security Measure
Blocking certain user agents can improve the security of a WordPress site by preventing certain types of bots or automated scripts from accessing the site.
It is also important to note that blocking user agents is not a complete solution to securing a WordPress site.
User agents can be easily faked by malicious actors in order to bypass security measures that are in place to block certain types of user agents. This can be done by modifying the user agent string that is sent with HTTP requests to make it appear as if the request is coming from a different type of device or browser.
For example, a hacker could change their user agent to that of a search engine crawler in order to gain access to parts of a website that are otherwise blocked to regular users.
is Google using OVH servers now?
When it comes to user agents, you can be any big organization you want.
It is important to note that, while blocking specific user agents can be an effective security measure, it is not foolproof, as malicious actors can easily change or spoof their user agent to bypass these restrictions. Therefore, it’s recommended to use a combination of security measures to protect your website, such as using a web application firewall and monitoring traffic for unusual behavior.
How to Block User Agents with Cloudflare Firewall Rules
Blocking user agents with Cloudflare is a simple process that can help protect your website from unwanted traffic or bots. Here’s how to do it:
- Log into your Cloudflare account and select the website that you want to block user agents for.
- Click on the “Security” tab, then click on the “WAF” button.
- Click on the “create firewall rule” button
- Name the rule
- Choose “user agent” from the field dropdown menu
- Choose “contains” from the field dropdown menu
- Add a keyword in the “value” field
- use “or” to target other user agents
- Choose “block“
- Click the “Deploy Firewall Rule” button
As you can see, the process is really simple.
You can update remove or add more keyword as you consider necessary
