How to Block Unwanted User-Agent & Referrers in Apache, Nginx and WordPress?

As a website owner, blocking unwanted user-agents and referrers is an important step to protect your site. In my experience as a developer and cybersecurity analyst, I want to provide a detailed guide on effective methods to block problematic traffic in Apache, Nginx, and WordPress.

Why Unwanted Traffic is Harmful

Bad bots and spam referrers can seriously impact site performance and security. Here are some key reasons why they need to be blocked:

Overload Web Servers – Scrapers, spam bots, and vulnerability scanners can generate huge volumes of requests and overload web servers. This drains resources and affects site speed for real visitors.
Steal Bandwidth – Bandwidth consumed by bad bots costs money and provides no value. A study by Distil Networks found over 30% of website traffic came from "nuisance" bots.
Steal Content – Content scrapers and aggregators steal and republish original content without permission, affecting SEO and monetization.
Spread Malware – Malicious bots probe sites for vulnerabilities and can be used to spread malware, redirect visitors, or stage attacks.
Skew Analytics – Bad bot traffic gives false inflated numbers in site analytics, obscuring real human traffic metrics. Up to 40% of analytics traffic could be fake according to studies.

Clearly, blocking unwanted traffic can directly improve performance, security, and get accurate site metrics.

Apache – Leveraging mod_rewrite

The Apache web server powers over 30% of all active websites. Here is how to block bad bots in Apache:

Enable mod_rewrite if not already active by adding:

LoadModule rewrite_module modules/mod_rewrite.so

Add blocking rules to your .htaccess file or VirtualHost section:

# Block User-Agents
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} badbot [NC,OR]
RewriteCond %{HTTP_USER_AGENT} bot\.io [NC] 
RewriteRule ^ - [F]

# Block Referrers
RewriteCond %{HTTP_REFERER} badsite\. [NC,OR]  
RewriteCond %{HTTP_REFERER} spambot\.com [NC]
RewriteRule ^ - [F]

This blocks the user-agent "badbot" and "bot.io", and referrers from "badsite." and "spambot.com". The [F] flag blocks the request.

Restart Apache for new rules to take effect.

Tip: Check access logs regularly for any new bad bots and referrers to block proactively.

Nginx – Returning 403/404 Errors

Nginx powers over 30% of the top 10 million websites. Here is how to block unwanted traffic in Nginx configs:

Add the following inside your server { } contexts:

# Block User-Agents
if ($http_user_agent ~ "badbot|scraper") {
  return 403; 
}

# Block Referrers
if ($http_referer ~ "badsite\.com") {
 return 404;
}

Save Nginx config file and reload/restart Nginx.

This will return 403 errors for the badbot and scraper user-agents, and 404 errors for the referrer.

Pro Tip: Use Nginx map directive to externalize the bot and referrer blacklist for easy maintenance:

map $http_user_agent $bad_useragent {
  default 0;

  "~badbot" 1;
  "~scraper" 1; 
}

server {
  if ($bad_useragent) {
    return 403;
  }
}

WordPress – Leverage Security Plugins

For WordPress sites, security plugins like Wordfence offer easy bot blocking:

Install and activate Wordfence plugin.
Go to Wordfence "Blocking" settings.
Enable blocking for:

Known Bots
Common Scanners

Customize blacklist to block other specific agents or referrers.

This leverages Wordfence‘s curated blacklist of known bad bots and referrers for effective blocking at the application layer.

Pro Tip: Use a Web Application Firewall (WAF) like Wordfence Threat Defense for advanced bot protection based on behavior, IP reputations, country blocking, and malicious payloads.

Going Beyond – Layered Defense

For optimal protection, I recommend implementing bot blocking at multiple layers:

Edge Network – Block bad bots closest to the source at your CDN, load balancer or firewall.
Web Server – Second layer of defense at the web server as discussed above.
Application – Third layer at the app/CMS itself via plugin or custom code.

Statistics:

30%+ of website traffic is from "nuisance" bots (Distil Networks)
40%+ of analytics traffic could be fake/bots (Industry studies)
10x+ improvement in site performance by blocking scrapers at the edge (Cloudflare)

So in summary, actively blocking unwanted bots and referrers can significantly improve your site‘s security posture, analytics accuracy, and performance. Employ multiple layers for optimal defense.

Let me know if you have any other questions! I‘m always happy to help fellow site owners better protect their online properties.