Not all automated traffic is welcome. Some ignores robots.txt entirely, so blocking has to happen at the server.
Ask politely first
robots.txt is the correct first step and well-behaved crawlers respect it:
User-agent: BadBotName Disallow: /
You can also ask for a slower crawl rate. Bots that ignore this need a firmer approach.
Blocking by user agent
In .htaccess:
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (BadBot|AnotherBot) [NC]
RewriteRule .* - [F,L]
They receive a 403 and use almost no resources.
Blocking by address
Useful for a single persistent source, but most aggressive scraping comes from many addresses, so this is often a losing battle.
Identify before you block
Look at your raw access logs, or the visitor statistics in cPanel, to see which user agents are actually consuming your traffic. Blocking a bot you guessed at achieves nothing.
Be careful
- Never block Googlebot or Bingbot unless you want to disappear from search.
- Verify before blocking something claiming to be a search engine – bad bots frequently pretend to be Google.
- Do not block by user agent so broadly that you catch real browsers.
Caching is often the better answer
A cached page costs almost nothing to serve. Good caching can make bot traffic a non-issue without blocking anything – and without the risk of blocking something you needed.
