Prevent bad bots from crawling your website


Prevent bad bots from crawling your website

Bad bots here, I mean bot like AhrefsBot, which sucked all my VPS resources out, including but not limited to CPU usage up to 30%.

So I decided to block it from crawling my website for now.

Tried the following two steps, and let us wait to see.

  1. Add rules below to your robots.txt
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /

  1. Add rules for your nginx configuration

if ($http_user_agent ~* AhrefsBot) {
return 403;
}

if ($http_user_agent ~* MJ12bot) {
return 403;
}

Leave a comment

Your email address will not be published. Required fields are marked *