Prevent bad bots from crawling your website
Bad bots here, I mean bot like AhrefsBot, which sucked all my VPS resources out, including but not limited to CPU usage up to 30%.
So I decided to block it from crawling my website for now.
Tried the following two steps, and let us wait to see.
- Add rules below to your robots.txt
User-agent: AhrefsBot
Disallow: /
User-agent: MJ12bot
Disallow: /
- Add rules for your nginx configuration
if ($http_user_agent ~* AhrefsBot) {
return 403;
}
if ($http_user_agent ~* MJ12bot) {
return 403;
}