Bot Rules

Generate robots.txt, Cloudflare WAF, and Apache .htaccess rules from a reviewed bot catalogue.

Bot Rules

Generate Rules To Block Unwanted Bots

Open Bot Rules

Bot Rules helps you block scrapers, AI crawlers, and other unwanted bots. Search a reviewed catalogue, choose the bots you want to block, and copy a robots.txt, Cloudflare WAF, or Apache .htaccess snippet. robots.txt is a request. Cloudflare and .htaccess can enforce the block.

  • Browse the catalogue - Open Bot Rules and search or filter by name, category, country, or how often major sites already disallow the bot in robots.txt. That figure is not a risk score.
  • Select your bots - Add individual bots, or use Add this page to add the current results. Leave crawlers you rely on, such as Googlebot, off the list.
  • Choose an output - Generate robots.txt, a Cloudflare WAF custom rule, or an Apache .htaccess snippet.
  • Copy and review - Paste the result into your site, Cloudflare dashboard, or Apache configuration. Review every line before you deploy.

Things To Remember

  • robots.txt - This asks well-behaved crawlers to stay away. Many scrapers ignore it. Merge the fragment into your existing robots.txt and keep rules you still need. More information on robots.txt
  • Cloudflare WAF - A deployed custom rule can block matching traffic at the Cloudflare edge, before it reaches your server. Custom rules are often limited to about 4,096 characters, so a large selection may need splitting. Cloudflare WAF rules | Why use Cloudflare WAF for security
  • Apache .htaccess - This can enforce a block on Apache 2.4 when mod_setenvif, mod_authz_core, and AllowOverride are available. The request still reaches your server. It is for Apache only. Nginx and IIS need different configuration. Merge the snippet into the existing file and keep a backup, because a mistake can make the site unreachable.

You can use more than one layer, for example robots.txt plus Cloudflare WAF or .htaccess. They are not a substitute for a full security programme.

Open Bot Rules

A Web Application Firewall (WAF) monitors and filters HTTP traffic between a web application and the internet. It operates at the application layer and can help block unwanted clients before they reach the application.

Thanks

Catalogue observations are collated with thanks to sources such as Known Agents.

If you need Nginx configuration, this GitHub project is a wider Nginx bad-bot blocking stack. Bot Rules does not generate Nginx rules.

Disclaimer

Generated output is a starting point, not a managed security service. Check that the rules suit your setup before you deploy them. Blocking the wrong crawler can affect availability or search visibility, and a faulty .htaccess rule can make a site unreachable.