Robots.txt Rules for AI Crawlers

Robots.txt is a discovery gate

Robots.txt tells compliant crawlers which paths they may request. It does not add rankings by itself, and it does not guarantee that a crawler will use your content, but it can create a hard blocker. If important pages are disallowed, search engines and AI crawlers may not be able to fetch the content needed to understand your site.

What to allow and what to block

Most public marketing sites should allow the homepage, resources, blog posts, documentation, pricing, about and contact pages. Private admin routes, APIs, internal dashboards and lead tables should usually be blocked or protected. The key is not to block broad folders that contain public content.

• Allow public content pages
• Block private dashboards and admin paths
• Avoid blocking CSS or JavaScript needed for rendering
• Include the sitemap URL
• Test important URLs after every robots.txt change

Common mistake: blocking too much

A broad disallow rule such as Disallow: /resources or Disallow: /blog can quietly remove valuable content from crawler access. Another mistake is assuming that allowing one bot means every crawler has the same permissions. Review both the general User-agent: * rule and any crawler-specific rules.

How ScanSiteAI uses robots.txt checks

ScanSiteAI reviews whether key crawler families can access public pages and whether blocked paths make sense. A good setup usually allows normal public content, blocks private areas and exposes sitemap.xml. After changes, inspect affected URLs in Google Search Console and run a fresh scan to confirm the signal changed.