robots.txt and AI bot tester

How robots.txt matching actually works

A crawler reads your file and picks one group of rules, then uses only that group. It looks for its own name in a User-agent line, matched without regard to case. If it finds one, the * group is ignored completely, even when * is the stricter of the two. This catches people out. Add a Googlebot group to allow one folder, and Googlebot stops reading every other rule you wrote under *.

Inside the group, the longest matching rule wins. Allow: /blog/post beats Disallow: /blog/ for the URL /blog/post, because the allow pattern is longer. On an equal-length tie, Allow wins. Two characters do extra work: * stands for any run of characters, and $ pins the pattern to the end of the URL. So Disallow: /*.pdf$ blocks /guides/price-list.pdf but leaves /guides/price-list.pdf?v=2 open, because the query string sits after the .pdf. A Disallow: with nothing after it blocks nothing.

Rules are prefix matches. Disallow: /private also blocks /privateer and /private-clients, so put the trailing slash in when you mean the folder.

Blocking a crawler is not the same as hiding a page from search

robots.txt controls crawling. It says nothing about indexing. If a blocked URL picks up links from other sites, Google can still list it, usually with no description and sometimes with a title pulled from the anchor text. There is a worse version of this problem: because Google cannot fetch the page, it never sees the noindex tag you put there, so the two instructions cancel each other out.

If you want a page kept out of search, let crawlers fetch it and serve <meta name="robots" content="noindex"> or an X-Robots-Tag header. Keep robots.txt for crawl waste: filtered category URLs, internal search results, endless calendar pages. For anything confidential, use a password. robots.txt is a public file and a polite request, and plenty of scrapers ignore it.

What blocking the AI crawlers does and does not do

The AI companies split their work across separate tokens, so one line rarely does what people expect. GPTBot collects pages for training. OAI-SearchBot builds the index behind ChatGPT search, and ChatGPT-User fetches your page live when someone asks about you in a chat. Block GPTBot and you are out of the training data. Block the other two and you are out of ChatGPT's answers, which is where the referral traffic comes from. Anthropic splits the same way, with ClaudeBot for training and Claude-SearchBot and Claude-User for answers.

Google-Extended is not a crawler. It is a switch telling Google whether Gemini and AI Overviews may use content Googlebot has already fetched, and blocking it changes nothing about your ordinary rankings. Applebot-Extended works the same way for Apple. What no robots.txt rule can do is pull your content out of a model that has already been trained on it. Decide bot by bot. Most of the businesses we work with block the training crawlers and leave the search ones open.

The mistakes that cost people traffic

Disallow: /wp-content/ is the common one on WordPress. It hides your images from Google Images and stops Google loading your CSS and JavaScript, so the page renders in testing as a broken skeleton and the mobile checks fail. Leave /wp-content/ open. Block /wp-admin/ instead, with the usual Allow: /wp-admin/admin-ajax.php exception.

After that: a Disallow: / left behind from staging after launch, rules sitting in the wrong group so they never apply to anyone, a file uploaded to /blog/robots.txt where it does nothing because only the root file counts, and a subdomain running its own file that nobody has read in years. Check the file the day a site goes live, and again after any migration.