Free robots.txt tester and AI crawler checker: fetch or paste a robots.txt, test any URL, and see whether Googlebot, Bingbot, GPTBot, ClaudeBot, OAI-SearchBot and 11 more crawlers are allowed, with the exact rule that decided it.
Test a robots.txt
Search and AI crawlers on /
Crawler
What it does
Status
Matching follows RFC 9309 and Google's documented rules: the crawler's own User-agent group wins over *, the longest matching path wins, and Allow wins a tie. Each crawler's operator decides how strictly it follows robots.txt.
How to use this robots.txt tester
01
Fetch or paste the file
Enter a site URL to load its live /robots.txt, or paste a draft you have not published yet. Edits re-test instantly.
02
Pick a URL and a crawler
Type any path or full URL, choose a crawler, and the verdict shows whether it is allowed plus the exact line that decided it.
03
Check every AI crawler at once
The table tests the same URL against Google, Bing and the main AI crawlers, so you can see who can train on it, cite it, or check your ads.
Training, search and ads use different crawlers
Most AI companies now run more than one crawler. OpenAI uses GPTBot to collect training data, OAI-SearchBot to find pages for ChatGPT search, ChatGPT-User when someone asks ChatGPT to open a link, and OAI-AdsBot to check the landing pages of ChatGPT ads. Anthropic and Perplexity split their crawlers the same way. A single Disallow: / aimed at the wrong one can quietly remove you from an AI assistant's answers, or stop your ads from running.
This tester applies the same matching rules Google documents and RFC 9309 standardises: a crawler uses the group with its own name before falling back to User-agent: *, the longest matching rule wins, and Allow wins a tie. It shows the deciding line for every crawler, so you can see why a URL is blocked, not just that it is.
Everything runs in your browser. The optional fetch reads a site's public robots.txt through the same small server function the SERP simulator uses, since a browser cannot read another site's files directly. Want to check how your page looks once it is crawled? Run it through the AI Overview checker.
Common questions
A plain text file at the root of your site, such as example.com/robots.txt, that tells crawlers which URLs they may request. It holds groups that start with a User-agent line naming a crawler, followed by Allow and Disallow rules. It controls crawling, not indexing: a blocked URL can still appear in Google if other sites link to it.
Google first picks the group whose User-agent matches the crawler's name, ignoring case, and falls back to the User-agent: * group if none does. Inside that group the most specific rule wins, meaning the longest matching path. If an Allow and a Disallow rule match with the same length, Allow wins. This tester applies exactly that logic and shows you the line that decided the result.
Add a group for each crawler you want to block, for example User-agent: GPTBot followed by Disallow: /, then the same for ClaudeBot, CCBot and the others. Blocking the training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot) keeps your content out of model training without touching Google Search. Paste your file above and check the table to confirm each one shows as blocked.
No. GPTBot only collects training data. ChatGPT search uses a separate crawler, OAI-SearchBot, and ChatGPT ads check landing pages with OAI-AdsBot. You can block GPTBot and still allow OAI-SearchBot so ChatGPT can find and cite your pages. The same split exists at Anthropic (ClaudeBot for training, Claude-SearchBot for search) and Perplexity.
Google-Extended is not a crawler with its own visits. It is a robots.txt token Googlebot reads to decide whether your content can be used to train and ground Gemini models. Blocking it has no effect on how your pages rank or appear in Google Search. Applebot-Extended works the same way for Apple's AI features.
If /robots.txt returns a 404 or another 4xx error, crawlers treat the whole site as open to crawling. A 5xx server error is different: Google treats an unreachable robots.txt as a reason to slow down or pause crawling the entire site until the file comes back. This tester reports the HTTP status when you fetch a site.
Neither. Google ignores Crawl-delay, though Bing and some other crawlers honour it. Google stopped supporting noindex in robots.txt in September 2019; to keep a page out of the index, use a robots meta tag or an X-Robots-Tag header and leave the page crawlable so Google can see it.
Google reads the first 500 KiB of a robots.txt file and ignores every rule after that. Most files are a few hundred bytes, so this only matters for sites that generate huge rule lists. The tester flags any file over the limit.
No. Parsing and matching run entirely in your browser. The only network request is the optional fetch, which reads a site's public robots.txt through a small server function, because browsers cannot read another site's files directly. Nothing you test is logged or stored.