Enter a domain and we fetch its robots.txt, list every rule by user agent, flag syntax errors and risky blocks, and test any URL against Googlebot, Bingbot, GPTBot, ClaudeBot and more.
Type the domain and press Check. We fetch robots.txt over HTTPS, then HTTP if needed, and show the file with line numbers, the groups it defines, the sitemaps it lists, and every problem we found: lines no crawler can parse, rules that block CSS and JavaScript, a Disallow that hides the whole site. Then type any path or URL, choose a bot, and see whether it is allowed and which line decided it. The test uses the same longest-match rules Google does.
A robots.txt error is silent. The site keeps working, the pages keep loading, and Google quietly stops crawling the folder with your product pages in it, or reads a file that a stray character has turned into no rules at all. Which bots you block matters more than it used to: GPTBot, ClaudeBot and PerplexityBot decide whether your site can be cited in AI answers. Checking takes seconds, and it is the first thing to do when traffic drops without a reason.
Enter your domain above, then type the URL or path into the test box and choose the crawler. The result shows allowed or blocked and the exact line that matched. Longer rules beat shorter ones, and Allow beats Disallow when they tie, which is how Google resolves it.
Everything on the site is off limits to that user agent. Under User-agent: * it blocks every crawler that obeys robots.txt. It is the most common way a launch goes wrong: a staging rule copied to production.
Yes. Test against GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended and CCBot to see exactly what each one may read. Blocking the training bots and allowing the search bots is a common middle ground.