robots.txt tester & generator
Check whether Googlebot, Bingbot or AI crawlers like GPTBot and PerplexityBot can access a URL, spot errors in the file and create a new robots.txt.
Default policy
AI crawlers
Free runs left today: 40
Three steps, zero setup
Enter the domain
Or paste a robots.txt to test it before publishing.
Pick URL and user-agent
Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot or custom.
Read the verdict
Allowed or blocked, with the rule that decides it.
A tiny file with a huge impact
One wrong line in robots.txt can block crawling of your entire site, or shut out exactly the AI crawlers that feed ChatGPT and Perplexity. The tester applies Google's rules: the longest matching rule wins and, on a tie, Allow beats Disallow.
Besides the verdict for the selected user-agent, a table shows what the main search and AI bots can see, and syntax problems are flagged. In the Generate tab you can build a new file with your AI bot policy.
You might also need
Page speed & Core Web Vitals
Performance score, LCP, CLS, INP and Google's opportunities.
Redirect & status checker
301/302 chains, loops and status codes, including www/https variants.
On-page SEO checker
SEO score and a prioritized issue list for any URL.