Free robots.txt checker for AI
AI crawler checker: which AIs can read your site
Paste your website. We read your robots.txt as each AI company's crawler would, and show which can find you, which can train on you, and the line that decides it.
Or paste your robots.txt
Free, no sign-up, no email. Most answers take a few seconds.
codarus.com
We haven't checked codarus.com with this tool yet. Press the button above to check it now.
Three kinds of AI crawler
Search crawlers find your pages for AI answers. Training crawlers collect pages to train models. Fetchers open a page when a person asks an assistant about it. Blocking one kind doesn't block the others.
How we read it
By the standard (RFC 9309), as each company's crawler reads it: the group that names it, else the * group; the longest matching rule wins. For each crawler we show the line that decides. We also ask your home page once with our own signed crawler, to see whether your host lets crawlers in.
What it doesn't do
We never pretend to be another company's bot. And robots.txt is a request, not a lock: some fetchers say they may not follow it, and a host or firewall can block crawlers whatever it says.
Questions
If I block GPTBot, do I disappear from ChatGPT?
No. OpenAI says GPTBot is for training and OAI-SearchBot for ChatGPT search, and each setting is separate. Block GPTBot to say no to training; keep OAI-SearchBot allowed to be found.
Does blocking Google-Extended remove me from Google?
No. Google says Google-Extended doesn't affect inclusion or ranking in Search; it controls use for Gemini training and grounding. AI Overviews draw on the normal Search index.
Why can an AI still open my page after I blocked it?
Fetchers that act when a person asks, such as ChatGPT-User and Perplexity-User, may not follow robots.txt, their owners say. And robots.txt is a request, not a lock.
My robots.txt allows everything, but AIs still can't read my site. Why?
Your host or firewall may block bots before robots.txt matters, for example a CDN setting that blocks AI bots. We tell you if our own signed crawler was stopped.
What if my robots.txt returns an error?
If /robots.txt answers with a server error (5xx), the standard (RFC 9309) tells crawlers to treat the whole site as off-limits. A missing file (404) means everything is allowed.
Should I block AI crawlers?
It's your call. Blocking training crawlers keeps your pages out of future models; blocking search crawlers keeps you out of AI answers. You can do one without the other.
More free tools
- Llms.txt checker
- Llms.txt generator
- MCP server tester
- UCP and ACP checker
- Agent inbox checker
- The full AI agent readiness check
- All free tools
Guides