Who can crawl your site?

robots.txt is a plain text file at the root of a site, such as https://example.com/robots.txt, that tells crawlers which paths they may fetch. Each group names a user agent and lists Allow and Disallow rules. RFC 9309 defines the format: the most specific matching rule wins, and a crawler with no group of its own follows the * group.

It is a request, not access control. A polite crawler obeys it, a rude one fetches the page anyway, and no header or firewall is involved. A missing file, HTTP 404, means every path is allowed. A file that answers a server error is different: RFC 9309 tells crawlers to treat the site as fully disallowed until it can be read.

Twelve named agents in four groups. Traditional search: Googlebot and Bingbot. AI search: OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Training and datasets: GPTBot, ClaudeBot, Google-Extended, and CCBot. User-triggered fetches: ChatGPT-User, Claude-User, and Perplexity-User. Each row links the provider's own crawler documentation, and the check also reads the page's robots meta tag and X-Robots-Tag header for indexing directives.

robots.txt controls fetching. A noindex directive, in a robots meta tag or an X-Robots-Tag header, controls indexing: the crawler may fetch the page but must not list it. The two do not combine the way people expect. A page blocked in robots.txt is never fetched, so its noindex is never seen, and Google can still index the address from links alone. To keep a page out of results, allow the crawl and send noindex.

Blocked means the published robots.txt has a Disallow rule that matches the checked page's final path for that crawler's user agent, and no more specific Allow rule overrides it. The row quotes the group and the exact rule. Allowed means no rule blocks the path, or the file is missing. Unknown means the file could not be read in this check. Not controlled marks fetchers whose provider says robots.txt may not apply to them.

The headline verdict reads search and AI search only: fail means every search crawler checked is blocked or the page sends noindex, and warn means the picture is mixed. Training and dataset rows are reported without judgment, because whether to allow GPTBot or CCBot is the owner's choice, not a fault.

Find the rule the row quotes and change it. To open one path for every crawler, add an Allow: line for that path under User-agent: *; a longer matching pattern wins over a shorter one. To open one crawler, give it its own group, because a crawler with a group of its own ignores the * group entirely. Paste the draft into the simulator under the result before you deploy: it runs the same matcher in your browser against the same agents and shows what changes.

The simulator starts with the checked page's final path. You can replace it with another path and optional query, up to 2,048 characters, to test the draft rules locally. A different path has no published-file comparison and no fresh HTML or response-header check. Unknown published decisions are never counted as newly allowed or blocked. Editing either the path or rules clears the old simulation until you run it again.

Google refreshes its copy of robots.txt about once a day, so a change can take that long to reach the crawler. This check reads the file the site serves now, so run it again after the deploy to confirm the live rules.

No. Allowed means the published rules do not stop the crawler at that path. It does not prove the crawler gets through a firewall, a CDN rule, a bot challenge, or a login, and it does not promise indexing, ranking, a citation in an AI answer, or use in model training. The check reads one page and the redirects to it; other paths on the site can carry other rules.

Yes. No account, no card, no premium plan, no upgrade nag, no hidden paywall. IndieCrawl is free, all of it.

There is no public lookup history. A network check sends the public input needed for that check to its named source. Local file tools process file contents in your browser. Analytics is currently off, so no visit or job data is sent to a tracker. Anything the tool saves for you stays in your browser.