# IndieCrawl > A free crawler access check for search engines and AI bots. See what robots.txt, robots meta, and X-Robots-Tag say about one URL. The canonical page is https://indiecrawl.com/. ## Use the tool Open [IndieCrawl](https://indiecrawl.com/) to complete the job. The questions below describe the supported input, result, and limits. Every answer names its source next to it. Source: [The checked website and RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html). ## Questions and limits ### What is robots.txt? robots.txt is a plain text file at the root of a site, such as `https://example.com/robots.txt`, that tells crawlers which paths they may fetch. Each group names a user agent and lists `Allow` and `Disallow` rules. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) defines the format: the most specific matching rule wins, and a crawler with no group of its own follows the `*` group. It is a request, not access control. A polite crawler obeys it, a rude one fetches the page anyway, and no header or firewall is involved. A missing file, HTTP 404, means every path is allowed. A file that answers a server error is different: RFC 9309 tells crawlers to treat the site as fully disallowed until it can be read. ### Which crawlers does the check cover? Twelve named agents in four groups. Traditional search: Googlebot and Bingbot. AI search: OAI-SearchBot, Claude-SearchBot, and PerplexityBot. Training and datasets: GPTBot, ClaudeBot, Google-Extended, and CCBot. User-triggered fetches: ChatGPT-User, Claude-User, and Perplexity-User. Each row links the provider's own crawler documentation, and the check also reads the page's robots `meta` tag and `X-Robots-Tag` header for indexing directives. ### What is the difference between robots.txt and noindex? robots.txt controls fetching. A `noindex` directive, in a `robots` meta tag or an `X-Robots-Tag` header, controls indexing: the crawler may fetch the page but must not list it. The two do not combine the way people expect. A page blocked in robots.txt is never fetched, so its noindex is never seen, and [Google can still index the address](https://developers.google.com/search/docs/crawling-indexing/block-indexing) from links alone. To keep a page out of results, allow the crawl and send noindex. ### What does blocked mean in the result? Blocked means the published robots.txt has a `Disallow` rule that matches the checked page's final path for that crawler's user agent, and no more specific `Allow` rule overrides it. The row quotes the group and the exact rule. Allowed means no rule blocks the path, or the file is missing. Unknown means the file could not be read in this check. Not controlled marks fetchers whose provider says robots.txt may not apply to them. The headline verdict reads search and AI search only: fail means every search crawler checked is blocked or the page sends noindex, and warn means the picture is mixed. Training and dataset rows are reported without judgment, because whether to allow GPTBot or CCBot is the owner's choice, not a fault. ### How do I unblock a crawler in robots.txt? Find the rule the row quotes and change it. To open one path for every crawler, add an `Allow:` line for that path under `User-agent: *`; a longer matching pattern wins over a shorter one. To open one crawler, give it its own group, because a crawler with a group of its own ignores the `*` group entirely. Paste the draft into the simulator under the result before you deploy: it runs the same matcher in your browser against the same agents and shows what changes. The simulator starts with the checked page's final path. You can replace it with another path and optional query, up to 2,048 characters, to test the draft rules locally. A different path has no published-file comparison and no fresh HTML or response-header check. Unknown published decisions are never counted as newly allowed or blocked. Editing either the path or rules clears the old simulation until you run it again. Google refreshes its copy of robots.txt about once a day, so a change can take that long to reach the crawler. This check reads the file the site serves now, so run it again after the deploy to confirm the live rules. ### Does allowed mean my page will be indexed? No. Allowed means the published rules do not stop the crawler at that path. It does not prove the crawler gets through a firewall, a CDN rule, a bot challenge, or a login, and it does not promise indexing, ranking, a citation in an AI answer, or use in model training. The check reads one page and the redirects to it; other paths on the site can carry other rules. ## Promises - The core is free: no account, no card, no premium plan, no upgrade nag, no hidden paywall. - Private lookup activity is never published. - Analytics is currently off. Private lookup activity is never published or sent to a tracker. No lookup history becomes public inventory. ## More - [Notes](https://indiecrawl.com/blog.md): posts by the maker, newest first - [More tools](https://indiecrawl.com/tools.md): the other tools by turushan - [security.txt](https://indiecrawl.com/.well-known/security.txt): how to report a vulnerability Made by turushan, https://twitter.com/turushan.