Can AI crawlers access your site?

What this does not check

It does not check whether AI answers mention your brand, which is a visibility question that Ahrefs, Semrush and a dozen others already answer well, for free. This answers the question underneath: whether the crawlers are permitted to read the site at all. If the door is locked, nothing you do about the other question will open it, and the answer sits in one file that anyone can read.

What robots.txt does, and does not do

It controls crawling. Nothing else.

A Disallow line asks a well-behaved crawler not to fetch those paths in future. That is the whole mechanism.

It does not undo what a model already learned.

Pages that were crawled before the rule was added are already in whatever training sets they were in. Blocking now changes nothing about that.

It does not stop citation.

An answer engine can still describe and link your site from other sources: pages that quote you, directories, social posts, archives. It just cannot read the page itself.

Google-Extended and Applebot-Extended are training opt-outs.

They have no effect on ordinary search crawling. Googlebot and Applebot still index the site for search whatever these two say. Blocking them changes only whether the content is used to train models.

Being read by answer engines hinges on the search and user-fetch tokens.

OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes; ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks about it. Those are the rows to read first if you want to be found.

What is stored

Nothing. The check fetches robots.txt, llms.txt and the homepage of the URL you enter, over http or https only, from a public address only, and shows you the result. It is not written to a database and there is no link to come back to. Run it again any time.

Want the wider picture? The channel report reads the same site and says which of the 19 distribution channels are open to it.