Can AI answer engines read your site?

What robots.txt does, and does not do

It controls crawling. Nothing else.

A Disallow line asks a well-behaved crawler not to fetch those paths in future. That is the whole mechanism.

It does not undo what a model already learned.

Pages that were crawled before the rule was added are already in whatever training sets they were in. Blocking now changes nothing about that.

It does not stop citation.

An answer engine can still describe and link your site from other sources: pages that quote you, directories, social posts, archives. It just cannot read the page itself.

Google-Extended and Applebot-Extended are training opt-outs.

They have no effect on ordinary search crawling. Googlebot and Applebot still index the site for search whatever these two say. Blocking them changes only whether the content is used to train models.

Being read by answer engines hinges on the search and user-fetch tokens.

OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes; ChatGPT-User, Claude-User and Perplexity-User fetch a page when someone asks about it. Those are the rows to read first if you want to be found.

What is stored

Nothing. The check fetches robots.txt, llms.txt and the homepage of the URL you enter, over http or https only, from a public address only, and shows you the result. It is not written to a database and there is no link to come back to. Run it again any time.

Want the wider picture? The channel report reads the same site and says which of the 19 distribution channels are open to it.