Findings · AI Search
robots.txt allows AI search bots
For AI engines like ChatGPT, Claude, and Perplexity to cite your site in their answers, their crawlers need to be allowed in robots.txt. Default-allow is fine; explicit Disallow blocks them entirely.
Why it matters
This check belongs to AI Search — can chatgpt, claude, perplexity cite it? It's rated medium severity. For AI engines like ChatGPT, Claude, and Perplexity to cite your site in their answers, their crawlers need to be allowed in robots.txt. Default-allow is fine; explicit Disallow blocks them entirely.
The fix
Vettd's Fix Pack includes a copy-paste-ready solution for this check — auto-generated for your specific site, in the right format for your stack. See pricing, or scan first to see what your site needs.
Check your own site for this.
Vettd runs this check (and 81 others) on any URL. Free, no signup, ~30 seconds.
Scan your site →