Check a site’s robots.txt and sitemaps
Paste any URL to read the site’s robots.txt, see the rules grouped by crawler, check whether the AI and answer-engine crawlers are allowed, and count the URLs its sitemaps declare.
How this tool works, and what it cannot see
It fetches /robots.txt at the origin, parses the user-agent groups, then fetches each declared sitemap (or /sitemap.xml if none is declared) and counts <loc> entries.
The AI crawler table is a literal check for a full disallow of each named agent — it does not guess intent from wildcards.
A sitemap that declares thousands of URLs is not the same as thousands of indexed pages. Use Search Console for what is actually indexed.
FAQ
Should I block GPTBot and the other AI crawlers?
It is a trade, not a best practice: blocking them keeps your text out of training and out of AI answers, which is also where a growing share of discovery happens. Most publishers allow the search-facing agents (OAI-SearchBot, PerplexityBot) and decide separately about training agents.
Does robots.txt stop a page being indexed?
No. It stops crawling, and an uncrawled URL can still be indexed from links pointing at it. To keep a page out of the index use a noindex robots meta tag on a page that is crawlable.
Why does my sitemap show fewer URLs than my site has?
Either it is a sitemap index (each entry is another sitemap, which this tool tells you), or your generator is excluding pages — often paginated archives or anything with a noindex tag.