AI Crawler Visibility Checker
See exactly what GPTBot, ClaudeBot, PerplexityBot see on your site — robots.txt verdict + content
How to use
- Paste your URL
- Click 'Test all AI crawlers'
- Review per-bot verdicts + content excerpts
Use cases
- AEO audits
- AI citation debugging
- robots.txt verification
About the AI Crawler Visibility Checker
For AI search and citation to surface your content, the crawlers need to fetch it successfully and extract meaningful text. Many sites quietly fail this: a staging robots.txt blocks GPTBot, a client-rendered app shows blank HTML to PerplexityBot, a CDN serves a challenge page to ClaudeBot. This tool tests all 11 major AI crawlers in parallel and shows exactly what each sees.
Features
- Tests against 11 real AI / search crawlers simultaneously
- Per-bot robots.txt verdict with matching rule
- Extracted title, H1, description, visible word count, JSON-LD count
- Detects x-robots-tag overrides
- Server-side fetch — bypasses CORS
How it works
- Paste the URL you want to audit.
- Click 'Test all AI crawlers'.
- For each bot, see HTTP status, robots.txt verdict, visible word count, JSON-LD count, and a content excerpt.
Frequently asked questions
Why might a bot see something different from my browser?
+
Client-rendered SPAs often send a nearly-empty HTML shell to bots (which don't run JavaScript). If visible word count is near zero, you're invisible to AI crawlers. Solutions: server-render, use SSG, or emit pre-rendered content with a bot user-agent check.
Which bots should I care about most?
+
For AI citations: GPTBot + OAI-SearchBot (ChatGPT), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini), Applebot-Extended (Siri). Googlebot still dominates search but is converging with these.
My robots.txt shows 'no-robots' — is that bad?
+
No — if robots.txt is absent, crawling is implicitly allowed. But you lose the ability to declare preferences (like blocking training-data bots while allowing search bots). Best practice: always ship a robots.txt.
Can I block all AI training data collection?
+
Use our robots.txt generator's 'Block all AI crawlers' preset. Note: only compliant bots honor robots.txt. Bad actors ignore it.
What does the content excerpt show?
+
The first 2,000 characters of rendered text the crawler receives — exactly what gets indexed and potentially cited.