See exactly what GPTBot, ClaudeBot, PerplexityBot see on your site — robots.txt verdict + content
For AI search and citation to surface your content, the crawlers need to fetch it successfully and extract meaningful text. Many sites quietly fail this: a staging robots.txt blocks GPTBot, a client-rendered app shows blank HTML to PerplexityBot, a CDN serves a challenge page to ClaudeBot. This tool tests all 11 major AI crawlers in parallel and shows exactly what each sees.
Client-rendered SPAs often send a nearly-empty HTML shell to bots (which don't run JavaScript). If visible word count is near zero, you're invisible to AI crawlers. Solutions: server-render, use SSG, or emit pre-rendered content with a bot user-agent check.
For AI citations: GPTBot + OAI-SearchBot (ChatGPT), ClaudeBot (Anthropic), PerplexityBot, Google-Extended (Gemini), Applebot-Extended (Siri). Googlebot still dominates search but is converging with these.
No — if robots.txt is absent, crawling is implicitly allowed. But you lose the ability to declare preferences (like blocking training-data bots while allowing search bots). Best practice: always ship a robots.txt.
Use our robots.txt generator's 'Block all AI crawlers' preset. Note: only compliant bots honor robots.txt. Bad actors ignore it.
The first 2,000 characters of rendered text the crawler receives — exactly what gets indexed and potentially cited.
