Web Crawler Simulator
Test how 20+ search engine and AI crawlers see your site. Verify robots.txt rules, check content visibility, and optimize for better indexing.
Test any crawler
Select a crawler and enter your URL to see what the bot sees.
AI Assistants & LLMs
Crawlers used by AI chatbots and large language models to fetch web content for answers and training.
User agent used by ChatGPT to fetch pages for AI answers and...
User agent used by Claude web to fetch pages for AI answers ...
User agent used by Grok to fetch pages for AI answers and pr...
Perplexity's crawler user agent used to answer questions and...
Cohere's primary crawler user agent in their AI services/too...
Allen Institute's AI2Bot crawler, used for research corpora ...
Search Engines
Traditional search engine crawlers that index your content for search results.
Primary user agent used by Google's indexing bot, Gemini, AI...
User agent used by Bing's indexing bot. Also powers Microsof...
DuckDuckGo's primary indexing bot user agent....
Brave's primary crawler user agent for their search engine....
Yandex's primary indexing bot user agent....
You.com's bot used for search and preview generation, fetchi...
Social Media
Crawlers that fetch Open Graph data for link previews on social platforms.
Voice Assistants & Enterprise
Crawlers for voice search, smart speakers, and enterprise applications.
Data & Research
Crawlers used for web archiving, research datasets, and AI training corpora.
All Supported Crawler User Agents
| Crawler | User-Agent | Purpose | Simulate |
|---|---|---|---|
|
ChatGPT Fetch |
Mozilla/5.0 (compatible; ChatGPT-User/1.... |
User agent used by ChatGPT to fetch pages for AI a... | Test |
|
Claude Fetch |
Claude-Web/1.0... |
User agent used by Claude web to fetch pages for A... | Test |
|
Grok Search |
Grok-Search/1.0... |
User agent used by Grok to fetch pages for AI answ... | Test |
|
PerplexityBot |
PerplexityBot/1.0 (https://www.perplexit... |
Perplexity's crawler user agent used to answer que... | Test |
|
Googlebot |
Mozilla/5.0 (compatible; Googlebot/2.1; ... |
Primary user agent used by Google's indexing bot, ... | Test |
|
Bingbot |
Mozilla/5.0 (compatible; bingbot/2.0; +h... |
User agent used by Bing's indexing bot. Also power... | Test |
 Facebook Crawler |
facebookexternalhit/1.1 (+http://www.fac... |
Fetches pages to render Open Graph previews on Fac... | Test |
 LinkedInBot |
LinkedInBot/1.0 (+http://www.linkedin.co... |
Fetches pages to render Open Graph previews on Lin... | Test |
 Amazonbot |
Amazonbot/1.0 (+https://developer.amazon... |
Amazon's crawler used for product and content disc... | Test |
 Applebot |
Mozilla/5.0 (compatible; Applebot/0.1; +... |
Apple's crawler used for Siri and Spotlight sugges... | Test |
 Bytespider |
Bytespider; (+http://www.bytedance.com/s... |
ByteDance's web crawler for AI model training and ... | Test |
|
DuckDuckBot |
DuckDuckBot/1.0; (+http://duckduckgo.com... |
DuckDuckGo's primary indexing bot user agent.... | Test |
|
CohereAI |
CohereAI/1.0... |
Cohere's primary crawler user agent in their AI se... | Test |
|
AI2Bot |
AI2Bot/1.0 (+http://www.allenai.org)... |
Allen Institute's AI2Bot crawler, used for researc... | Test |
 CCBot |
CCBot/2.0 (+https://commoncrawl.org/faq/... |
Open web-scale archive crawler used to build publi... | Test |
 Diffbot |
Diffbot/1.0 (+http://www.diffbot.com)... |
Diffbot's primary crawler user agent, used for web... | Test |
 Omgili |
omgili... |
Aggregated discussion indexer bot.... | Test |
|
TimpiBot |
TimpiBot/1.0... |
Independent web indexer bot.... | Test |
|
YouBot |
YouBot/1.0 (+https://you.com/bot)... |
You.com's bot used for search and preview generati... | Test |
|
|
Bravebot/1.0... |
Brave's primary crawler user agent for their searc... | Test |
|
|
Mozilla/5.0 (compatible; YandexBot/3.0; ... |
Yandex's primary indexing bot user agent.... | Test |
More Free SEO Tools
Continue auditing your site with these companion tools.
Related Guides
Dive deeper with these step-by-step guides and tutorials.
Crawler Simulator FAQ
Make your site crawler-friendly
JavaScript sites often have indexing issues. Encited (formerly LovableHTML) pre-renders your SPA into crawler-friendly HTML so every bot can read your content.
