Web Crawler Simulator
Test how 20+ search engine and AI crawlers see your site. Verify robots.txt rules, check content visibility, and optimize for better indexing.
Test any crawler
Select a crawler and enter your URL to see what the bot sees.
AI Assistants & LLMs
Crawlers used by AI chatbots and large language models to fetch web content for answers and training.
User agent used by ChatGPT to fetch pages for AI answers and...
User agent used by Claude web to fetch pages for AI answers ...
User agent used by Grok to fetch pages for AI answers and pr...
Perplexity's crawler user agent used to answer questions and...
Cohere's primary crawler user agent in their AI services/too...
Allen Institute's AI2Bot crawler, used for research corpora ...
Search Engines
Traditional search engine crawlers that index your content for search results.
Primary provider bucket for Google's indexing, inspection, a...
User agent used by Bing's indexing bot. Also powers Microsof...
DuckDuckGo's primary indexing bot user agent....
Brave's primary crawler user agent for their search engine....
Yandex's primary indexing bot user agent....
You.com's bot used for search and preview generation, fetchi...
Social Media
Crawlers that fetch Open Graph data for link previews on social platforms.
Voice Assistants & Enterprise
Crawlers for voice search, smart speakers, and enterprise applications.
Data & Research
Crawlers used for web archiving, research datasets, and AI training corpora.
All Supported Crawler User Agents
| Crawler | User-Agent | Purpose | Simulate |
|---|---|---|---|
|
Googlebot Smartphone |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus... |
Google Search's mobile-first indexing crawler as i... | Test |
|
Googlebot Desktop |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... |
Google Search's desktop indexing crawler as it app... | Test |
|
Google Inspection Tool Smartphone |
Mozilla/5.0 (Linux; Android 6.0.1; Nexus... |
Mobile user-agent used by Google Search Console UR... | Test |
|
Google Inspection Tool Desktop |
Mozilla/5.0 (compatible; Google-Inspecti... |
Desktop user-agent used by Google Search Console U... | Test |
|
Googlebot Image |
Googlebot-Image/1.0... |
Google crawler for images, logos, favicons, Discov... | Test |
|
Googlebot Video |
Googlebot-Video/1.0... |
Google crawler for video-related Search features a... | Test |
|
GoogleOther |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... |
Google's generic crawler used by product teams for... | Test |
|
Storebot-Google |
Mozilla/5.0 (X11; Linux x86_64; Storebot... |
Google Shopping crawler used across Shopping surfa... | Test |
|
OAI-SearchBot |
Mozilla/5.0 (Macintosh; Intel Mac OS X 1... |
OpenAI crawler used to surface websites in ChatGPT... | Test |
|
GPTBot |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... |
OpenAI crawler used for training and improving gen... | Test |
|
ChatGPT-User |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... |
OpenAI user-triggered fetcher used when ChatGPT or... | Test |
|
OAI-AdsBot |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... |
OpenAI fetcher used to validate landing pages subm... | Test |
|
ClaudeBot |
ClaudeBot/1.0... |
Anthropic's autonomous crawler for discovering and... | Test |
|
Claude-SearchBot |
Claude-SearchBot/1.0... |
Anthropic crawler associated with Claude search an... | Test |
|
Claude-User |
Claude-User/1.0... |
Anthropic user-triggered fetcher used when Claude ... | Test |
|
Claude-Web |
Claude-Web/1.0... |
Claude web fetcher used by Claude experiences when... | Test |
|
anthropic-ai |
anthropic-ai... |
Legacy Anthropic crawler identifier seen in server... | Test |
|
PerplexityBot |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... |
Perplexity crawler used to surface and link websit... | Test |
|
Perplexity-User |
Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... |
Perplexity user-triggered fetcher used to retrieve... | Test |
|
Grok Search |
Grok-Search/1.0... |
xAI Grok search fetcher used for AI answers and ci... | Test |
|
GrokBot |
GrokBot/1.0... |
xAI Grok crawler identifier seen in server logs.... | Test |
|
Grok DeepSearch |
Grok-DeepSearch/1.0... |
xAI Grok DeepSearch crawler/fetcher.... | Test |
|
Bingbot |
Mozilla/5.0 (compatible; bingbot/2.0; +h... |
Microsoft Bing's primary indexing crawler.... | Test |
|
BingPreview |
Mozilla/5.0 (Windows NT 10.0; Win64; x64... |
Microsoft Bing preview fetcher used for page previ... | Test |
|
Bravebot |
Bravebot/1.0... |
Brave Search's primary indexing crawler.... | Test |
|
Brave Search |
Brave-Search/1.0... |
Brave Search fetcher/crawler identifier used by Br... | Test |
|
DuckDuckBot |
DuckDuckBot/1.0; (+http://duckduckgo.com... |
DuckDuckGo's primary indexing crawler.... | Test |
|
DuckAssistBot |
DuckAssistBot/1.0... |
DuckDuckGo fetcher associated with DuckAssist answ... | Test |
 Meta-ExternalAgent |
Meta-ExternalAgent/1.1... |
Meta crawler used for external link previews and A... | Test |
|
Meta-ExternalFetcher |
Meta-ExternalFetcher/1.1... |
Meta fetcher used to retrieve external page metada... | Test |
|
FacebookBot |
FacebookBot/1.0... |
Facebook crawler identifier used for page fetches.... | Test |
|
facebookexternalhit |
facebookexternalhit/1.1 (+http://www.fac... |
Facebook/Instagram link preview crawler for Open G... | Test |
|
ChatGPT Fetch (default) |
Mozilla/5.0 (compatible; ChatGPT-User/1.... |
User agent used by ChatGPT to fetch pages for AI a... | Test |
|
Claude Fetch (default) |
Claude-Web/1.0... |
User agent used by Claude web to fetch pages for A... | Test |
|
Grok Search (default) |
Grok-Search/1.0... |
User agent used by Grok to fetch pages for AI answ... | Test |
|
PerplexityBot (default) |
PerplexityBot/1.0 (https://www.perplexit... |
Perplexity's crawler user agent used to answer que... | Test |
|
Googlebot (default) |
Mozilla/5.0 (compatible; Googlebot/2.1; ... |
Primary provider bucket for Google's indexing, ins... | Test |
|
Bingbot (default) |
Mozilla/5.0 (compatible; bingbot/2.0; +h... |
User agent used by Bing's indexing bot. Also power... | Test |
|
Facebook Crawler (default) |
facebookexternalhit/1.1 (+http://www.fac... |
Fetches pages to render Open Graph previews on Fac... | Test |
 LinkedInBot (default) |
LinkedInBot/1.0 (+http://www.linkedin.co... |
Fetches pages to render Open Graph previews on Lin... | Test |
 Amazonbot (default) |
Amazonbot/1.0 (+https://developer.amazon... |
Amazon's crawler used for product and content disc... | Test |
 Applebot (default) |
Mozilla/5.0 (compatible; Applebot/0.1; +... |
Apple's crawler used for Siri and Spotlight sugges... | Test |
 Bytespider (default) |
Bytespider; (+http://www.bytedance.com/s... |
ByteDance's web crawler for AI model training and ... | Test |
|
DuckDuckBot (default) |
DuckDuckBot/1.0; (+http://duckduckgo.com... |
DuckDuckGo's primary indexing bot user agent.... | Test |
|
CohereAI (default) |
CohereAI/1.0... |
Cohere's primary crawler user agent in their AI se... | Test |
|
AI2Bot (default) |
AI2Bot/1.0 (+http://www.allenai.org)... |
Allen Institute's AI2Bot crawler, used for researc... | Test |
 CCBot (default) |
CCBot/2.0 (+https://commoncrawl.org/faq/... |
Open web-scale archive crawler used to build publi... | Test |
 Diffbot (default) |
Diffbot/1.0 (+http://www.diffbot.com)... |
Diffbot's primary crawler user agent, used for web... | Test |
 Omgili (default) |
omgili... |
Aggregated discussion indexer bot.... | Test |
|
TimpiBot (default) |
TimpiBot/1.0... |
Independent web indexer bot.... | Test |
|
YouBot (default) |
YouBot/1.0 (+https://you.com/bot)... |
You.com's bot used for search and preview generati... | Test |
|
Bravebot (default) |
Bravebot/1.0... |
Brave's primary crawler user agent for their searc... | Test |
|
|
Mozilla/5.0 (compatible; YandexBot/3.0; ... |
Yandex's primary indexing bot user agent.... | Test |
More Free SEO Tools
Continue auditing your site with these companion tools.
Related Guides
Dive deeper with these step-by-step guides and tutorials.
Crawler Simulator FAQ
Make your site crawler-friendly
JavaScript sites often have indexing issues. Encited pre-renders your SPA into crawler-friendly HTML so every bot can read your content.
