Encited logo
Pricing
CRAWLER SIMULATOR

Web Crawler Simulator

Test how 20+ search engine and AI crawlers see your site. Verify robots.txt rules, check content visibility, and optimize for better indexing.

SIMULATE

Test any crawler

Select a crawler and enter your URL to see what the bot sees.

AI Assistants & LLMs

Crawlers used by AI chatbots and large language models to fetch web content for answers and training.

Search Engines

Traditional search engine crawlers that index your content for search results.

Social Media

Crawlers that fetch Open Graph data for link previews on social platforms.

Voice Assistants & Enterprise

Crawlers for voice search, smart speakers, and enterprise applications.

Data & Research

Crawlers used for web archiving, research datasets, and AI training corpora.

All Supported Crawler User Agents

Crawler User-Agent Purpose Simulate
Googlebot SmartphoneGooglebot Smartphone
 Mozilla/5.0 (Linux; Android 6.0.1; Nexus... Google Search's mobile-first indexing crawler as i... Test
Googlebot DesktopGooglebot Desktop
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... Google Search's desktop indexing crawler as it app... Test
Google Inspection Tool SmartphoneGoogle Inspection Tool Smartphone
 Mozilla/5.0 (Linux; Android 6.0.1; Nexus... Mobile user-agent used by Google Search Console UR... Test
Google Inspection Tool DesktopGoogle Inspection Tool Desktop
 Mozilla/5.0 (compatible; Google-Inspecti... Desktop user-agent used by Google Search Console U... Test
Googlebot ImageGooglebot Image
 Googlebot-Image/1.0... Google crawler for images, logos, favicons, Discov... Test
Googlebot VideoGooglebot Video
 Googlebot-Video/1.0... Google crawler for video-related Search features a... Test
GoogleOtherGoogleOther
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... Google's generic crawler used by product teams for... Test
Storebot-GoogleStorebot-Google
 Mozilla/5.0 (X11; Linux x86_64; Storebot... Google Shopping crawler used across Shopping surfa... Test
Google-CloudVertexBotGoogle-CloudVertexBot
 Google-CloudVertexBot/1.0... Google crawler that fetches sites on the owner's r... Test
OAI-SearchBotOAI-SearchBot
 Mozilla/5.0 (Macintosh; Intel Mac OS X 1... OpenAI crawler used to surface websites in ChatGPT... Test
GPTBotGPTBot
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... OpenAI crawler used for training and improving gen... Test
ChatGPT-UserChatGPT-User
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... OpenAI user-triggered fetcher used when ChatGPT or... Test
OAI-AdsBotOAI-AdsBot
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... OpenAI fetcher used to validate landing pages subm... Test
ClaudeBotClaudeBot
 ClaudeBot/1.0... Anthropic's autonomous crawler for discovering and... Test
Claude-SearchBotClaude-SearchBot
 Claude-SearchBot/1.0... Anthropic crawler associated with Claude search an... Test
Claude-UserClaude-User
 Claude-User/1.0... Anthropic user-triggered fetcher used when Claude ... Test
Claude-WebClaude-Web
 Claude-Web/1.0... Legacy Claude web fetcher identifier; no longer li... Test
anthropic-aianthropic-ai
 anthropic-ai... Legacy Anthropic crawler identifier seen in server... Test
PerplexityBotPerplexityBot
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... Perplexity crawler used to surface and link websit... Test
Perplexity-UserPerplexity-User
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... Perplexity user-triggered fetcher used to retrieve... Test
BingbotBingbot
 Mozilla/5.0 (compatible; bingbot/2.0; +h... Microsoft Bing's primary indexing crawler.... Test
BingPreviewBingPreview
 Mozilla/5.0 (Windows NT 10.0; Win64; x64... Microsoft Bing preview fetcher used for page previ... Test
BravebotBravebot
 Bravebot/1.0... Brave Search's primary indexing crawler.... Test
Brave SearchBrave Search
 Brave-Search/1.0... Brave Search fetcher/crawler identifier used by Br... Test
DuckDuckBotDuckDuckBot
 DuckDuckBot/1.0; (+http://duckduckgo.com... DuckDuckGo's primary indexing crawler.... Test
DuckAssistBotDuckAssistBot
 DuckAssistBot/1.0... DuckDuckGo fetcher associated with DuckAssist answ... Test
Amzn-SearchBotAmzn-SearchBot
 Mozilla/5.0 AppleWebKit/537.36 (KHTML, l... Amazon search crawler powering Alexa and Amazon se... Test
Meta-ExternalAgentMeta-ExternalAgent
 Meta-ExternalAgent/1.1... Meta crawler used for external link previews and A... Test
Meta-ExternalFetcherMeta-ExternalFetcher
 Meta-ExternalFetcher/1.1... Meta fetcher used to retrieve external page metada... Test
FacebookBotFacebookBot
 FacebookBot/1.0... Legacy Facebook crawler identifier; no longer list... Test
facebookexternalhitfacebookexternalhit
 facebookexternalhit/1.1 (+http://www.fac... Facebook/Instagram link preview crawler for Open G... Test
Meta-WebIndexerMeta-WebIndexer
 meta-webindexer/1.1... Meta crawler used to improve Meta AI search result... Test
ChatGPT FetchChatGPT Fetch (default)
 Mozilla/5.0 (compatible; ChatGPT-User/1.... User agent used by ChatGPT to fetch pages for AI a... Test
AnthropicAnthropic (default)
 ClaudeBot/1.0... Catch-all for Anthropic crawler user agents.... Test
GrokGrok (default)
 GrokBot/1.0... xAI publishes no official crawler user agents and ... Test
PerplexityBotPerplexityBot (default)
 PerplexityBot/1.0 (https://www.perplexit... Perplexity's crawler user agent used to answer que... Test
GooglebotGooglebot (default)
 Mozilla/5.0 (compatible; Googlebot/2.1; ... Primary provider bucket for Google's indexing, ins... Test
BingbotBingbot (default)
 Mozilla/5.0 (compatible; bingbot/2.0; +h... User agent used by Bing's indexing bot. Also power... Test
Facebook CrawlerFacebook Crawler (default)
 facebookexternalhit/1.1 (+http://www.fac... Fetches pages to render Open Graph previews on Fac... Test
LinkedInBotLinkedInBot (default)
 LinkedInBot/1.0 (+http://www.linkedin.co... Fetches pages to render Open Graph previews on Lin... Test
AmazonbotAmazonbot (default)
 Mozilla/5.0 (Macintosh; Intel Mac OS X 1... Amazon's crawler used for product and content disc... Test
ApplebotApplebot (default)
 Mozilla/5.0 (compatible; Applebot/0.1; +... Apple's crawler used for Siri and Spotlight sugges... Test
BytespiderBytespider (default)
 Bytespider; (+http://www.bytedance.com/s... ByteDance's web crawler for AI model training and ... Test
DuckDuckBotDuckDuckBot (default)
 DuckDuckBot/1.0; (+http://duckduckgo.com... DuckDuckGo's primary indexing bot user agent.... Test
CohereAICohereAI (default)
 CohereAI/1.0... Cohere's primary crawler user agent in their AI se... Test
AI2BotAI2Bot (default)
 AI2Bot/1.0 (+http://www.allenai.org)... Allen Institute's AI2Bot crawler, used for researc... Test
CCBotCCBot (default)
 CCBot/2.0 (+https://commoncrawl.org/faq/... Open web-scale archive crawler used to build publi... Test
DiffbotDiffbot (default)
 Diffbot/1.0 (+http://www.diffbot.com)... Diffbot's primary crawler user agent, used for web... Test
OmgiliOmgili (default)
 omgili... Aggregated discussion indexer bot.... Test
TimpiBotTimpiBot (default)
 TimpiBot/1.0... Independent web indexer bot.... Test
YouBotYouBot (default)
 YouBot/1.0 (+https://you.com/bot)... You.com's bot used for search and preview generati... Test
BravebotBravebot (default)
 Bravebot/1.0... Brave's primary crawler user agent for their searc... Test
YandexBotYandexBot (default)
 Mozilla/5.0 (compatible; YandexBot/3.0; ... Yandex's primary indexing bot user agent.... Test

Crawler Simulator FAQ

GET STARTED

Make your site crawler-friendly

JavaScript sites often have indexing issues. Encited pre-renders your SPA into crawler-friendly HTML so every bot can read your content.

Book a demo
Avatar
How can we help?
Get instant answers to your questions or leave a message for an engineer will reach out
Ask AI about Encited
See our docs
Contact support
Leave a message
We'll get back to you soon
Avatar
Ask AI about Encited
Team is also here to help
Thinking
Preview
Drop an image to attach
Powered by ReplyMaven