CCBot Simulator & Tester

How to use the CCBot simulator

1. Step 1

   Enter a URL

   Paste the full URL of any page on your website, such as a product page, a blog post, or the home page.

2. Step 2

   Select the CCBot user agent

   The tester sends the request with the CCBot user-agent string, so your server answers the way it answers a real CCBot visit.

3. Step 3

   Run the crawl

   The simulator requests the page as CCBot and follows redirects, the same way the crawler does.

4. Step 4

   Inspect the HTML

   Check the status code, redirects, response headers, raw HTML, word count, and Open Graph tags CCBot gets. If your main content is missing from the HTML, CCBot cannot read it.

What is CCBot?

CCBot is the web crawler for Common Crawl, a nonprofit that maintains a free, open archive of web crawl data. Many AI companies, researchers, and organizations use Common Crawl data for training AI models, research, and analysis. Major AI models including GPT and Claude have used Common Crawl data.

Control how CCBot crawls your website using robots.txt directives. Add these rules to your robots.txt file at the root of your domain, then run the simulator again on the same URL to confirm the change.

# Allow Common Crawl
User-agent: CCBot
Allow: /

# Block Common Crawl
User-agent: CCBot
Disallow: /

CCBot simulator FAQ

Related Crawler Simulators

ChatGPT Fetch

Simulator

Anthropic

Simulator

AI2Bot

Simulator

CohereAI

Simulator