Encited logo
PricingTalk to sales
All Crawler Simulators
Diffbot
Crawler simulator

Diffbot Crawler Simulator & Tester

Last updated

Simulate a Diffbot visit to any URL on your website. Crawl the page and inspect the HTML Diffbot extracts structured data from.

SIMULATE

Simulate a Diffbot visit

Enter any URL on your website to crawl it as Diffbot and inspect the HTML it gets.

How it works

How to use the Diffbot simulator

  1. Step 1

    Enter a URL

    Paste the full URL of any page on your website, such as a product page, a blog post, or the home page.

  2. Step 2

    Select the Diffbot user agent

    The tester sends the request with the Diffbot user-agent string, so your server answers the way it answers a real Diffbot visit.

  3. Step 3

    Run the crawl

    The simulator requests the page as Diffbot and follows redirects, the same way the crawler does.

  4. Step 4

    Inspect the HTML

    Check the status code, redirects, response headers, raw HTML, word count, and Open Graph tags Diffbot gets. If your main content is missing from the HTML, Diffbot cannot read it.

Overview

What is Diffbot?

Diffbot is a web scraping and data extraction company that uses AI to understand and structure web content. Their crawler automatically extracts articles, products, events, and other structured data from websites. Many businesses use Diffbot for competitive intelligence, content aggregation, and data services.

Why allow it

Why allow Diffbot?

Be included in Diffbot's knowledge graph
Support content aggregation services
Enable structured data extraction
Power various business intelligence applications
Robots.txt

Diffbot robots.txt configuration

Control how Diffbot crawls your website using robots.txt directives. Add these rules to your robots.txt file at the root of your domain, then run the simulator again on the same URL to confirm the change.

Allow Diffbot

# Allow Diffbot
User-agent: Diffbot
Allow: /

Block Diffbot

# Block Diffbot
User-agent: Diffbot
Disallow: /

User-Agent String: Diffbot/1.0 (+http://www.diffbot.com)

Diffbot simulator FAQ

Enter any URL from your website in the tester above and run the crawl. The simulator requests the page with the Diffbot user-agent string and shows the status code, response headers, and raw HTML it gets. Inspect the HTML to confirm your main content, title, and meta tags are there before any JavaScript runs.

Diffbot extracts structured data from web pages (articles, products, events) and provides this data via APIs and their Knowledge Graph product. Customers use it for market research, content aggregation, and competitive analysis.

No, Diffbot is a data extraction and structuring service. They don't operate a consumer search engine. Blocking them won't affect your search engine visibility.

Consider whether you want your content used for data extraction and aggregation services. If you have proprietary data or prefer not to be scraped for third-party services, blocking is reasonable.

Yes, Diffbot uses AI-powered rendering to understand JavaScript-heavy pages. It's one of the more sophisticated web scrapers in terms of content understanding.

Common uses include: price monitoring, news aggregation, lead generation, market research, content curation, and competitive intelligence. Various B2B applications rely on Diffbot's extraction.
Google
Gemini
ChatGPT
Claude
Grok
Meta
Perplexity

Make your site crawlable

JavaScript sites often have indexing issues. Encited pre-renders your SPA into crawler-friendly HTML so Diffbot and other bots can read your content.

All crawlers
Avatar
How can we help?
Get instant answers to your questions or leave a message for an engineer will reach out
Ask AI about Encited
See our docs
Contact support
Leave a message
We'll get back to you soon
Avatar
Ask AI about Encited
Product questions, troubleshooting and
Thinking
Preview
Drop an image to attach
Powered by ReplyMaven