Who crawls your site and what they get back
"Is Google even looking at this site?" is normally answered with a guess. Search Console reports crawl stats days late and in aggregate, and raw server logs mean pulling files off a host and parsing them. Neither tells you what a bot received when it arrived.
Your crawl log records every visit from a known crawler: which bot, which page, when, the status it got, how long it waited, and whether it was served a snapshot or passed through to your origin.
The chart starts on AI agents#
Googlebot and Bingbot shape what ranks. GPTBot, ClaudeBot, PerplexityBot and Google's AI crawlers shape what gets quoted in answers.
The activity chart opens filtered to AI agents. Switching to All crawlers widens the same chart to include search engines and SEO tools alongside them.
Questions it answers#
| You want to know | Where to look |
|---|---|
| Whether Googlebot visits, and how often | All crawlers, on the activity chart |
| Whether GPTBot, ClaudeBot and Perplexity read your content | The crawler cards and the default chart view |
| Whether yesterday's new page has been crawled | Crawled URLs, filtered to that path |
| Whether a traffic dip is a crawling problem | The activity chart over the last 30 days |
| What a bot actually received | Crawl Logs, in the status and outcome columns |
The outcome column is the part raw server logs can't give you. A 200 tells you the request succeeded. The outcome tells you whether the bot got a pre-rendered snapshot or your origin's own response.
Before you start#
Pre-rendering needs to be connected, and bots need a few days to arrive. A site connected this morning has an empty log, which is expected rather than a fault.