Which AI crawlers visit your site, and how to tell

Requests from AI products reach a site in three forms. Each form has its own user agent token, its own purpose, and its own relationship to robots.txt. Counting the forms separately is the first step in knowing what reads your content.

The three kinds

Training crawlers collect content that may be used to train models. GPTBot (OpenAI) and ClaudeBot (Anthropic) are examples. OpenAI states that disallowing GPTBot indicates a site’s content should not be used in training its generative AI foundation models.

Search crawlers build the index behind an assistant’s search feature. OAI-SearchBot surfaces sites in ChatGPT’s search features. PerplexityBot builds Perplexity’s index. Claude-SearchBot navigates the web to improve search result quality for Anthropic’s users.

User-triggered fetchers retrieve a page because a person asked an assistant about it. ChatGPT-User, Claude-User and Perplexity-User belong here. OpenAI notes that robots.txt rules may not apply to ChatGPT-User because a user initiates the action, and Perplexity documents that Perplexity-User generally ignores robots.txt rules for the same reason.

Google follows the same pattern. Googlebot is the search crawler. Google-Extended is a robots.txt token with no crawler or user agent behind it, so no request ever arrives under that name. Search Console’s testing tools fetch pages with Google-InspectionTool, which Google groups with its user-triggered fetchers.

Which operators publish ways to verify them

  • OpenAI publishes IP ranges for GPTBot, OAI-SearchBot and ChatGPT-User as JSON files.
  • Perplexity publishes ranges for PerplexityBot and Perplexity-User.
  • Google publishes ranges for its crawlers and fetchers, and documents a reverse DNS check.
  • Anthropic’s support documentation states that it does not publish IP ranges because it uses service provider addresses, and that this may change. Check the current documentation before relying on either position.

Four ways to count them

1. Web server or CDN logs

Any log format that records the user agent works. This counts requests per token in a set of logs:

zcat -f access.log* \
  | grep -oiE 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Bytespider|CCBot|meta-externalagent|Amazonbot|Applebot' \
  | tr 'A-Z' 'a-z' | sort | uniq -c | sort -rn

Logs record every request that reaches the server, including requests from agents that fetch HTML and run no scripts.

2. CDN dashboards

Cloudflare’s AI Crawl Control lists which AI services access a domain, with per-crawler allow and block actions and robots.txt compliance tracking. It is available on all plans, and free plans show a 24-hour metrics window. It sees traffic that passes through Cloudflare.

3. Agent analytics services

Services such as Known Agents (formerly Dark Visitors) connect through WordPress, a CDN or an API. They report crawler, scraper and AI agent activity, together with referrals from language model products.

4. A script on the page

A script records the visitors that run it. Clientlog’s script sends each page view, exit, scroll milestone and click as an event, groups events into sessions, and classifies each session by user agent. In the dashboard each session receives a class: search crawler, AI training, AI search index, AI user fetch, link preview, other bot, or browser.

On one small site, 997 sessions over 25 days divided like this:

Class Sessions
Browser 493
AI training 237
Search crawler 201
Other bot 62
AI search index 4

Bytespider accounted for 235 of the AI training sessions. The search crawler sessions were Googlebot (93), Applebot (67), Sogou web spider (25), Google-InspectionTool (13) and bingbot (3). OAI-SearchBot made up the four AI search index sessions.

Logs and a script show overlapping sets. Agents that request HTML without running scripts appear in logs only. Browsers that block scripts appear in logs only. Sessions with a full action sequence, referrer and location appear in the script data. Using both gives the full picture.

Verifying who sent a request

A user agent string is text that any client can send. Treat it as a label, and verify the requests that matter.

  • Google. Run a reverse DNS lookup on the IP address from the log. The host name must end in googlebot.com, google.com or googleusercontent.com. Run a forward lookup on that host name and confirm it returns the original address.
host 66.249.66.1
host crawl-66-249-66-1.googlebot.com
  • OpenAI and Perplexity. Match the source address against the operator’s published ranges.
  • Anthropic. Follow the guidance in the operator’s current documentation.

In the script data, the network field (geo.org) gives a coarse check. A Googlebot user agent that arrives from a network Google does not own warrants inspection.

What a visit shows

A visit shows that an agent requested a URL. It does not show that the page was stored, ranked or cited. OpenAI recommends allowing OAI-SearchBot in robots.txt and permitting its published IP ranges for sites that want to appear in ChatGPT search results. Changes to robots.txt rules can take up to about 24 hours to take effect for OpenAI’s search and for Perplexity.

A crawler can be absent from logs for reasons on the site’s side. Bot protection features at a CDN or firewall can challenge or block automated requests before they reach the server. If an expected crawler never appears, check those settings.

A checklist

  1. Fetch the live robots.txt and read the rules for each token above.
  2. Count requests per token in logs for a week.
  3. Verify the counts that matter, by IP range or reverse DNS.
  4. Check firewall and bot protection settings for rules that challenge crawlers.
  5. Watch search crawlers and user-triggered fetchers separately.

To check whether a specific new page was crawled, read how to check whether Googlebot crawled a new page.

Sources