Did Googlebot crawl my new page? Checking Google, Bing and AI crawlers

A new page passes through separate stages. A crawler fetches it, a search engine decides whether to store it, and a ranking or answer system decides whether to show it. Each stage has its own evidence.

Question Evidence Where to look
Did a crawler fetch the page? The request itself Search Console, server logs, CDN tools, a script on the page
Is the page in the search index? Index status Search Console URL Inspection, Bing Webmaster Tools
Does an AI answer mention or cite the page? Model output for chosen prompts Tools that query the model providers

Tools in the third row ask assistants such as ChatGPT, Gemini, Claude, Copilot and Perplexity a set of prompts and record the answers. A fetch shows that the first stage happened. It does not show the other two, so “crawled” and “indexed” name different facts.

Google: the URL Inspection tool

Search Console’s URL Inspection tool reports what Google holds for one URL on a verified property. Enter the URL in the search bar at the top. The report shows:

  • whether the URL is on Google, with a reason when it is not;
  • the time of the last crawl;
  • whether a smartphone or desktop crawler made it;
  • whether robots.txt allowed the fetch;
  • the page fetch result.

The default view is the version Google last indexed. It is a snapshot of the last crawl, so it can trail the live page by days. For a URL that has never been crawled, the report says so.

Two actions sit in the report. Test live URL fetches the page now and shows how Google sees it. Request indexing asks Google to crawl the URL, and the crawl can take from a few days to weeks. Live tests arrive at the server with the Google-InspectionTool user agent, which makes them visible in logs and in a page script.

For many URLs, the URL Inspection API returns the same fields programmatically, with limits of 2,000 URLs a day and 600 a minute.

Google: the Crawl Stats report

Under Settings in Search Console, Crawl stats shows daily totals of Google’s crawl requests, broken down by response, file type, purpose and Googlebot type. The report is available for properties verified at domain level, and a URL-prefix property for a subdomain does not show it. It reports aggregates with sample URLs, so it shows whether crawling happens and whether responses are errors. The URL Inspection tool answers the question for one page.

Server logs

Logs show every request. Filter for the page and the Googlebot token:

zcat -f access.log* | grep 'Googlebot' | grep '/blog/my-new-post'

A user agent string can be copied by any client. Verify the address by reverse DNS: the host name must end in googlebot.com, google.com or googleusercontent.com, and a forward lookup on that name must return the same address. Google also publishes its crawler address ranges as JSON files.

Bing and IndexNow

Bing Webmaster Tools reports crawl and indexing signals for a site. Bing and several other engines also accept submissions through IndexNow, an open protocol co-created by Microsoft and Yandex. A submission to one participating engine is shared with the rest. The participants include Bing, Yandex, Naver, Seznam and Yep. Google does not participate and relies on its own crawling, so sitemaps and Search Console remain the route for Google.

IndexNow needs an API key, a key file in the root of the site, and an HTTP notification that lists the new or changed URLs.

AI crawlers

The same logs show OAI-SearchBot, PerplexityBot and the other agents described in which AI crawlers visit your site, and how to tell. Allow the search crawlers in robots.txt, allow up to about 24 hours after a change, and count requests for the new URL by token.

With a script on the page

A script records a visit when the agent runs it. Googlebot executes scripts when it renders a page, and Clientlog records sessions with the Googlebot user agent. On one small site, 25 days of data included 93 Googlebot sessions and 13 Google-InspectionTool sessions, which are the live tests run from Search Console.

A workflow for a new page:

  1. Publish the page, add it to the sitemap, and link it from one internal page.
  2. In Search Console, run Test live URL, then Request indexing.
  3. Within minutes the live test appears as a Google-InspectionTool session on the page. This confirms that the page renders and that the script ran.
  4. Googlebot arrives afterwards. In Clientlog the session carries the class “search crawler”.

To list Googlebot sessions and their first page from the command line:

clientlog --format json sessions list --limit 1000 \
  | jq -r '.[]
      | (.ua | if type == "string" then fromjson else . end) as $ua
      | (.action_map | if type == "string" then fromjson else . end) as $acts
      | select($ua.browser.family == "Googlebot")
      | [(.start_time / 1000 | floor | todate), $acts[0][0]]
      | @tsv'

Action names carry the page: page.view.blog.my-new-post is the page /blog/my-new-post.

The script sees sessions that run it. Agents that fetch HTML only appear in logs, so the two sources suit use together.

A checklist for a new page

  1. The page returns status 200, robots.txt allows it, and no noindex directive applies.
  2. The page appears in the sitemap and has at least one internal link.
  3. Inspect the URL in Search Console, run the live test and request indexing.
  4. Watch logs or the script for Google-InspectionTool, then Googlebot.
  5. Inspect the URL again after the crawl and read the index status.
  6. For Bing and its partners, send an IndexNow notification.
  7. For AI search, confirm robots.txt allows the search crawlers and count their requests.

Sources