Fake Googlebot in your logs: verify before you count

Across seven sites whose server logs we analyse, 35,714 requests introduced themselves as Googlebot. 20,621 of them came from addresses Google does not own.

On its own, that number means very little. Site by site, the share of fake Googlebot runs from 8.8% to 79.2%. The total is pulled up by one heavily targeted site; the median is 19.8%.

Even the median is enough to corrupt everything people usually read from access logs: crawl budget, the pages Google visits, the errors it hits, how much attention AI crawlers pay to the site. This piece covers how to separate real crawlers from impostors, what verification cannot settle, and why two serious studies can report 5% and 90% fake traffic without either of them being wrong.

The data

Logs from seven sites were imported between 10 October and 6 November 2026 and re-read on 8 November. They hold 550,109 crawler requests. At import, each one was checked against the IP ranges published by the operator whose name it carries.

AI crawlers are spoofed even more. Across the eleven AI crawlers whose operators publish their addresses, 19,534 of 37,532 requests were fake, or 52.0%. Three names produced nothing but fakes over the whole period: Claude-SearchBot (1,551 requests), Perplexity-User (1,452) and MistralAI-User (1,268). Not one genuine visit.

Some names, on the other hand, are almost never borrowed. AdsBot-Google verified at 100%, Bingbot at 96.7%. Impostors pick the names that open doors.

A user agent is a claim

The user agent is a text field the client fills in however it likes. Writing Googlebot/2.1 into a request takes one line of code, and plenty of sites treat that name better than any other: no blocking, no rate limit, the full page served.

What the fake crawlers ask for says a lot. Out of the 43,771 spoofed requests in the data:

• 7,321 target a .env configuration file in one form or another, and others ask for server.key, SSH keys, /proc/self/environ or cloud credentials. That is credential theft wearing a search engine's name.
• 20,216 get a real page back, with a 200. That is content scraping under a name nobody dares to block.
• 9,230 hit a 404, because they probe for paths the site never had.

Genuine crawlers get a page 86.5% of the time. The two profiles have nothing in common.

Three states, not two

A request carrying a known crawler name can get one of three verdicts, and the third one is what matters.

• Verified: the address belongs to the operator the request names.
• Spoofed: the operator publishes its addresses, and this one is not among them.
• Unverifiable: the operator publishes nothing, or our copy of its list is too old to rely on.

In our data, 72.6% of crawler requests land in the third state: uptime monitors, SEO tools, many bots that only send a generic name, and every operator that publishes no address list.

The temptation is to count those requests as genuine, since nobody can prove they are fake. That is the mirror image of the mistake being fixed. An unverifiable bot added to your AI crawler total inflates an "AI visibility" figure that was never measured. The only honest report keeps the three columns apart and shows the denominator of each.

The opposite rule matters just as much: a request is never declared spoofed because it could not be judged. Wrongly accusing a real Googlebot is how sites end up blocking it.

Checking by hand

Operators that publish their addresses do it as a JSON file at a stable URL. For Google, the crawler list lives on developers.google.com, and four more files cover its special-purpose crawlers and the fetchers that act on a user's request. Bing, Apple, OpenAI, Perplexity, Anthropic, Mistral, DuckDuckGo and Common Crawl do the same. Meta publishes no list: the only source is the routing announcement for its network, AS 32934.

The check fits in a few lines. This script reads Google's list and judges every address it is given:

import ipaddress, json, sys, urllib.request

LIST_URL = "https://developers.google.com/static/search/apis/ipranges/googlebot.json"

with urllib.request.urlopen(LIST_URL) as response:
    data = json.load(response)

ranges = [
    ipaddress.ip_network(p.get("ipv4Prefix") or p.get("ipv6Prefix"))
    for p in data["prefixes"]
]

for line in sys.stdin:
    ip = ipaddress.ip_address(line.strip())
    verdict = "verified" if any(ip in r for r in ranges) else "spoofed"
    print(ip, verdict)

Pull the addresses of every request that claims to be Googlebot out of your logs, pipe them in, and count. On 8 November 2026 it returned this:

$ printf "66.249.66.1\n34.141.20.10\n" | python3 verify.py
66.249.66.1 verified
34.141.20.10 spoofed

The script is deliberately minimal. It knows one operator and one list, and it cannot say "unverifiable". A real check loads every list an operator publishes and judges by operator, not by crawler: a genuine OpenAI fetcher carrying a sibling's name is still an OpenAI crawler, while a Google address under an OpenAI name is a spoof.

The reverse DNS trap

The method people copy most often is reverse DNS: ask for the hostname behind the address, check that it ends in googlebot.com or google.com, then confirm in the forward direction.

It works on one condition: that no other suffix is accepted. Yet the second address in the script above has a perfectly reassuring name:

$ host 34.141.20.10
10.20.141.34.in-addr.arpa domain name pointer 10.20.141.34.bc.googleusercontent.com.

That is a machine anyone can rent on Google Cloud. A check that accepts any hostname containing "google", or accepts googleusercontent.com without asking which crawler is involved, lets through exactly the best-equipped impostors. Published ranges do not have that weakness: a rented machine never shows up in Googlebot's list.

Reverse DNS is still the right tool for operators that only document that method, such as Amazon or Yandex. There it always comes with the forward confirmation, never on its own.

Why 5% and 90% are both true

Public studies of fake crawlers report numbers that look incompatible. One study of aggregated traffic across large networks finds a few percent of AI crawler requests spoofed. Single-site readings on small sites find more than 80%.

Our own table explains it: 57.7% in total, 19.8% at the median. Site 7 alone receives 17,907 fake requests, more than the other six combined. It dominates any total and counts as one row in seven for a median. The volume of impostors does not track a site's real traffic; it tracks how interesting the site is to attack, and across our seven sites the same measure varies by a factor of nine.

Two takeaways. A fake-bot percentage means nothing without its aggregation method. And a market figure says nothing about your site; only your own logs can.

A stale list manufactures culprits

Address lists change. While preparing this data, we found two official lists our configuration was not reading yet, one from Google and one from OpenAI, and added them. An operator adding a range plus a tool that never re-reads its copy produce a false spoof together: a real crawler on a new address, flagged as an impostor.

Two rules prevent it. Re-read every list daily. And when a copy could not be refreshed for several days, return "unverifiable" rather than "spoofed" for every crawler of that operator. A list that is stale on our side must never get a crawler accused.

What it costs

None of this is expensive to compute. The cost is elsewhere.

• Keeping the lists current. Eighteen lists from ten operators, re-read every day, each with a freshness check. A list that disappears or changes format must fail the refresh, not wipe the existing copy.
• Living with the gap. Close to three quarters of crawler requests stay unverifiable. You can report them separately; you cannot call them genuine.
• Keeping fakes visible. Taking them out of crawler figures does not mean deleting them: they stay in the request total, and their count gets reported. A site hit by 18,000 fake Googlebot requests needs to know.

What an access log cannot prove

Log analysis has limits, and they are worth knowing.

Some operators now sign their requests with a cryptographic signature carried in HTTP headers. A standard access log does not contain those headers: the proof exists, just not in the file.

An agent that sends an ordinary browser user agent is not a crawler as far as the log is concerned. No verification will bring it back.

And behind a CDN or a load balancer, the address written to the log may belong to the intermediary rather than the client. Before verifying anything, make sure your logs carry the real origin address: our export guides show how.

Next

For the full method, see our guide to server log analysis for SEO. To decide which AI crawlers to let in, see robots.txt for AI crawlers. For the engineering side of this check, run over 39 million requests, read measuring the agentic web without trusting the user agent. And if your real crawl budget turns out smaller than you thought, make sure your important pages are actually linked: an orphan page does not show up in its link count.

Our log file analysis judges every crawler request at import against eighteen lists refreshed daily. Fakes leave every crawler figure, crawl budget included, and their count is reported separately.