Measuring the agentic web without trusting the user agent

Python 3, standard library only, no dependencies.

A user agent string is a claim. Anyone can send Mozilla/5.0 (compatible; GPTBot/1.1), and nothing about receiving it tells you an AI company was on the other end. Every log analysis tool we know of, including the one we ship, counts those strings and calls the result bot traffic.

We wanted to know what that number is worth. So we built the verification, ran it over three production access logs, and counted two things separately: what machine readable paths agents actually request, and which of the agents making those requests are who they say they are.

39,319,497 requests, zero unreadable lines. Three sites, three windows, one instrument. A SaaS product site in launch, a high volume national news site, and a national e-commerce catalogue. Logs provided for audit; no site is named here, and no address appears in any output.

The sentence we could not write

The last engineering note published here ended on an admission. It described the log parser that ingests those files at constant memory, and it closed with a list of things we had not done. The first item was this:

Reverse DNS verification is still missing. We classify crawlers by user agent, which is trivially spoofable. Until then, our bot attribution is a declared identity, not a verified one, and the interface should say so.

That sentence has been sitting in the log pipeline article since it went up. This is the work that closes it, done outside the product first, because measuring something is cheaper than shipping it and because the answer determines whether shipping it is worth anything.

The instrument is a single Python file with no dependencies outside the standard library. It reads combined format logs, gzipped or not, and emits JSON. It exists separately from the product for a reason that matters, and that reason is the last section of this article.

Reverse DNS alone is not a proof

The verification most people describe is a reverse lookup: take the address, ask for its PTR record, check the hostname ends in .googlebot.com. That check is worthless on its own. Whoever controls the in-addr.arpa zone for an address block writes whatever they like in it, including crawl-1-2-3-4.googlebot.com.

What closes the loop is the forward lookup. Resolve the hostname you just received and require that it resolves back to the address you started from. The reverse zone and the forward zone are controlled by different parties, and an attacker needs both.

# scripts/agentic-demand/analyze.py
def rdns_ok(self, operator, ip):
    suffixes = self.rules.get(operator, {}).get("rdns_suffixes", [])
    if not suffixes or not self.use_rdns:
        return None
    host = self._rdns(ip)
    if host is None:
        return None                       # nobody answered
    host = host.rstrip(".").lower()
    if not any(host.endswith(sfx) for sfx in suffixes):
        return False                      # a fact: not theirs
    addrs = self._forward(host, ip)
    if addrs is None:
        return None
    target = ipaddress.ip_address(ip)
    return any(ipaddress.ip_address(a) == target for a in addrs)

Three return values from one function, and the difference between them is the whole subject. True and False are findings. None means the check did not run, and it must never be read as either.

The second mechanism is simpler and better where it exists. Seven operators publish the address ranges their crawlers use, as JSON, at a documented URL. Membership in a published range is a proof, and it costs no network call at analysis time once the file is on disk. We hold thirteen such files, 2,975 prefixes, covering OpenAI, Anthropic, Google, Bing, Apple, Perplexity and DuckDuckGo.

Amazon, Yandex and Baidu publish a reverse DNS convention instead. Meta, Common Crawl and ByteDance publish neither.

That asymmetry has a consequence worth stating before any number below is read. An operator verified against a published range is held to a different test from one verified by reverse DNS, and the reverse DNS test is stricter in a specific way: an address with no PTR record fails it, even if the operator really does own the address and simply never published the record. Refusal rates are therefore comparable within a verification method and only loosely across them.

Three states, and a fourth that is not one

That last group is where most tools quietly lie. Faced with a request claiming meta-externalagent and no way to check it, the honest output is not "unverified" in the sense of suspicious, and it is certainly not "verified" by default. It is a declaration that verification is impossible, and it stays visibly different from the two verdicts we can pronounce.

# scripts/agentic-demand/analyze.py
def verdict(self, operator, ip):
    if not operator:
        return UNVERIFIABLE
    rule = self.rules.get(operator)
    if rule is None or rule.get("means") == "none":
        return UNVERIFIABLE

    checked = False
    if self.ranges_usable(operator):
        checked = True
        if self.in_ranges(operator, ip):
            return VERIFIED

    r = self.rdns_ok(operator, ip)
    if r is True:
        return VERIFIED
    if r is False:
        checked = True

    return SPOOFED if checked else UNAVAILABLE

Four outcomes, and only three of them describe the world.

VERIFIED, SPOOFED and UNVERIFIABLE are findings. UNAVAILABLE is the admission that the check did not execute here: the range file is missing from disk, or the resolver stayed silent, or reverse lookups were switched off. It is published next to the other three and folded into none of them. Collapsing it into UNVERIFIABLE turns a DNS outage into an operator policy. Collapsing it into SPOOFED accuses a legitimate crawler of impersonation because our network was down.

The checked flag is the entire mechanism. An accusation is only pronounceable when a means of verification existed and ran and refused the address.

Which operator publishes what is data, not code:

# scripts/agentic-demand/operators.json
"anthropic":   {"means": "ranges", "range_files": ["anthropic-bots"]},
"amazon":      {"means": "rdns",   "rdns_suffixes": [".crawl.amazonbot.amazon"]},
"google":      {"means": "both",   "range_files": [...],
                "rdns_suffixes": [".googlebot.com", ".google.com"]},
"meta":        {"means": "none",   "note": "no published range file or rDNS
                convention was located on 23 August 2026"},
"commoncrawl": {"means": "none",   "note": "Common Crawl publishes no means
                of verification"}

"means": "none" is a load bearing value. It is what makes an accusation against Meta or Common Crawl impossible to pronounce, no matter how the traffic looks.

A reference that returns 200 and is not there

Fetching those published files looked like the boring part. It was where the first real defect lived.

One of the operator endpoints returns HTTP 200 with an HTML page for a path that does not exist. We found it by probing an invented filename on each host as a negative control, which is a habit we have because a supplier sandbox once accepted every field name we invented.

$ curl -sL -o /dev/null -w "%{http_code}\n" https://<operator>/applebot.json
200
$ curl -sL -o /dev/null -w "%{http_code}\n" https://<operator>/nexistepas-xyz.json
200

A fetcher that trusts the status code writes that HTML to apple-applebot.json, the parser finds no prefixes, and Applebot silently becomes unverifiable forever. No error, no red build, just a check that stopped checking. So the fetcher validates the payload rather than the response:

# scripts/agentic-demand/analyze.py
try:
    doc = json.loads(body.decode("utf-8"))
except Exception:
    failures.append((name, url, f"non-JSON payload (status {code})"))
    continue
if not isinstance(doc, dict) or not isinstance(doc.get("prefixes"), list) or not doc["prefixes"]:
    failures.append((name, url, f"JSON without a non-empty prefixes array (status {code})"))
    continue

The same reasoning applies to anything downloaded and then relied upon. The proof is never that the command succeeded. The proof is that the effect happened.

The timeout that did not exist

The second defect cost more, and it is the one worth carrying away.

The largest of the three corpora is 36.5 million lines. Parsing consumed five and a half minutes of CPU and finished. Then the process sat at 0.1 percent CPU for fifteen minutes, wrote nothing, logged nothing, and raised nothing.

$ ps -p 36486 -o etime=,time=,%cpu=
  21:29   5:34.87   0.1

Five and a half minutes of CPU consumed, twenty one minutes elapsed. It was blocked in DNS resolution, and the option meant to bound that was --rdns-timeout, documented, defaulted to three seconds, and doing nothing at all. socket.setdefaulttimeout() applies to socket objects. socket.gethostbyaddr() goes through the system resolver and ignores it.

A timeout that lies is worse than no timeout, because it makes you believe the worst case is bounded. We removed it and wrote a DNS client that speaks UDP directly, where settimeout() is real:

# scripts/agentic-demand/analyze.py
DNS_NO_ANSWER = object()   # nobody answered: this is not a fact

def dns_query(qname, qtype, servers, timeout, attempts=2):
    ...
    sock.settimeout(timeout)
    sock.sendto(packet, (server, 53))
    ...
    return DNS_NO_ANSWER

One hundred and forty seven lines for query encoding, compression pointer following with a loop guard, and answer parsing for PTR, A and AAAA. That is more code than calling the standard library, and it buys the one property the standard library would not give: a bounded worst case, and a resolver silence that is distinguishable from a resolver answer.

That distinction is not cosmetic. An address with no PTR record is a fact: an operator whose only verification means is reverse DNS cannot be behind an address that has none. An address whose lookup timed out proves nothing. The two are one return value apart:

# scripts/agentic-demand/analyze.py
ans = dns_query(qname, "PTR", self.servers, self.rdns_timeout)
if ans is DNS_NO_ANSWER:
    host = None      # a failure, verdict becomes UNAVAILABLE
elif not ans:
    host = ""        # a fact, verdict can become SPOOFED
else:
    host = ans[0]

The fix ships with the witness that would have caught it, because a guard nobody has watched run is not a guard:

# scripts/agentic-demand/analyze.py, selftest
t0 = time.monotonic()
r = dns_query("203.65.249.66.in-addr.arpa", "PTR", ["192.0.2.1"], 0.8, attempts=2)
ok = r is DNS_NO_ANSWER and time.monotonic() - t0 < 4.0

v_mute = Verifier(cfg, args.ranges, use_rdns=True, rdns_timeout=0.8,
                  nameservers=["192.0.2.1"])
ok = v_mute.verdict("amazon", "203.0.113.7") == UNAVAILABLE

ok = v._rdns("203.0.113.7") == ""

192.0.2.1 is a documentation address that answers nothing, and querying it returns in 1.6 seconds against a 0.8 second budget and two attempts. The second assertion is the one that matters most: with a silent resolver and an operator whose only verification means is reverse DNS, the verdict has to come back UNAVAILABLE and never SPOOFED. The third holds the opposite line, that an address with no PTR record is a finding rather than a failure.

Nineteen checks run in total. Those three did not exist before the stall, and the first is the only reason we would notice if the bound ever broke again.

What the log is actually asking for

Now the measurement. Two things get counted, and they never merge: a 404 on a machine readable path is demand without supply, and a 200 is consumption.

/llms.txt was requested on all three sites, across the whole window, by a widening set of clients.

The days column is there because the three observation windows have different lengths, and a per day rate does not fix that. A path requested once in fourteen days and a path requested every day can show a similar rate. The share of days separates a recurring request from a single pass, and it compares across windows. On this measure /llms.txt is recurring on all three sites, from eighteen distinct clients on the busiest one.

Two of the three answer 404 or a redirect. The one that answers 200 is the site you are reading, which serves the file. That is the entire supply side of the agentic web on this sample.

The newer vocabulary tells a very different story, and reading it carelessly is how you would get this article wrong. On the e-commerce corpus, fifteen distinct paths carrying agent vocabulary were requested over fifteen days:

2026-08-03   /.well-known/mcp.json                           1 req   404
2026-08-03   /.well-known/mcp-server.json                    1 req   404
2026-08-03   /.well-known/mcp/server-card.json               1 req   404
2026-08-03   /.well-known/webmcp                             1 req   404
2026-08-03   /.well-known/webmcp.json                        1 req   404
2026-08-03   /.well-known/agent.json                         1 req   404
2026-08-03   /.well-known/agents.json                        1 req   404
2026-08-03   /.well-known/agent-card.json                    1 req   404
2026-08-03   /.well-known/agent-skills/index.json            1 req   404
2026-08-03   /.well-known/ai-catalog.json                    1 req   404
2026-08-03   /.well-known/acp.json                           1 req   404
2026-08-03   /.well-known/api-catalog                        1 req   404
2026-08-03   /.well-known/http-message-signatures-directory  1 req   404
2026-07-31   /.well-known/oauth-protected-resource/mcp       3 req   404, 302
2026-07-31   /.well-known/oauth-protected-resource/api/mcp   3 req   404, 302

Fifteen paths, two days, and across all of them exactly two distinct user agent strings:

AgentRadar-Research/1.0 (BCG Henderson Institute; agentic-web-research)
mcp-census/prm-spray/1.0

Both self declare as research. One walks the candidate list in a single pass on 3 August; the other probes two OAuth discovery paths three days earlier. Neither is an agent trying to use this catalogue. They are people measuring the agentic web.

Which is what we are doing. Our own audit crawler accounts for 46,693 requests on the news corpus, and it is counted and published in the same tables as everything else rather than filtered out. The honest reading of this block is that on a national e-commerce site over fifteen days, the entire observed demand for MCP and agent card discovery came from two censuses and from nobody else, and that a third census was in the building at the same time.

Publishing those fifteen rows as "MCP discovery traffic observed in the wild" would be true in the narrowest possible sense and worthless in every useful one. /llms.txt at eighteen distinct clients across eight of nine days is a different object from /.well-known/mcp.json at one client on one day, and an instrument that cannot separate them will report a movement that does not exist. That is why the distinct user agent count sits next to every request count in the output, and why it is the column to read first.

Who is actually asking

Now the part that changes how the first table reads.

Share of requests, per named agent, where the claimed identity was refused by the operator's own published reference:

Every figure carries its denominator because the rates are not comparable without one. A third of PerplexityBot requests on the news site is 850 requests. A third on the e-commerce site is 97. Those two numbers deserve very different amounts of attention, and a bare percentage hides which is which.

Two things hold across both sites, and only two. PerplexityBot sits near a third on each, 33.7 and 32.9 percent, which is the steadiest figure in the table. And Googlebot, Bingbot and ClaudeBot stay under 4 percent on both.

Everything else moves, and it moves a lot. Amazonbot goes from 18.7 percent to 0.4. ChatGPT-User goes from 5.4 to 38.3, in the opposite direction. One site on its own would have supported a tidy story about crawlers with twenty year old names being harder to impersonate than new ones. The second site does not support it, and that is the only reason this paragraph is not that story.

Amazonbot also carries the method caveat from earlier: its only published means is reverse DNS, so an address with no PTR record fails, and an operator verified against a range list is not being asked the same question.

One more count, which needed no address to be published anywhere:

A single machine cannot be at Google and at Anthropic. Each of those rows is one sender rotating its labels. On the smallest site, where a single such host is visible against a low background, the combination it cycled through was:

Baiduspider + Bingbot + CCBot + ChatGPT-User + ClaudeBot
+ GPTBot + Googlebot + OAI-SearchBot + PerplexityBot + YandexBot

Ten operator identities, one address, one window of under two days. Its reverse lookup resolves to a hosting provider. Real Googlebot on the same site verified cleanly at the same time, which is what makes the contrast a measurement rather than an impression.

Set against that, 3.6, 2.0 and 2.2 percent of all traffic on the three corpora comes from operators who publish no means of verification at all. For those requests, the user agent string is genuinely all anyone has, and no amount of engineering on our side changes it.

Why this cannot live in the product yet

The instrument is a standalone script, and that is not an accident of scheduling.

The product's log module stores no client address. Not hashed, not truncated, not in a column anybody forgot about. The parser reads the address because the common log format is positional and the fields after it will not line up otherwise, and then the line goes out of scope and takes it with it. That property is the reason a customer can hand over a raw access log at all.

Verification needs the address. So there is exactly one place it can happen: inside the parser, in memory, on the line that already holds it, with only the verdict surviving into storage. Not a later job over stored rows, because there are no stored rows to work from.

That constraint is stricter than it first looks, and it is why this ran outside the product first:

• The parser is synchronous and processes tens of thousands of lines per second. A DNS round trip per line would reduce that to the speed of the network, several orders of magnitude slower.
• So verification has to be deferred to the distinct addresses of an ingest rather than its lines, held in a map that never leaves the process, and resolved once per address.
• And a verdict is only worth storing next to a fact if the four outcome model comes with it. A boolean column called verified would recreate exactly the lie this whole exercise was built to avoid.

Knowing the shape of that work, and knowing what it finds, is worth more than having shipped a guess at it. The measurement cost a day. Shipping it into the ingest path is a real piece of work, and it now has a specification instead of an intention.

What it costs

• Thirteen files of somebody else's data, on our disk. Published address ranges go stale. We record the URL, the fetch date, a hash and the prefix count for each, and a missing file yields UNAVAILABLE rather than a wrong verdict. It still means a verification whose freshness is an operational responsibility.
• Two hundred lines of hand written DNS. We own compression pointer parsing now. The alternative was an unbounded worst case, so we take the trade, but it is a trade.
• The three states are contagious. Every surface that consumes a verdict has to carry four cases. Any of them that collapses to a boolean puts the lie back, at that surface only, silently.
• A verified crawler is still just a verified crawler. Nothing here says an agent does what its documentation claims once it has your page. Verification answers who connected, and only that.

What this measurement will not support

Four limits, stated because the numbers above are otherwise easy to over-read.

The three windows do not overlap, not even pairwise. They are three consecutive periods, not one moment seen from three places. Every cross site difference in this article mixes the effect of the site with the effect of the period, and nothing separates them. It is why volumes are never compared directly and why the share of days exists.

Two of the three corpora sit behind a cache. The logs come from the origin, so anything served from that cache is absent. Those volumes are floors, and the distance between the floor and the truth is not measured.

The instrument is in its own measurement. Our audit crawler appears in the logs it analyses, at 46,693 requests on one corpus and 132 on another. Those stay in the denominators, counted separately and published, because removing them would move every rate with no way to move it back.

Three sites are three measurements, not a sample. They were chosen because their logs were reachable. Nothing here generalises to a population of websites.

The full data note, with every table, the complete list of /.well-known/* paths observed, and the sections on what the data does not say, sits in the repository next to the script. The script runs on any combined format log, and its nineteen self checks run before any of its numbers should be believed.

Measurements: three production access logs analysed 23 August 2026, 39,319,497 requests total, windows of 1.77, 7.42 and 14.37 days between 21 July and 23 August 2026, 0 unparseable lines. Operator address references fetched 23 August 2026, 13 files, 2,199 prefixes. Reverse lookups performed with a stdlib UDP DNS client, 2.0 second timeout, two attempts per nameserver. Parsing throughput measured at 136,942 lines per second on a 200,000 line sample, Apple M-series laptop, Python 3.14.5.