What your servers know and Search Console will not tell you
Search Console shows you a sample, with a delay. Your servers’ logs keep a trace of every request actually received. We parse them, separate bots from visitors, and cross the result with the rest of the audit, without keeping a single IP address.
Access opens in waves: we email you when yours is ready.
Our numbers, with their provenance
-
52,000log lines per second measured
The engines came through, but where?
A crawl starts from what is linked, and Search Console reports what it chooses to report. Between the two sits an entire category of findings that only the trace of real requests can produce.
-
01
Your important pages are not being crawled
The crawl budget goes into URL parameters, faceted filters and redirects while your commercial pages wait their turn, sometimes for weeks.
-
02
Pages exist that nobody ever requests
No inbound link, no engine request, no visit. They appear in no crawl, precisely because a crawl follows links: it cannot find what has none.
-
03
The team holding the logs is not the team that needs them
The request goes through infrastructure, comes back as a multi-gigabyte archive, and the file stays closed. The SEO question is asked somewhere other than where the data lives.
What we measure
What the parser extracts
A parser written for this one purpose, which streams the file and keeps only what supports a decision. Everything else is dropped while reading, including anything that would identify a person.
- Engine traffic, crawler by crawler (every request is attributed to the agent that issued it, and automated traffic is separated from human traffic. Two distinct quantities, never summed into a total that would mean nothing.)
- Ignored pages and ghost pages (what engines request often, what they never request, and what responds although nothing links to it. A crawl on its own can produce none of those three findings.)
- The join with the rest of the audit (logs, Search Console, rank tracking and site structure meet on the same normalised URL. A missing source leaves the cell empty and says so: never a zero, which would be a measurement.)
- A throughput measured on a bench (the parser has a known throughput, published with its proof in our claims registry. A large archive is analysed without spending the night on it, and without loading the file into memory.)
Further reading
Server log analysis: what your servers know and Search Console does not
The tutorial: collecting your logs and reading them, format by format.
Read ComparisonNessFlow vs Botify & Oncrawl
What an enterprise platform brings to log analysis, and what it asks in return.
Read GlossaryGlossary
Crawl budget, ghost page, orphan page: the definitions.
ReadOther modules
Frequently asked questions
Which log formats do you accept?
The common web server formats, including compressed ones. The parser streams the file rather than loading it into memory: a multi-gigabyte archive goes through without the machine noticing. Its throughput is published together with the bench measurement behind it, so you can estimate the duration before sending anything.
What do you do with IP addresses?
They serve the sorting pass while reading, then they are gone: none is written to the database, none is sent to a model. This is not a switchable setting, it is how the parser is built, and it is the only reason a team can hand us their logs without opening a three-month compliance review.
Why cross logs with Search Console?
Because they do not say the same thing. Search Console reports what Google chooses to report, delayed and sampled; the log reports what your server actually received. A page heavily requested by crawlers yet absent from impressions (or the reverse) is a finding neither source produces on its own.
How long do you keep the logs?
The imported file is processed and then not kept. What remains are aggregates per URL and per agent, subject to a documented retention. Our privacy page publishes those durations as they are written in the purge commands, not as we would like to announce them.
What happens to your logs
A server log is personal data before it is SEO data. The processing was written that way from the start: it is not a configuration option somebody could switch off.
What we cover
- Common web server log formats, read as a stream
- Bot/human separation and per-request attribution
- The join with the crawl, Search Console and rank tracking
Stated limits
- No IP address stored or sent to a model: it serves the sorting pass, then it is gone
- No real time: a log describes what happened, not what is happening
- No explanation: a log says a page is not requested, the cause is still yours to find
- No reverse DNS, and therefore no DNS-based verification of a crawler’s identity
Your logs, without a compliance review
No IP address kept, no data sent to a model: that is what makes a first import possible this week rather than next quarter.
Access opens in waves: we email you when yours is ready.