SEO log file analyzer
What your servers know and Search Console will not tell you
Search Console shows you a sample, with a delay. Your servers’ logs keep a trace of every request actually received. We parse them, separate bots from visitors, and cross the result with the rest of the audit, without keeping a single IP address.
Access opens in waves: we email you when yours is ready.
Our numbers, with their provenance
-
52,000log lines per second measured
The engines came through, but where?
A crawl starts from what is linked, and Search Console reports what it chooses to report. Between the two sits an entire category of findings that only the trace of real requests can produce.
-
01
Your important pages are not being crawled
The crawl budget goes into URL parameters, faceted filters and redirects while your commercial pages wait their turn, sometimes for weeks.
-
02
Pages exist that nobody ever requests
No inbound link, no engine request, no visit. They appear in no crawl, precisely because a crawl follows links: it cannot find what has none.
-
03
The team holding the logs is not the team that needs them
The request goes through infrastructure, comes back as a multi-gigabyte archive, and the file stays closed. The SEO question is asked somewhere other than where the data lives.
What a log file analysis tells you
Six questions an SEO asks every time a log file finally opens. Each answer comes from the requests your server actually received, cross-checked with the rest of the audit.
-
Which crawlers really visit, and are they who they claim to be?
Every bot request is checked against the IP ranges its operator publishes (Google, Bing, Apple, OpenAI, Anthropic, Perplexity and others) and comes out verified, spoofed or unverifiable. Spoofed traffic is removed from every count instead of inflating your crawl statistics.
-
What did the crawlers receive?
The share of successful responses, redirects, client errors and server errors served to bots, and the list of URLs that answered with an error, by crawler and by status code.
-
How often do the engines come back?
A daily curve that keeps search engines, AI crawlers, other bots and humans apart, the daily average of each crawler, and the last time each URL was requested.
-
Which pages does Googlebot still request that your site no longer links?
URLs served successfully to a verified Googlebot that your latest crawl did not reach: old sections, forgotten landing pages, parameters still in circulation. Each one is a page to relink, redirect or retire.
-
Is the crawl budget going to the pages that earn traffic?
With Search Console connected, every page is placed on the path known, crawled, indexable, performing. Pages that earn clicks while Googlebot stays away, and pages crawled heavily without a single impression, come out as two separate lists.
-
Do AI crawlers read your content?
Training crawlers and answer-engine crawlers are counted separately, then compared with what your robots.txt allows and with the responses they received. An AI crawler that never shows up is only reported as absent after a full week of logs.
What we measure
What the parser extracts
A parser written for this one purpose, which streams the file and keeps only what supports a decision. Everything else is dropped while reading, including anything that would identify a person.
- Engine traffic, crawler by crawler (every request is attributed to the agent that issued it, and automated traffic is separated from human traffic. Two distinct quantities, never summed into a total that would mean nothing.)
- Ignored pages and ghost pages (what engines request often, what they never request, and what responds although nothing links to it. A crawl on its own can produce none of those three findings.)
- The join with the rest of the audit (logs, Search Console, rank tracking and site structure meet on the same normalised URL. A missing source leaves the cell empty and says so: never a zero, which would be a measurement.)
- A throughput measured on a bench (the parser has a known throughput, published with its proof in our claims registry. A large archive is analysed without spending the night on it, and without loading the file into memory.)
How to analyze your server log files for SEO
Four steps, from the archive your host sends you to a list of decisions. The first is the only one that asks for a little of your time.
-
01
Export your access logs
Ask for at least a week of access logs, a month if you can: a shorter window describes a coincidence. Common formats are read as delivered, compressed archives included: combined (nginx, Apache), IIS W3C, Cloudflare and Caddy JSON, CloudFront, load balancer and object storage logs, Traefik, or a custom format string.
Export guides, format by format
-
02
Import them into your project
A guided import sends the file in chunks and resumes where it stopped if the connection drops. The same file sent twice is recognised and refused, so a period is never counted twice.
-
03
Let the parser sort and verify
Each line is attributed to a crawler or to a visitor, each bot is checked against its operator’s published IP ranges, and the address is dropped as soon as that check is done. The raw file is deleted once processed.
-
04
Cross the result with the audit
Logs meet the crawl, Search Console and rank tracking on the same normalised URL. You leave with error URLs to fix, orphan pages to relink or retire, and an Excel export of every table.
Further reading
Server log analysis: what your servers know and Search Console does not
The tutorial: collecting your logs and reading them, format by format.
Read ComparisonNessFlow vs Botify & Oncrawl
What an enterprise platform brings to log analysis, and what it asks in return.
Read GlossaryGlossary
Crawl budget, ghost page, orphan page: the definitions.
ReadOther modules
Frequently asked questions
Which log formats do you accept?
The common web server formats, including compressed ones. The parser streams the file rather than loading it into memory: a multi-gigabyte archive goes through without the machine noticing. Its throughput is published together with the bench measurement behind it, so you can estimate the duration before sending anything.
What do you do with IP addresses?
They serve the sorting pass while reading, then they are gone: none is written to the database, none is sent to a model. This is not a switchable setting, it is how the parser is built, and it is the only reason a team can hand us their logs without opening a three-month compliance review.
Why cross logs with Search Console?
Because they do not say the same thing. Search Console reports what Google chooses to report, delayed and sampled; the log reports what your server actually received. A page heavily requested by crawlers yet absent from impressions (or the reverse) is a finding neither source produces on its own.
How long do you keep the logs?
The imported file is processed and then not kept. What remains are aggregates per URL and per agent, subject to a documented retention. Our privacy page publishes those durations as they are written in the purge commands, not as we would like to announce them.
What is SEO log file analysis?
Reading your web server’s access logs to see what search engine and AI crawlers actually requested, when, and what your server answered. A crawl shows what can be found and Search Console shows what Google chooses to report; the log is the only record of every request received, which makes it the reference for crawl budget, crawl errors and crawler verification.
Can you tell a real Googlebot from a fake one?
Yes. Each request that claims to come from a known crawler is compared with the IP ranges its operator publishes, and receives one of three verdicts: verified, spoofed or unverifiable. Spoofed requests are kept out of every figure. When an operator’s published list is more than a week old, its crawlers are marked unverifiable instead of being guessed. The check relies on published ranges only, without a reverse DNS lookup.
What happens to your logs
A server log is personal data before it is SEO data. The processing was written that way from the start: it is not a configuration option somebody could switch off.
What we cover
- Common web server log formats, read as a stream
- Bot/human separation and per-request attribution
- The join with the crawl, Search Console and rank tracking
Stated limits
- No IP address stored or sent to a model: it serves the sorting pass, then it is gone
- No real time: a log describes what happened, not what is happening
- No explanation: a log says a page is not requested, the cause is still yours to find
- No reverse DNS, and therefore no DNS-based verification of a crawler’s identity
Your logs, without a compliance review
No IP address kept, no data sent to a model: that is what makes a first import possible this week rather than next quarter.
Access opens in waves: we email you when yours is ready.