AI & crawlers/p/[project]/crawlers

What the AI assistants do with your site

A growing share of the people reading your writing never load your page. They ask an assistant, and the assistant loads it for them. This report separates that from bulk corpus collection, because they are completely different events with completely different value, and almost every analytics tool collapses them into "bot traffic" and discards both.

The AI and crawlers page: headline figures above a table of AI agents labelled answering or ingesting, with the pages they read and the referrals they sent back.The AI and crawlers page: headline figures above a table of AI agents labelled answering or ingesting, with the pages they read and the referrals they sent back.
AI assistants, by purpose

Answering is not ingesting

ChatGPT-User means a person asked a question and is waiting for the answer; GPTBot means a corpus crawl. Classifiers that treat both as one agent lose the more valuable of the two. Every agent is listed with its vendor, request volume, share and the exact robots.txt token that would block it.

A table of AI agents (ChatGPT, Claude, Perplexity, GPTBot, ClaudeBot, Google-Extended and others), each labelled answering or ingesting, with request counts, share and robots.txt token.A table of AI agents (ChatGPT, Claude, Perplexity, GPTBot, ClaudeBot, Google-Extended and others), each labelled answering or ingesting, with request counts, share and robots.txt token.

The pages they actually read

Which of your pages assistants fetch, ranked. This is the closest thing there is to a readership report for an audience that leaves no other trace, and it is often a very different list from your human top pages.

A panel ranking the pages AI agents fetched most often.A panel ranking the pages AI agents fetched most often.

Referrals back, the figure that proves it worked

Visitors arriving from an assistant, as a channel of their own. The reading itself is invisible in every other report you have; this is the only number that shows the reading produced a reader.

A panel showing visitors referred from AI assistants, by source.A panel showing visitors referred from AI assistants, by source.

An honest caveat, stated on the page itself

Identification comes from the user-agent string, which any client can set. This is a faithful record of the agents that identify themselves, and it says nothing about the ones that do not. Blocking an ingesting agent is also a real trade: it stops the corpus collection, and it stops the assistant describing your page accurately to the next person who asks.

A breakdown of AI agent requests split between answering and ingesting purposes.A breakdown of AI agent requests split between answering and ingesting purposes.
Specifics

How it works, precisely

Signatures
25+ bot signatures, including AI crawlers, maintained in the collector
Classification
Answering (a person is waiting) versus ingesting (corpus collection)
Filterable
bot_name is an allow-listed dimension in the query layer
Referral channel
Assistants are classified as a distinct acquisition source
Questions

AI & crawlers: the usual questions

Do bots count in my visitor numbers?

No. Bot traffic is filtered out of the human-facing reports at collection and reported separately here, so your visitor counts are people and your crawler figures are complete.

Should I block AI crawlers in robots.txt?

That is a judgement this report exists to inform rather than make. The page shows the robots.txt token for each agent and states the trade plainly: blocking an ingesting agent stops it collecting your writing, and also degrades how accurately an assistant can describe your page to somebody who asks about it.

Run it yourself this afternoon

Clone the repository, bring up the stack with Docker Compose, paste one script tag. The data never leaves your machines.