Quality
Crawler
Also called bot crawler.
Crawler is software that requests pages without a person sitting behind the browser. Search engines, uptime checks and scrapers all qualify.
How it is measured
A crawler fetches a URL, reads the response, extracts links and queues them. It usually identifies itself with a user-agent string and may obey robots.txt. Server logs record every fetch.
A crawler that does not run JavaScript never fires the Tarsier script, so it appears in logs but not in pageviews.
Worked example
A documentation site checks its logs and finds 38,000 monthly requests for /docs/ pages, 24,000 from search crawlers and 6,000 from an SEO tool the marketing team subscribed to. Analytics shows 7,400 pageviews.
The ratio explains why the log-based number never matched the dashboard.
How it differs
A crawler is any automated fetcher. A spider is the link-following kind that search engines run. Bot traffic is the broader label. Server log analytics is where crawlers show up.
Common errors
Treating every crawler as bad. Blocking Googlebot by mistake. Trusting robots.txt to stop abuse. Counting crawls as visits. Forgetting your own monitoring tools.
In practice
List the crawlers in your logs by volume once a quarter and decide which ones you want. Allow the useful ones and limit the rest.