Technical
Bot management
Bot management is the set of rules and signals that sort incoming requests into people, good crawlers, and abusive automation, then allow, challenge, or block each one. It acts per request, using IP, headers, TLS fingerprint, and behaviour.
How it is measured
Signals include the user-agent checked against reverse DNS (a real Googlebot IP resolves under `googlebot.com`), request rate per IP and per path, missing headers such as Accept-Language, the TLS fingerprint, and whether JavaScript ran. The output is a score or label per request, logged next to the action taken.
Tune it against the access log. Count requests by action (allow, challenge, block) per day and per path. A login endpoint with 12,000 POSTs from 300 IPs and no earlier GET is automation, whatever the user-agent claims.
Worked example
An events WordPress site sees `/wp-login.php` take 9,400 POSTs in an hour from 1,800 addresses, each trying one password. A CDN rule challenges any POST to that path that lacks a prior cookie. Challenges served: 8,900. Solved: 31.
Meanwhile verified Googlebot fetches 2,300 pages in the same hour. It had to be named in an allow rule first, or the challenge would have made the crawl fail.
How it differs
Bot management judges and acts on each request. A Turnstile challenge is one tool inside it that asks the client to prove it is a real browser. Bot management also covers rate limits and silent blocks that never show a challenge, and Turnstile on its own makes no judgment you did not ask for.
Common errors
Trusting the user-agent string. Blocking every data-center range and cutting off uptime monitors and your own API clients. Challenging crawlers you want. Counting challenge pages as real page loads. Writing a rule once and never checking the false-positive rate.
In practice
Pull a day of access logs, group by IP and path, and read the top 20. Allow verified crawlers and your monitors by name, challenge the sensitive endpoints, and rate-limit the rest. Once a week, check that real customers are not being shown the challenge.