"AI bot detection" splits into two jobs: crawler policy and text detection.

Two completely different things. Crawler / agent detection spots when GPTBot, ClaudeBot, PerplexityBot or autonomous AI agents are hitting your site, and lets you allow, block or monetise that access. Cloudflare AI Crawl Control, Known Agents and TollBit live here. AI-generated content detection tries to work out whether a piece of text was written by a human or an AI. GPTZero, Originality.ai and Winston AI live here. The two share the search query but solve different problems.

If the question is "who is crawling my site and should I let them", you need crawler management - Cloudflare AI Crawl Control if you are already on Cloudflare, Known Agents for visibility, TollBit if you want to charge for access. If the question is "did a human write this", you need a content detector - GPTZero or Originality.ai are the usual starting points. Marketing and SEO teams almost always want category one. Editors, lecturers and publishers almost always want category two. Sometimes both, in which case start with the question that has the bigger commercial cost attached.

Depends on the business model. For most marketing sites, allow the AI search bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot. These are the bots that affect whether AI search can find and cite you. GPTBot and ClaudeBot only control training, and Google-Extended covers Gemini training and grounding. For paywalled publishers or licensable content, blocking or charging through TollBit makes sense. Default is allow unless you have a specific reason to block.

Server logs show user-agent strings. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Meta-ExternalAgent identify themselves clearly. Google-Extended never shows up in logs: it is only a robots.txt token, and Google crawls under its normal user agents. Use the free AI Bot Robots.txt Tester for a quick policy check, Known Agents for ongoing visibility, or Cloudflare AI Crawl Control if you already terminate at Cloudflare.

For most sites already on Cloudflare, yes. AI Crawl Control and its per-bot allow/block controls cover the common needs, on every plan including Free. One catch since 15 September 2026: Block now also stops Googlebot, Bingbot and Applebot, so it affects search as well as training. To stop training and keep search, use Disallow AI Training. Move to a specialist tool only if Cloudflare lacks something specific - usually publisher monetisation (TollBit), enterprise scraping defence (DataDome, Akamai) or agentic-traffic governance (HUMAN AgenticTrust).

Not always. Anthropic says its bots honour robots.txt. The others carve out fetches a person asks for. OpenAI says robots.txt rules may not apply to ChatGPT-User. Google says its user-triggered fetchers, including Google-Agent, added in March 2026, generally ignore robots.txt. Perplexity says the same of Perplexity-User, and Meta says Meta-ExternalFetcher may bypass it. In August 2025 Cloudflare reported Perplexity using undeclared crawlers on sites that had blocked it, and removed it from its verified bot list. Smaller or less reputable AI crawlers may not respect robots.txt at all. If the content is worth scraping and reputation does not deter, a CDN-level block is more reliable than robots.txt alone.

Less accurate than their marketing suggests, and none hit 100%. The test I use here comes from Scribbr, which sells its own detector and ranked it first. Scribbr published the test on 22 April 2026 and updated it on 21 July 2026, using GPT-3.5 and GPT-4 texts: Scribbr premium 84%, QuillBot and Scribbr free 78%, Originality.ai 76%, ZeroGPT 64% and GPTZero 52%. Winston AI was not tested. False positives are most common with edited human writing, ESL text and short samples. Treat any detector output as a signal that triggers a human review, not as a verdict.

It depends which bots. Blocking OAI-SearchBot keeps you out of ChatGPT search answers, though you can still appear as a navigational link. Blocking Claude-SearchBot may reduce your visibility in Claude search, and Perplexity recommends allowing PerplexityBot if you want to appear in its results. Blocking GPTBot and ClaudeBot only stops training. Blocking Google-Extended stops Gemini training and grounding, but does not remove you from AI Overviews or AI Mode. Those follow your Googlebot rules and Search Console's generative AI control. User-triggered fetchers such as ChatGPT-User and Perplexity-User may ignore robots.txt. For some publishers blocking is the right call. For most marketing and B2B sites, blocking the search bots removes a growing channel of qualified traffic.