Latest / Answer Engine Optimization (AEO): The AI Search Podcast / Cloudflare's AI Bot Controls: Search Visibility vs. Training Blockers
Transcript
- [Host] Welcome to the A.E.O. Engine AI Search Show — the number one podcast for brands looking to get cited by ChatGPT, Gemini, and Perplexity. I am your host, Aria Chen. Every day we bring you fresh episodes on A.E.O. tactics, S.E.O. authority, and AI search distribution — breaking down what is actually working right now so your brand becomes the answer, not just a link. Today's guest is Marcus Reid. Marcus is an industry analyst and ex-Google Ads refugee who now spends his days watching AI agents eat the web. Marcus, welcome. [Guest] Hey Aria, happy to be here. And I promise not to mention my failed martech startup more than twice. [Host] We'll hold you to that. So let's start with something that's been buzzing in every S.E.O. Slack I'm in. Picture this: you run a mid-sized e-commerce brand. You finally get a citation in ChatGPT for your best product category. Traffic spikes. Then, three days later, it stops. You check your analytics — nothing from AI sources. You dig deeper and find out your site is returning a 403 for any AI crawler. You never set that block. But Cloudflare did. For you. By default. [Guest] That is the exact panic that's been rippling through the publisher community since July first. And there's actually a name for why that happens now — or rather, a framework. Cloudflare rolled out what they call AI traffic buckets. Instead of a single on-off switch for all AI bots, they split them into three categories: Search, Agent, and Training. [Host] Right. Search is the one you want — that's ChatGPT, Google AI Overviews, Perplexity citing your content. Agent is A.I. tools acting on behalf of users, like an assistant that books a flight by scraping your site. Training is the model-scraping crawlers that build the next GPT. And here's the kicker: for new domains onboarding to Cloudflare, Training and Agent are blocked by default on pages that show ads. Search stays allowed by default. But if you're an existing site that just got migrated or updated, your settings might have flipped without you knowing. [Guest] Exactly. And that's the source of the alarm. Cloudflare sits in front of roughly twenty percent of the web. So when they change defaults, it ripples instantly. In July 2026, they replaced their old one-click AI bot blocker — which was a blunt instrument — with this granular three-category control panel. They also opened a waitlist for a payment gateway that lets any protected website bill AI agents directly for access. That's the monetization angle. [Host] Let's talk about how it actually works under the hood, because I think a lot of marketers hear 'bot management' and glaze over. Cloudflare isn't just guessing if a request is from a bot — they give each request a bot score using detection IDs and bot tags. They maintain a list of Verified Bots — reputable crawlers that declare themselves and follow robots.txt. For AI bots specifically, they now look at HTTP headers that declare intent: 'I'm here to search,' 'I'm here to act,' 'I'm here to train.' [Guest] And they take it a step further. Website owners can opt for managed robots.txt — Cloudflare will automatically generate and update your robots.txt file based on your AI preferences. You can even tune by path: allow AI search bots on your blog, block them on your premium checkout pages. And if you don't want to outright block a bot, you can send a 'Payment Required' HTTP response. It's a digital tollbooth. Cloudflare has a waitlist for an actual payment gateway to process those micro-transactions from AI agents. [Host] So the mechanism is there. But why does this matter beyond the technical detail? It matters because it fundamentally redefines how the web earns money from machine traffic. For years, publishers watched AI companies scrape their content, train models, and then serve answers that reduced clicks to the original source. Now any site owner — even on Cloudflare's free tier — can say 'pay me if you want to train on this.' Or 'you can search my site for answers, but you cannot store my data to build a competitor.' [Guest] And the reaction has been split. The creator community largely cheered — finally a way to say no to free training while still staying visible in AI search. But the AI developer community is alarmed. On subreddits like r/AI_Agents, developers are asking 'Why is no one talking about this?' because agents will be blocked by default on a fifth of the web. They worry about a walled garden where only big AI companies with licensing deals can access content, freezing out open-source and indie agents. [Host] There's also a fascinating nuance called 'transitive trust' that Cloudflare's own engineers have noted. If trust becomes something a bot must carry with them — via verified headers and declared intent — then large tech companies can afford to be identifiable and verified. Smaller developers and researchers may not be able to. That creates an unequal web. You could argue it protects publishers, but it also concentrates power. [Guest] And here's where this lands for the A.E.O. Engine audience. If you are a brand that wants to be cited in ChatGPT, Claude, or Google AI Overviews, you need to be in the Search bucket. But if you let all bots through indiscriminately, you risk your content being used for model training without compensation or even attribution. Cloudflare's tool gives you a dial, not just a switch. But that dial demands active management. You cannot set it and forget it. [Host] That's exactly where A.E.O. Engine comes in. Our whole model is about making your content the canonical answer for AI search engines. But if your site is accidentally blocking the Search bucket because you thought you were just blocking Trainers, you vanish from AI answers. We help brands audit their AI bot access settings and align them with their visibility goals. You want to be the answer in Perplexity and ChatGPT, not get blocked because your Cloudflare default was too aggressive. [Guest] And that's the operator insight: don't just block everything and feel safe. Understand what each bucket means. Allow Search. Monitor Agent. Block or monetize Training. Tune by path. And if you don't have the time or expertise to do that yourself, that's exactly the kind of thing A.E.O. Engine automates. They've helped brands achieve nine hundred twenty percent average lift in AI-driven traffic, and part of that is making sure you're visible where it counts. [Host] So let's wrap with the takeaway. Cloudflare's new AI bot controls are a powerful lever for protecting your content and monetizing it — but they come with a sharp edge. If you block too broadly, you lose AI search citations. If you block too little, your content gets scraped for free. The answer is granularity: allow Search, control Agent, restrict or charge for Training. And if you need a system to manage that at scale while optimizing for AI answer engines, you know where to find us. [Guest] It's not set-and-forget. It's set-and-monitor. And the brands that monitor their AI bot access will win the next phase of search. [Host] Thanks for joining, Marcus. [Guest] Thanks, Aria. [Host] That's all for this episode of the A.E.O. Engine AI Search Show. If you want to make sure your brand is cited — not blocked — by AI search engines, head to https://aeoengine.ai. We'll help you dominate the answer. See you next time.