Latest / Found in AI: AI Search Visibility, SEO, & GEO / Should You Block AI Bots From Your Website? | AJ Ghergich, Botify [Part 3 of 3]

Transcript

Should You Block AI Bots From Your Website? | AJ Ghergich, Botify [Part 3 of 3] · Found in AI: AI Search Visibility, SEO, & GEO

• 0:41
  Hey, welcome back to Found and AI.
• 0:43
  I'm Cassie Clark, a fractional content strategist, an AI search optimization expert, and the host of the show where we talk about AI search, GEO, AEO, and what all of it actually means so we don't get lost in this new wave of user search behavior.
• 0:58
  My cat is at the my feet under my desk and he's making a lot of noise.
• 1:02
  He's orange and he's sneezing.
• 1:04
  He's not the only one sneezing, I'm also sick.
• 1:12
  But today we're getting into something that does not get nearly enough airtime.
• 1:17
  AI bot governance.
• 1:19
  I don't mean that in the dry compliance y kind of way, but I mean it in the real messy question that every company is wrestling with right now.
• 1:26
  Which bots can access your site, what they're allowed to crawl, and whether the content they grab is used to train a model or to answer a customer in live retrieval.
• 1:36
  Spoiler, those are two very different problems with very different answers.
• 1:41
  This is the third part of the series with AJ Gergic.
• 1:44
  He leads consulting in the AI team at Botify.
• 1:47
  He's been having these conversations with some of the biggest brands in the world every day about what to do with these AI bots.
• 1:53
  We get into why governance falls through the cracks between marketing, IT, and legal.
• 1:58
  The three questions that most CMOs still can't answer what a good plan looks like versus a bad one.
• 2:04
  And I take us on a little side road about the LOMS.txt file and kind of thank goodness that I did at the time because Google is changing guidance on that by the minute, it feels like.
• 2:32
  Yeah, I I think you know governance means different things to different people, obviously.
• 2:38
  I think inside of a company right now, um the main question people are saying is AI governance marketing, IT, or like a legal problem?
• 2:51
  And then the to be the obvious answer, I guess, is it's all three.
• 2:55
  Um, and that's why it falls through the cracks.
• 2:58
  Um, so it's it's really a governance is a it's a set of decisions about what which bots can access your site, what they can crawl, um, whether the content that they get used is for training or for live retrieval for a customer.
• 3:16
  Um, you know, and everybody cares about it, but for different reasons.
• 3:19
  So like marketing cares because it affects visibility.
• 3:23
  IT cares because they have that issue where they're getting inundated, crushed with bot traffic that they have to serve, and that's a cost and an infrastructure problem versus humans, right?
• 3:35
  So it's like this huge load on the IT side.
• 3:38
  Um, and then legal cares because maybe your product data is showing up, you an AI you didn't authorize, maybe it's hallucinating facts, and now are you liable for that?
• 3:49
  It it said it was good for this, but it's not, right?
• 3:53
  It that could be a problem.
• 3:55
  Like, who's whose fault is that gonna be?
• 3:56
  Is it gonna be Open AI's fault or yours?
• 3:59
  Um, so everyone cares about it, but I think it's really um a combination right now of marketing IT and legal um just wrestling with those questions.
• 4:10
  And I I think that kind of leads to the obvious question of like, where do you start then?
• 4:17
  Right.
• 4:17
  I you know, yeah.
• 4:18
  Well, where do we start?
• 4:19
  Where do we start with that?
• 4:22
  So uh, you know, again at Botify, I get to I mentioned I get talked to a lot of a lot of folks, a lot of big companies, global companies, um, and and growth companies.
• 4:30
  Um, so this is the type of conversation I have all the time.
• 4:33
  Like, how do we start?
• 4:34
  What should we do?
• 4:35
  You know, that kind of thing.
• 4:37
  Um, I think it's you start with some simple questions.
• 4:41
  Um, so as a brand, do you know what data of yours is being used to train AI models?
• 4:50
  And if you don't have like if you have like an extremely loose answer to that, and you're you're not just a you know one woman shop, you know, like like you you should be able to answer that question as an organization.
• 5:04
  Um and then comes the the next question is that data accurate?
• 5:09
  Is it updated?
• 5:10
  Um and then do you have a plan in place for what you're gonna allow AI to retrieve for training and in real time?
• 5:20
  And I, like I said, talk to a lot of brands.
• 5:24
  Most cannot answer that question.
• 5:25
  Some of the biggest brands in the world, at least I would say a year ago, it's getting better now, could not answer those three questions.
• 5:32
  So I would ask a CMO, I'd ask her those three questions, couldn't answer it.
• 5:36
  So I don't want anybody who is like, I can't answer those questions to panic.
• 5:40
  Like you're not, it's not uncommon.
• 5:42
  But I do think it's you need to start thinking about that.
• 5:45
  Um, you need to start thinking about um what bots are hitting your site, how often, um, what information they're getting, all the way down to like, let's say you're not a technical person.
• 5:55
  Let's say you you're like, AJ, I'm a I I I work on branding.
• 6:00
  You know, I'm a brand manager.
• 6:02
  Okay.
• 6:03
  Well, what if OpenAI has three CMOs ago mission statements about your brand and its training data?
• 6:12
  How could that create confusion about what you say your mission is today, right?
• 6:17
  What about your fonts?
• 6:18
  Have the website, your colors, everything.
• 6:20
  Like, think about all the confusion that's out there that it may be having from old deprecated data about your brand and your positioning versus today's positioning.
• 6:31
  Like, so this is an everyone conversation.
• 6:34
  You can't just go, well, that's IT, let them deal with the bots.
• 6:38
  It's like, no, that's it's a brand conversation, right?
• 6:41
  Like, so it everyone should get involved.
• 6:44
  Um, and then the question I think becomes um, you know, who's running the show?
• 6:52
  Right, right.
• 6:53
  I mean, I would have been saying just for like AI search optimization in general, that this is not just a marketing thing.
• 7:00
  It is, but it touches every layer of the organization.
• 7:03
  So even with this conversation, it's still every layer of the organization.
• 7:09
  So let me ask you a question because I just was talking to a brand earlier in the week.
• 7:14
  She was asking me about the files that you might may or may not need on your website, talking about the robot.txt file and the LM LS file.
• 7:24
  Do we need those?
• 7:26
  So this is the new like SEO bar fight question.
• 7:31
  So it used to be um root domain versus subdomain, which I won't dork out, but um if I'll just say this if your if your IT ever asks you if you want to put some content on a subdomain or your main domain, put it on your main.
• 7:47
  So it just it it's it's better for everyone.
• 7:51
  But um this used to that used to cause all kinds of issues with inside the SEO community.
• 7:57
  This is the new version of that.
• 7:59
  Um the short answer is for 95% of brands, you don't need the LLM's TXT.
• 8:08
  Um it's not getting picked up um uh in mass.
• 8:12
  Google's representatives will come out and say they they're not using it.
• 8:17
  Um now that said, you could we track, we bot Botify tracks log files, we look at it.
• 8:23
  They do get used in Hit.
• 8:25
  It's just here's the thing there's 10 other things you should be doing.
• 8:30
  That this is the the 11th, right?
• 8:33
  And so that's that's where I'm just like when you're doing this, you're not doing something else.
• 8:38
  And I'd much rather you be doing something else than that, and then you got to maintain it and all that.
• 8:43
  It feels good, it's a shiny object, and it's like, ooh, we're cool.
• 8:47
  We have uh LLMs TXT, it's not gonna do you any good, it's not gonna ring the register.
• 8:52
  Um, I wouldn't do it for 95% of brands.
• 8:56
  Okay, that one hit my head, and I'm like, I have to ask it right now.
• 8:58
  But let's go back to the AI bot governance plan.
• 9:02
  What does a good plan look like versus what does a bad one look like?
• 9:06
  And when do you just call it done?
• 9:08
  Yeah, so I I think a bad plan is very binary.
• 9:13
  Like we block AI bots, um, you know, or we don't, that kind of thing.
• 9:19
  Um you need to have enough data at your fingertips to know what are good bots and what are bad bots.
• 9:26
  Um, and then what bots um are training bots and what bots are live retrieval.
• 9:33
  So the for a retailer, a lot of folks think I need to get my data into Chat GPT.
• 9:42
  Like I just need to know about my brand and like my product.
• 9:45
  But you gotta realize so while we're talking, maybe it's happening or maybe it's gonna happen in an hour, like ChatGPT is gonna release their new model 5.5, right?
• 9:55
  And so when that happens, the moment that happens, it's gonna be six months out of date, maybe a year out of date.
• 10:03
  And so my point is it doesn't know if those sneakers are in size 11, uh, if you know pink is in stock, it doesn't know, right?
• 10:14
  Um, and so most retailers actually need to ri win at the moment of live retrieval.
• 10:21
  When an when a search uh when an AI searches on a consumer's behalf to see if size 11 is in stock in pink at this price, what is the shipping?
• 10:34
  And so a lot of folks don't think about it that way, and they're not looking at it that way.
• 10:39
  Um, I would do it almost like a governance plan should include like user testing, but for AI agents, right?
• 10:46
  So can the AI agent at live retrieval see if it's in stock and peak in size 11 or not?
• 10:54
  Or does it have have trouble in accessibility?
• 10:56
  So um it's not a binary choice.
• 10:59
  You have to have data for the plan to work, and then honestly, it really should have a hub.
• 11:05
  Like I believe that um AI can be a hub for all of your spokes.
• 11:10
  Um, and so you know, you see a lot of folks putting a chief AI officer.
• 11:15
  You could also um even think about your uh chief uh customer officer becoming uh having AI in in her purview, perhaps, um, so that you you're connecting all of these different spokes under this AI hub because you're right, it's not just marketing, it's it's everybody.
• 11:35
  It's absolutely everyone.
• 11:37
  So I mean you make an interesting point about the training data and live retrieval being different.
• 11:43
  So if you're noticing that your training data or what the training data says about your brand is not correct, how are you going about fixing that?
• 11:52
  Yeah, so that's a it's a it's a bit of a longer cycle, obviously, right?
• 11:56
  Because like it's it, it's like it's it's um it's already out of the bag, like right, like it's it's already in the in the uh in the training set.
• 12:06
  Uh the first thing I would be doing is figuring out where the heck it got the information.
• 12:11
  So um, you know, I'll leave their name out of it.
• 12:14
  Uh, but uh a large uh cosmetics company I'm on the phone with them.
• 12:18
  We're we're this is maybe a year ago, going through um their AI visibility and looking.
• 12:23
  And essentially the AI started falsely saying that their cosmetics were not like animal friendly.
• 12:31
  And like, imagine the shitstorm that might cause, right?
• 12:36
  Like it's like like like that, that is a no, are you kidding me?
• 12:39
  And it's like, where is it getting the information?
• 12:41
  Because sometimes it's pulling the information from Reddit or a forum.
• 12:45
  Um, so some and sometimes it's your own information.
• 12:48
  Obviously, in this case, it wasn't their information, it was somebody else falsely stating.
• 12:52
  So the first thing you want to do is hunt down where is it getting the information and the facts?
• 12:57
  And that's why you need everybody at the table.
• 12:59
  You need marketing, brand, SEO, you know, you need everybody at the table because why is it why does it have our story wrong?
• 13:08
  So is it because it can't read JavaScript and it's getting half our content wrong?
• 13:12
  Is it because our data is telling three different stories?
• 13:16
  Is it because it's reading some sweaty Redditors post that is not accurate and we need to talk about PR and going in and correcting the record?
• 13:27
  So all of those things would be good.
• 13:29
  And then proactively is if you can get into a mindset, especially if you're e-commerce and retail, about pushing, um, getting as much information in your data feeds to push.
• 13:41
  Um, so you'll see in the next, you know, coming weeks and months, uh, all the major players will be uh adding agentic feeds uh for commerce.
• 13:53
  That's a great way to correct the record and add your data because you're pushing versus waiting for it to crawl and learn.
• 14:00
  So you're pushing your information in.
• 14:02
  So it's a complicated answer and not a perfect answer, but you've got to find the the source of the rot at first before you can do anything.
• 14:12
  So you said everyone at the table.
• 14:13
  Is there anyone that we're missing?
• 14:15
  So we're talking about brand, marketing, legal, IT.
• 14:18
  Anyone else?
• 14:20
  I honestly I would start, I would start there.
• 14:22
  I think um I think every organization is gonna be a little different.
• 14:26
  I I I wouldn't say um that's a closed group.
• 14:30
  There's gonna be there's gonna definitely need to be more people on the loop, but that's a good that's where I see 80, 90% of people starting is with those people at the table.
• 14:39
  And then I would widen the tent from there.
• 14:42
  Um but yeah, that's that's a you can't go too wrong with if that's your starting block.
• 14:49
  Okay, that was the final part of my conversation with AJ, and there are a few things that I want you to walk away with.
• 14:55
  AI bot governance isn't a binary block the bots or don't kind of decision.
• 15:00
  It's a set of decisions that needs marketing, brand, IT, and legal at the same table because no single one of them owns it.
• 15:09
  Start with AJ's three questions when you bring everybody to the table.
• 15:13
  First, do you know what data of yours is training the AI models?
• 15:17
  Is that data accurate and current?
• 15:19
  And do you have a plan for what you'll allow AI to retrieve both for training and in real time?
• 15:25
  If you can't answer those yet, you're in a good company.
• 15:28
  Everyone is still trying to figure it out, but that's the work that we need to start with.
• 15:32
  Now, quick update on my lolemass.dxc detour.
• 15:37
  The story has moved a little since we recorded that.
• 15:40
  Uh we recorded this like a month or so ago, and then now there's new guidance out.
• 15:45
  On the show, AJ's take was that for 95% of brands, that file is not worth your time.
• 15:51
  And then Google two weeks ago has said, hey, you don't need this for visibility.
• 15:57
  That's Google's official line still, but they've also added an LLMS.txt check to Gross Lighthouse under a new agentic browsing set of audits that flag whether your site even has the file.
• 16:11
  So naturally, people saw that and went, wait, I thought we didn't need this because it was announced within the same week of Google saying you don't need it.
• 16:20
  Here's the nuance with all of this.
• 16:22
  Those audits are about how ready your site is for AI agents and browser tools, not just search rankings.
• 16:28
  Google's John Mueller framed it as the difference between discovery, getting found, and functionality, meaning helping an agent do its job once they're already it's already on your page.
• 16:39
  So it doesn't really contradict AJ's point for most brands.
• 16:43
  If you're a developer docs or heavy agent traffic site, it might be worth looking into putting this on there.
• 16:50
  For everyone else, it's the same answer.
• 16:52
  There are 10 more important things to do first.
• 16:55
  If this helped you, this episode helped you any hit subscribe.
• 16:58
  I would love you forever.
• 17:00
  I would love you even more if you had 30 seconds and you could just leave a review.
• 17:03
  Those reviews do help the show get in front of more marketers and founders who are trying to navigate this stuff.
• 17:09
  And if you're starting your content strategies this week and you're thinking, okay, where do I even start with all of this?
• 17:14
  Head over to CassieClark Marketing.com to get an AI search visibility audit.
• 17:19
  I'll show you exactly where you're showing up, where you're invisible, what to do about it, why your competitors are winning.
• 17:26
  I hate saying that, but why they're winning, and how you can beat them.
• 17:30
  Alright, that's it for this week.
• 17:32
  I will see you in the next episode.
• 17:34
  Until then, stay visible.