Latest / Found in AI: AI Search Visibility, SEO, & GEO / How Should Brands Measure Visibility in AI Search?

Transcript

How Should Brands Measure Visibility in AI Search? · Found in AI: AI Search Visibility, SEO, & GEO

• 0:41
  (Transcribed by TurboScribe.ai. Go Unlimited to remove this message.) Hey, before we get started, I want to give you a quick heads up.
• 0:45
  Now, this episode might sound a little different than usual, not because the audio quality is bad, but because I'm dealing with some massive brain fog today, and it's making it harder than normal to find the words or get sentences out smoothly, even though I have notes right in front of me.
• 1:01
  So, I have a medical thing coming up that should address all this, and I am okay, I promise.
• 1:06
  But for today, I'm asking for a little patience if I sound slower or more jumbled than normal, which has been going on for a hot minute, but just now seems like it might be an actual thing.
• 1:18
  So, that said, the topic today is super important, and I don't want to skip it, so thank you for sticking it out with me throughout this episode, even if I stumble over a bunch of words.
• 1:30
  But welcome back, I'm Cassie Clark.
• 1:33
  I'm a content strategist and AI search optimization expert, which is still weird to say, but you are listening to Founded AI, the show dedicated to helping marketers and founders learn AI search and GEO so we don't get lost in this new wave of user search behavior that is coming or here, depending on how you look at it.
• 1:55
  Today, we're talking about a piece of research that made me go, finally, finally, the data has caught up, and I'm excited about it.
• 2:04
  Rand Fishkin at SparkToro just published new research last week showing what many of us elbow deep in AI search experiments have learned.
• 2:14
  AI search doesn't behave like classic search, it is super inconsistent and hecka probabilistic, and if you're trying to rank in number one in chat GPT, I need you to throw that phrase out of your vocabulary because it does not apply to engines like chat GPT, perplexity, AI overviews, whichever one that you're using.
• 2:38
  Here's why.
• 2:49
  Companies are spending real dollars on AI visibility tracking.
• 2:55
  As Rand points out in his post, there's already an estimated $100 million a year going into this category, and if you look at these tools, well, the obvious question is, are these AI engines consistent enough for rank tracking to mean anything?
• 3:16
  So Rand and a partner at Gumshoe AI decided to test it, and you know, as much as I do, we do love good experimentation over here.
• 3:25
  So for their research, this wasn't just a, I ran three prompts and had a thought about it.
• 3:31
  Instead, they had volunteers, like a bunch of them.
• 3:35
  600 volunteers ran 12 prompts across chat GPT, Claude, and Google's AI overviews, Gemini, Google, for a combined 2,961 runs, and then they normalized the results into ordered lists so that they could compare apples to oranges, or apples to apples, or apples to grapes, or whatever, and then they shared it with all of us.
• 4:03
  So here's the headline that came out of this research.
• 4:07
  If you ask the same AI tool the same best brands or products question 100 times, there's less than a 1 in 100 chance that chat GPT, or we'll call it Google AI, gives you the same list of brands twice, and for the same order, it's closer to 1 in 1,000.
• 4:30
  Also, the length of those lists changes.
• 4:34
  Sometimes the model gives you two to three recommendations, sometimes it's 10 or more from the same exact prompt.
• 4:42
  So if your dashboard is telling you, you moved from number four to number two in chat GPT, I need you to hold on just a second because that's more astrology with user interface than real insight.
• 4:59
  Now, here's where I really love Rand's research and his report that he put out, because he didn't stop at dunking on rankings, because the next part of it that was super important is that even when an order is random, the set of brands can be weirdly stable.
• 5:19
  Now when I say set, I mean the brands that the engine recommends over time.
• 5:24
  So for example, when Google AI was asked to recommend digital marketing consultants for e-commerce, one agency, Smart Sites, appeared 85 out of 95 times.
• 5:37
  So the ranking position in an AI engine, calling it a ranking position, is junk.
• 5:43
  The real question is, does the brand show up at all?
• 5:49
  That's what changes from traditional search, because we need to stop tracking position.
• 5:55
  When I say that, I am talking solely about AI engines.
• 5:59
  We still do need to track our position in traditional search, but we also need to start tracking probability instead for these AI engines.
• 6:09
  Rand and company also tested something that matters a lot in the real world.
• 6:14
  People don't all search the same way anymore.
• 6:19
  They measured the similarity of written human prompts and found the semantic similarity was something like 0.081, meaning people can have the same intent, but phrase it differently.
• 6:30
  And if that's not true to how humans talk and interact with each other, I don't know what's true.
• 6:36
  Despite all of this, the models still returned a relatively consistent core set of brands in some categories.
• 6:46
  So in one test, across 142 human-crafted prompts and 994 AI responses, headphone brands like Bose, Sony, Apple, and another one that I'm going to pronounce incorrectly because it's that kind of day, they appeared 55 to 77% of the time.
• 7:06
  So even when prompts are messy, AI engines pick up on intent recognition, and that's where visibility percentage really holds up in what we should be paying attention to.
• 7:19
  Now, before I go on, I want to slow down for a second.
• 7:22
  When we talk about AI visibility, we're really talking about two different measurements, and they answer two different questions.
• 7:29
  The first question is simple.
• 7:30
  Do you appear at all?
• 7:32
  That's visibility frequency.
• 7:34
  It's about how often your brand shows up across repeated runs of the same prompt or across closely related prompts.
• 7:44
  There's that stumbling coming in.
• 7:48
  Now, because AI answers are probabilistic, this number will fluctuate, but over time, those patterns emerge.
• 7:56
  That's why you'll see something like 55 to 75 or 77% of the time these certain headphones appear.
• 8:03
  Now, the second question is more specific.
• 8:06
  When you do appear, how much of the answer do you own?
• 8:11
  That's AI share voice.
• 8:12
  I've talked about it a couple of times on this podcast.
• 8:15
  The math for AI share voice needs to be right, but it's super, super simple.
• 8:20
  To calculate it, it's per prompt, not across batch.
• 8:24
  You just take the number of times your brand is mentioned in the AI answer and divide it by the total number of brand mentions for the same answer.
• 8:33
  If you ask, who are the best AI search optimization consultants in the AI list for companies total, including yours, your AI share voice for that prompt is 25%.
• 8:46
  That number actually means something.
• 8:49
  It doesn't really care if you were listed first, third, or last.
• 8:53
  It just tells you how much of the recommendation space you occupy in that moment.
• 8:58
  Now, here's where Rand's research connects directly to something I have been saying.
• 9:05
  Because rankings reshuffle constantly, the positioning in traditional SEO sense doesn't really matter, but when a brand appears consistently and then earns meaningful share of citation inside of those answers, well, it becomes part of the model's default consideration set from what we can see right now.
• 9:25
  That is what brands should be targeting.
• 9:27
  That brings me to the case study that I shared back in December titled The Displacement of Legacy Authority, a 125-hour AI share voice case study.
• 9:37
  I will link that in the show notes if you want to dive deeper.
• 9:40
  In this study, I found that after a targeted GEO update, my brand began appearing more often, and when it appeared, it occupied a larger portion of the answer.
• 9:52
  So, if you've heard me talk about it, you know the brand that I displaced was Search Engine Journal.
• 9:58
  My brand didn't outrank Search Engine Journal because there is no stable ranking to beat.
• 10:03
  But over time, the AI engine started selecting it instead of the legacy giant in certain responses.
• 10:10
  Now, again, we can't say that my brand outranked it because different runs of the same prompt result in different citations, but we can say that the probability of being mentioned increased because of the content being more aligned with the user intent.
• 10:27
  This is how smaller brands can compete in AI search.
• 10:30
  These engines don't care about domain authority or your website size.
• 10:36
  They do pick up on consistent reinforced patterns across channels, social media, landing pages, websites, wherever you are putting your brand out there on the internet.
• 10:49
  If the model learns that your brand is relevant, credible, and contextually appropriate, it will include you regardless of how big you are.
• 10:58
  So, back to Ran's report for a second.
• 11:01
  I'm going to read his conclusion opener word for word with the added context before I get into it.
• 11:07
  So, before his conclusion, he had a bunch of questions like, how many times should a user run a prompt to get statistically sound answers about a brand's relative visibility?
• 11:18
  His conclusion opener said, more data is needed.
• 11:22
  I fully agree with that.
• 11:23
  More people should look into these questions.
• 11:26
  I agreed.
• 11:27
  And then he says, anyone who's selling AI tracking should be ashamed of themselves if they don't publish transparent public reviewable reports on this stuff.
• 11:39
  Want to pause?
• 11:41
  Because let me preface all this by saying, tools are not the enemy.
• 11:46
  There are very smart teams building in this space who are working on tools to help with AI tracking.
• 11:53
  And I have spoken with many of the builders and many of them for the show.
• 11:57
  You've heard their episodes.
• 11:59
  Measurement will absolutely matter when the dust settles on AI search and how we optimize for it.
• 12:06
  But tools are amplifiers.
• 12:08
  They are not teachers.
• 12:10
  I say that to say, if you don't really understand how AI systems generate answers, why those rankings fluctuate, or what visibility actually represents, or which prompt matters to your brand, I'm talking about those money prompts, a dashboard's not going to help you too much.
• 12:28
  It will just give you numbers that you don't know how to interpret.
• 12:31
  But I say that lovingly because it goes with any tool that you use for anything.
• 12:37
  So my advice to you when thinking about AI visibility and how to track it is before you spend money, especially money that stretches your budget, because I know some tools are kind of pricey at the moment, I want you to take time to understand the mechanics first.
• 12:54
  Decide what visibility means in your category.
• 12:57
  Decide what success actually looks like.
• 13:00
  Then go chose a tool that aligns with those goals and can give you the measurements that you're after, not the other way around.
• 13:07
  No.
• 13:09
  Here's the too-long-didn't-read version of all this.
• 13:12
  Rand's research didn't prove that AI visibility can't be measured.
• 13:16
  It just proved that rankings are the wrong metric.
• 13:19
  Visibility is a probability game now.
• 13:23
  If you want help understanding how AI engines evaluate brands, measuring your real AI share of voice, or even help building a geo-strategy that increases your chance of being included, that's exactly what I help brands do.
• 13:35
  Head over to my website, CassieClarkMarketing.com, for more resources, and if you want to start your AI search visibility audit this month, check the show notes for the link.
• 13:45
  Okay, that's it for this episode.
• 13:48
  I will link Rand's research in the show notes below.
• 13:52
  Until the next episode, stay visible.