Latest / Found in AI: AI Search Visibility, SEO, & GEO / Does ChatGPT Recommend the Same Brands to Everyone? [NEW RESEARCH]
Transcript
- 0:41SPEAKER_00: Hey, welcome back to Found in AI, the podcast where we talk about AI search, generative engine optimization, and what marketers actually need to know to keep their brands visible as search changes.
- 0:53SPEAKER_00: I'm Cassie Clark, an AI Search Visibility Consultant, and I help brands understand how they're showing up in tools like ChatGPT, Google AI Search, all of them, and what to do about it.
- 1:04SPEAKER_00: And today we're talking about a problem that makes measuring that visibility a whole heck of a lot more complicated.
- 1:11SPEAKER_00: Do these AI engines recommend the same brands to everyone?
- 1:15SPEAKER_00: Because we tend to talk about AI visibility as if there's one answer to that question, your brand either shows up or it doesn't.
- 1:22SPEAKER_00: But AI responses can change based on the model, how you're accessing it, where you are, and what that model already knows about you.
- 1:31SPEAKER_00: So what does that mean for the way that we measure AI visibility?
- 1:35SPEAKER_00: So today I'm talking with Shaw from Fiction AI about a study we worked on together comparing brand recommendations across ChatGPT, Gemini, Claude, and Perplexity.
- 1:46SPEAKER_00: We tested API responses, we tested anonymous users, and we tested accounts that had been primed with different personas to see how much personalization actually changes the brands that AI recommends.
- 1:58SPEAKER_00: The answer is it does, but not necessarily in the way that you might expect it to.
- 2:03SPEAKER_00: Let's just get into it.
- 2:13SPEAKER_01: Cool.
- 2:13SPEAKER_01: So I'm Joan de Silva, I'm the co-founder of Friction AI.
- 2:17SPEAKER_01: So been working in this space for the last 12 months, and it all started because I am completely obsessed and passionate about AI in different uh levels.
- 2:30SPEAKER_01: It started with uh the lens of coding and how AI is helping like people uh coming from a more computer science background like myself, helping us code in a better, faster way.
- 2:43SPEAKER_01: And then evolved into a curiosity of trying to understand how AI works and why AI answers in certain ways.
- 2:51SPEAKER_01: And and that was pretty much the beginning of Friction AI, which is uh our startup, which basically uh is all around trying to understand uh how AI describes brands and potentially people as well in the future.
- 3:06SPEAKER_01: Um, and and that's pretty much how it all started, and it has been evolving over the last six months into understanding uh more and more how AI works, different models, different providers with the latest generations, and so on.
- 3:21SPEAKER_01: And uh yeah, that brings us today to today.
- 3:24SPEAKER_00: Yeah, so you reached out to me a couple months ago to tell me that you were running a couple different studies, and you mentioned this one in particular that we're about to start with.
- 3:33SPEAKER_00: What was the question that you wanted to find out?
- 3:37SPEAKER_01: I think uh uh there was curiosity in trying to understand whether what we were providing to our customers was exactly what their customers were seeing.
- 3:48SPEAKER_01: Um, I think obviously that was probably like the initial trigger, um, but as well to try to um you know to improve the platform and and to offer the best to our to our customers.
- 4:00SPEAKER_01: So I would say that's what triggered it.
- 4:02SPEAKER_01: Um so AI models do tend to uh change quite a bit depending on different parameters and uh who is asking and what model and so on.
- 4:13SPEAKER_01: So what we wanted to understand is um how much does what AI knows about their uh users influence the answers.
- 4:22SPEAKER_01: And so that's what triggered it.
- 4:24SPEAKER_01: And and then we started uh to uh set up the study, and and that's when I reached out uh to understand if this was also something that triggered curiosity for for you, and uh and uh that's how it started.
- 4:37SPEAKER_00: So, do you want to explain the experiment real quick before we get into the giggrity details and all the things that it means for marketing?
- 4:46SPEAKER_01: Yeah, so what we did was we looked into uh clouds, Gemini, ChatGPT, and perplexity.
- 4:55SPEAKER_01: And what we did is we set up the test in three different ways.
- 5:01SPEAKER_01: We uh had prompts being sent against those uh AI models via API, uh, so which is one form of technology of prompting the AI models.
- 5:13SPEAKER_01: We did as well send the same set of prompts uh against the same uh AI models but via uh uh the typical UI or web of those models in what we call an anonymous way.
- 5:27SPEAKER_01: So in uh in a setup where AI does not know the user.
- 5:32SPEAKER_01: And the third type that we did was we also prompted the same AI models, but with accounts that had been previously primed and given certain context so that AI knows a bit about those customers.
- 5:46SPEAKER_01: So the way we did that third uh type was we primed the accounts for two weeks.
- 5:52SPEAKER_01: So during two weeks, every night we would interact with uh with all those AI models uh by giving a bit away uh some information about the users.
- 6:03SPEAKER_01: So one of the personas we had created would be uh someone who lives in London, who has kids and uh who is 40 years old, who has certain uh income types and so on.
- 6:16SPEAKER_01: So we would give away certain characteristics about the persona, and for those two weeks uh we kept priming the persona.
- 6:22SPEAKER_01: So if you hear, if you hear saying about talk about priming, that's what it means.
- 6:26SPEAKER_01: We're just uh uh educating the AI about those those personas.
- 6:30SPEAKER_01: And so for those two weeks, we uh primed them, and then uh when we start the study, we started all the accounts at the same time.
- 6:38SPEAKER_01: So the API, the unprimed or anonymous um uh UI, and also as well the the primed uh users.
- 6:46SPEAKER_01: Uh for the primed users, we had selected together six different personas, uh, two personas per category.
- 6:54SPEAKER_01: Uh and uh and then basically what we did is we collected answers for I think around about two weeks from that point on, and then we compared the answers from those three different types uh at the end, and we uh understood the differences.
- 7:11SPEAKER_01: So it's quite complex, but we think it was a very robust and well thought through uh setup uh in order to understand basically what would be the difference between, like I said, API, uh anonymous and primed um responses.
- 7:27SPEAKER_00: Right.
- 7:28SPEAKER_00: So I was really interested in this study because I think it was Surfer months ago had mentioned that what the user was seeing with the API versus what a personalized account was seeing was different.
- 7:42SPEAKER_00: And so AI tools can't possibly measure all of this.
- 7:45SPEAKER_00: Before I ask my next next question, I also want to call out that we did the study for two weeks because models change very quickly, and like anything after that might not have been as accurate as it could have been based on the model differences.
- 7:58SPEAKER_00: Now, what did we find?
- 8:00SPEAKER_00: I will let you take the floor on that one.
- 8:03SPEAKER_01: Sure.
- 8:04SPEAKER_01: I think the results were, I would say, almost like what we were expecting, uh, in a way.
- 8:10SPEAKER_01: So we did see differences across the three of them, right?
- 8:13SPEAKER_01: We see we saw differences between uh API versus anonymous versus primed, and we also saw differences across the different models.
- 8:24SPEAKER_01: So uh, whenever we talk about personalization, I think rule number one is probably not generalize across all the models because the way Gemini behaves is very different from the way Claude behaves, and and I think this test helped us understand that.
- 8:37SPEAKER_01: So if we say that number one question, do AI models personalize?
- 8:43SPEAKER_01: Yes, but and the but is they all personalize in a different way.
- 8:48SPEAKER_01: Uh so what we saw is that while there was a set of brands that were consistent across the three uh types of uh collection of responses, so API and the primed and so on, um, there was always a change between a set of brands.
- 9:07SPEAKER_01: And what we saw is that around one brand always changed between the non-primed versus the primed responses.
- 9:18SPEAKER_01: However, certain models do this more than others.
- 9:22SPEAKER_01: And what we saw is that Claude does not uh seem to be very influenced by what it knows about the customer.
- 9:30SPEAKER_01: Same with perplexity, while ChatGPT and Gemini seem to be more influenced by what they know about the customer.
- 9:38SPEAKER_01: So we saw that um giving context and having context about the user influence the brands that they are recommending.
- 9:46SPEAKER_01: TPP, one brand out of five was different from between uh on Gemini and and uh and Chat GPT, while Claude uh did we did not see any difference.
- 10:50SPEAKER_01: Uh we saw some small uh variations, but not on the brands that were being mentioned.
- 10:55SPEAKER_00: I find that super interesting, and I know we talked about this when we were going through the study, but it makes a lot of sense with with Gemini and ChatGPT because those two models are really big on personalization.
- 11:06SPEAKER_00: Google with the personal personal intelligence that they've been pushing, like that makes a lot of sense.
- 11:12SPEAKER_00: But Claude, it's it's interesting that it just doesn't really seem to matter quite as much.
- 11:20SPEAKER_01: Yeah, Claude was a surprise.
- 11:22SPEAKER_01: What we did see is that the way Claude was researching or doing the web search when preparing the responses did change.
- 11:31SPEAKER_01: So it was localizing the questions more uh and and looking for sources that were more from the UK, given that we did the study was 100% in the UK.
- 11:41SPEAKER_01: I think I forgot to mention that.
- 11:42SPEAKER_01: Um, so Claude did localize the searches.
- 11:45SPEAKER_01: However, the brands that ended up recommending and mentioning did not end up changing.
- 11:51SPEAKER_01: Uh, so that that was an interesting aspect of it.
- 11:54SPEAKER_01: Um and that I was not expecting.
- 11:57SPEAKER_01: Like you said, Gemini, everyone expected Google knows so much about you, and and and it's one of their key uh selling points all the time is that we're gonna, you know, uh personalize the responses for you.
- 12:09SPEAKER_01: So I think that one was expected.
- 12:10SPEAKER_01: Uh Claude was definitely a surprise.
- 12:12SPEAKER_00: So I want to talk about the geography part for a second because I was talking with a brand the other day that mentioned something about can we show up in these answers if it's like a local super, super niche kind of thing.
- 12:25SPEAKER_00: And I think we saw that with the study, right?
- 12:28SPEAKER_01: Yeah, I I think what we saw is that um, so we used uh in order to collect the responses, we used only servers uh located in the UK.
- 12:38SPEAKER_01: And so that we would not have a biased uh collection.
- 12:42SPEAKER_01: And what we saw is that uh there was an increase, I believe up to 30%, if I'm not mistaken, of um of brands that are local brands being recommended and mentioned.
- 12:55SPEAKER_01: And yes, that's exactly what you're mentioning.
- 12:57SPEAKER_01: There were still global brands that they're such big players, and since they are still present in those markets, they were consistent.
- 13:04SPEAKER_01: And and those global brands are always going to be there.
- 13:07SPEAKER_01: Um it uh obviously it depends on the niche and on the on the category uh that your brands uh operate in.
- 13:14SPEAKER_01: So if we're talking about brands that um um or categories or niches that are very, very localized.
- 13:22SPEAKER_01: So if you're asking about the the best restaurant uh in a certain part, neighborhood of London, obviously a global brand will make uh you know no difference, right?
- 13:32SPEAKER_01: And so the categories that we selected for the study were categories that would include global brands.
- 13:38SPEAKER_01: And even then we saw roughly around 30% of the sources and the brands mentioned uh getting uh changed towards more local uh brands, uh, which I think just you know uh speaks for the point that you were making, that even if you're a smaller local brand, you still have a chance of showing up and the responses.
- 13:57SPEAKER_00: Right, right.
- 13:57SPEAKER_00: So when we're talking about localizing content, like let's talk about what this means for GEO, like your strategy all together.
- 14:05SPEAKER_00: So, does this change how brands should think about their authority and what they're doing when it comes to actually putting out content?
- 14:13SPEAKER_00: Um, because now for potentially optimizing for just a query, what's that look like now?
- 14:20SPEAKER_01: I think it depends a lot on the brands, and and I think that would be true for a lot of the answers is depends on your brand on and on your industry and and so on.
- 14:29SPEAKER_01: But I think if you're a brand that has a very high incidence on a certain market, I absolutely think and believe that it makes sense that your content gets more uh curated towards that market and that segment, obviously.
- 14:41SPEAKER_01: Uh, and that will influence a lot uh your you know the success uh or how often you get mentioned.
- 14:48SPEAKER_01: So yes, 100% content being localized uh makes a huge difference if your brand is operating in a more in that specific market or in a market that is more specific.
- 15:00SPEAKER_01: If it's a local brand, obviously it's gonna play less of a role, and your content strategy has to be a more uh you know globalized uh uh strategy.
- 15:09SPEAKER_01: So I would say it depends, but in in some cases, definitely makes a massive difference.
- 15:15SPEAKER_00: I I've been saying this for a while now that when it comes to figuring out your audience and how to show up inside of these AI answers, you really, really need to know who it is you're targeting, like to a T.
- 15:27SPEAKER_00: Like and I feel like this study really helped push that narrative just a bit more because personalization does matter if you are targeting Gemini and Chat GPT.
- 15:39SPEAKER_00: But um perplexity though, like I I know it wasn't much of a factor.
- 15:44SPEAKER_00: Can you tell us a little bit more about perplexity and personalization?
- 15:48SPEAKER_01: Yeah, so what we saw is we didn't see a tremendous change with perplexity.
- 15:53SPEAKER_01: I mean, I was just looking at the numbers.
- 15:55SPEAKER_01: It's one of those where uh there's nothing really that changed dramatically.
- 16:00SPEAKER_01: So we had put together as well a very interesting graph in our website where you can see the different characteristics of personalization of the different models, and and you will see that Claude uh pretty much didn't personalize anything like the recurring brands, the the brands that ranked number one, um the local citations and so on, but the search uh was was very different.
- 16:25SPEAKER_01: Uh what we saw for uh um for perplexity was that the the brands that were recurring they didn't change.
- 16:34SPEAKER_01: What changed a bit was the brand that was recommended number one, which typically plays a massive role.
- 16:40SPEAKER_01: So your average position being number one is quite important.
- 16:43SPEAKER_01: So perplexity did change a bit on that.
- 16:45SPEAKER_01: Um, and what we saw as well is that um the the answer was a bit more focused, so it had uh less brands on the uh the personalized.
- 16:56SPEAKER_01: So it's almost like um it gave uh unprimed, or when it doesn't know the users, it gave uh more brands as a response.
- 17:05SPEAKER_01: So let's say if you're asking what are the best uh gifts for a five-year-old, would give you like 10 brands in case it doesn't know you.
- 17:13SPEAKER_01: In case it does know you, it seems that it is uh being a bit more narrow in terms of the number of brands.
- 17:18SPEAKER_01: And we saw that uh around um there was a smaller number of brands that was being recommended with perplexity.
- 17:26SPEAKER_01: So it was more narrow in terms of the brands uh being recommended.
- 17:30SPEAKER_01: It would recommend imagine just five instead of the 10 uh when it knows you.
- 17:34SPEAKER_01: So that was the biggest difference with uh with perplexity.
- 17:38SPEAKER_01: Uh now perplexity is a model that it's not does not have its own native model and relies on top of other models, as you know.
- 17:46SPEAKER_01: So it also depends uh quite a bit on on what the models underneath are doing.
- 17:50SPEAKER_01: And it's also hard to understand like what's uh what additional uh layers do they add on top.
- 17:56SPEAKER_01: But that was the biggest change that we saw with perplexity was how narrow the answers were in terms of the number of brands mentioned.
- 18:03SPEAKER_00: Have you seen the news where Chat GPT is working?
- 18:05SPEAKER_00: I mean, it's rumored at this point, because we were recording in September, um, where they're working on their own index for lap, they're calling it Labrador.
- 18:13SPEAKER_00: Have you seen that?
- 18:14SPEAKER_00: I'm curious how this changes when that becomes a thing.
- 18:19SPEAKER_01: Yeah, yeah.
- 18:21SPEAKER_01: It's it's hard.
- 18:22SPEAKER_01: I mean, let's see if that's if that's true and if or if just something else that they're playing with it.
- 18:27SPEAKER_01: But yeah, it would it would be another ingredient and and it will make just uh harder for the brands who want to show up in all the different uh and the responses for all the different providers, it would just make it harder.
- 18:39SPEAKER_00: I would say, right, I agree with you.
- 18:41SPEAKER_00: So um what would you say to a company that looks at an AI visibility report and says, well, you're not appearing for these 50 prompts, so let's create 50 pieces of content.
- 18:50SPEAKER_00: Like, what do you what do you think about that?
- 18:52SPEAKER_00: Because I know when we talk about personalization and we're talking about, oh, you need to know your customer, and you need to know your customer, absolutely.
- 19:00SPEAKER_00: Let's just make all this content.
- 19:03SPEAKER_00: Should we be doing that?
- 19:05SPEAKER_01: I I don't think the answer is always more content.
- 19:08SPEAKER_01: Um, I think you need to understand first for which prompts you're not showing up.
- 19:13SPEAKER_01: And then that's question number one is trying to understand am I showing up for any of these prompts?
- 19:18SPEAKER_01: And by the way, that only you can only have the answer to that question if you do it in a repeated way.
- 19:24SPEAKER_01: Uh, it cannot be just one time, am I showing up or not?
- 19:28SPEAKER_01: Um so LLMs are not deterministic, they're probabilistic.
- 19:32SPEAKER_01: So we need to collect several samples in order to understand that.
- 19:35SPEAKER_01: I think everyone by now has understood that, but I think that's number one is where am I not showing up?
- 19:40SPEAKER_01: And then the second question is why am I not showing up?
- 19:43SPEAKER_01: Right.
- 19:43SPEAKER_01: And trying to understand what are where the queries that the LLM is doing in order to determine or to uh formulate a certain response and what are the uh what is the content that the LLM is using as sources and citations, and then trying to understand if you are already part of those or not, and what are the websites that are being cited?
- 20:04SPEAKER_01: Are those um first party websites or are those third-party?
- 20:08SPEAKER_01: Are those blogs?
- 20:09SPEAKER_01: Are those uh paid articles?
- 20:11SPEAKER_01: And and then from that point on, then you should start thinking of your strategy and where to invest.
- 20:17SPEAKER_01: And it could be that you're already being cited and uh so uh but not mentioned.
- 20:23SPEAKER_01: So uh, or your site could be used as a source, but not being mentioned, uh but your brand not being mentioned.
- 20:29SPEAKER_01: So I think it has to be you have to do an analysis trying to understand why you're not being uh mentioned as a brand.
- 20:36SPEAKER_01: And then if it is because you're lacking content in a certain certain area, then by all means go and create some additional content so that's uh your chances of getting uh mentioned are higher.
- 20:47SPEAKER_01: Uh, but it will be uh, I think you always need to do an analysis and understand why you're not being mentioned in the first place and what are what are the LLMs doing?
- 20:57SPEAKER_01: And they all do different things, they don't behave the same way, right?
- 21:00SPEAKER_01: It could be that you're being mentioned for ChatGPT but not for Gemini.
- 21:04SPEAKER_01: Uh, so what can you do uh on on uh in order to show up on on Gemini or what is Gemini uh mentioning in the first place?
- 21:12SPEAKER_01: So I don't think there is one recipe, and just saying create more content definitely is not what we recommend to brands.
- 21:20SPEAKER_01: What we do recommend is uh do your due diligence, try to understand what the root cause is, try to understand why you're not getting mentioned, and and then invest time either on improving your content or trying to collaborate with um third parties.
- 21:35SPEAKER_01: Maybe uh you can get uh mentioned on um some article that does a review of your brands.
- 21:42SPEAKER_01: Um, you know, so I think those those are uh you have different avenues to explore.
- 21:46SPEAKER_01: Uh, there's no one recipe and really depends on the root cause for not being mentioned.
- 21:51SPEAKER_00: Right.
- 21:51SPEAKER_00: So you just mentioned this being an investment.
- 21:55SPEAKER_00: And I know right now we're everyone in the brother are looking at different tools that they can use.
- 22:02SPEAKER_00: Um, some of them big investment.
- 22:05SPEAKER_00: Some of them are reasonable.
- 22:07SPEAKER_00: Um if I'm listening to this episode and I'm thinking, okay, well, personalization, it's makes a difference.
- 22:13SPEAKER_00: And I also read like the surfer study about the API being different, and I'm now kind of skeptical about looking at a tool.
- 22:21SPEAKER_00: What would you say to that person?
- 22:23SPEAKER_00: Because I still think measurement is super important.
- 22:26SPEAKER_00: But what do you say to that?
- 22:30SPEAKER_01: I I just think it plays uh even more um in or just re-emphasizes the importance of regular, consistent measurement.
- 22:41SPEAKER_01: Um, yes, LMs do personalize, we just found out that.
- 22:46SPEAKER_01: Uh, but we also found out that uh the gap is well big is not massive, right?
- 22:52SPEAKER_01: So I think the results that you're still measuring are relevant.
- 22:56SPEAKER_01: Um, and doing consistent um and regular measurements and not just relying on one single response um is even more important.
- 23:08SPEAKER_01: I think if there's one thing that we found out from this study is that um even though uh personalization does exist, the the methodology uh which platforms use as well plays a role, right?
- 23:22SPEAKER_01: So you have a lot of platforms that are using API, which we just saw in the study that gets close but not as close as when someone is using the actual UI, which is the cut what the customer sees.
- 23:33SPEAKER_01: So um I think choosing the right platform and ensuring they're using the right methodology, that they're using consistent and uh um regular ways of collecting the responses and then analyzing the responses over a period of time and not just relying on a single one.
- 23:51SPEAKER_01: Um I think it just speaks for you know, making sure you choose the right platform.
- 23:57SPEAKER_01: Uh, I I don't think it speaks against uh using a platform.
- 24:02SPEAKER_01: I think it's quite the contrary.
- 24:03SPEAKER_01: It's just like shows that uh doing things just once every now and then is not gonna give you an accurate representation of your AI visibility.
- 24:13SPEAKER_01: You need to measure consistently.
- 24:15SPEAKER_01: Um you need to see how your brand is performing over a period of time and not just over one day.
- 24:23SPEAKER_01: And and and platforms like uh AI visibility platforms, they they help you with that, right?
- 24:29SPEAKER_01: And and uh obviously they do come with the costs, but the cost is obviously most of the times because there's also like you know hidden costs behind it uh that's that require that uh that to be to be that way.
- 24:40SPEAKER_01: But I do think that the study shows that there is a need for a regular, uh, consistent uh collection and analysis.
- 24:49SPEAKER_00: So I have been ready to share about this study for a while now, and I think one of the biggest takeaways from all of this is that personalization changes AR responses.
- 25:00SPEAKER_00: Geography also changes them, the model you're using changes them.
- 25:04SPEAKER_00: Even the way you're accessing that model can change them.
- 25:07SPEAKER_00: But that doesn't make AI visibility measurement useless.
- 25:11SPEAKER_00: It just means that we have to stop treating a single response like a data point.
- 25:16SPEAKER_00: Which means that we need to measure repeatedly, we need to look for those patterns, and before we make a strategy decision based on any AI visibility tool, we need to understand how that tool is actually collecting its responses.
- 25:29SPEAKER_00: The goal is really to understand where your brain consistently appears, where it does it, and why.
- 25:35SPEAKER_00: If you want to dig into the study yourself, I've published a breakdown over on my website, including the methodology in the findings.
- 25:42SPEAKER_00: Friction AI also has their analysis on their website, and I'll post a link in the show notes to the actual study if you want to dig straight into the report.
- 25:51SPEAKER_00: And if you're wondering what all of this means for your own brand, that's exactly what I look at in my AI Search Visibility Audit.
- 25:57SPEAKER_00: I analyze where and how your brand appears across AI Search, what's influencing those answers, where the visibility gaps are, and what you should actually do to improve them.
- 26:07SPEAKER_00: You'll find the link to the audit in the show notes and all of that, all of the things that I have mentioned in the show notes as well.
- 26:14SPEAKER_00: That's it for today's episode.
- 26:16SPEAKER_00: I will see you in the next one.
- 26:18SPEAKER_00: Until then, stay visible.