Latest / Found in AI: AI Search Visibility, SEO, & GEO / Bing's New AI Visibility Data, ChatGPT Citation Shifts & the Attribution Problem

Transcript

Bing's New AI Visibility Data, ChatGPT Citation Shifts & the Attribution Problem · Found in AI: AI Search Visibility, SEO, & GEO

• 0:41
  SPEAKER_00: Welcome back to Found in AI, the podcast where we talk about AI search visibility, GEO, AEO, and what all of these changes actually mean for marketers.
• 0:52
  SPEAKER_00: I am Cassie Clark, an AI search visibility consultant, and we've got quite a bit to talk about today.
• 0:58
  SPEAKER_00: Bing has released another update to its AI performance reporting, and this one is particularly interesting if you are responsible for content strategy.
• 1:08
  SPEAKER_00: We've also got new research from Href suggesting ChatGPT is shifting towards citing more primary sources.
• 1:15
  SPEAKER_00: We have some pretty wild findings from Nozzle about what Google might be serving to rank tracking tools, and a potential attribution problem with ChatGPT that's going to make measuring AI search traffic that much more complicated.
• 1:31
  SPEAKER_00: There's a common thread between all of these stories today.
• 1:34
  SPEAKER_00: We're finally getting more information about how brands appear in search, AI search particularly, but at the same time we're seeing just how difficult it is to know whether the numbers we're looking at actually reflect what's actually happening.
• 1:48
  SPEAKER_00: So let's get into it.
• 1:58
  SPEAKER_00: Okay, first up, Bing.
• 2:01
  SPEAKER_00: On October 13th, Microsoft announced a new beta feature in Bing Webmaster Tools called Topics Portfolio.
• 2:08
  SPEAKER_00: And I have to say I really, really like the direction Bing is going with its AI search reporting.
• 2:13
  SPEAKER_00: If you've been listening to the podcast for a hot minute, you know I am a Microsoft fan.
• 2:18
  SPEAKER_00: I feel like they are the pioneers in AI search measurement at least.
• 2:23
  SPEAKER_00: And if you've been listening to the podcast for a while, you know I've talked about the limitations of trying to measure AI visibility using traditional SEO metrics.
• 2:33
  SPEAKER_00: Rankings don't tell us whether an AI engine understands your brand.
• 2:37
  SPEAKER_00: Traffic doesn't tell us how often your brand is being recommended, and a single screenshot of your company appearing in Chat GPT or an AI overview or whichever platform you're looking at isn't exactly a measurement strategy.
• 2:53
  SPEAKER_00: That's why I've been paying attention to what Bing is doing.
• 2:56
  SPEAKER_00: Earlier this year, Bing introduced its AI performance reporting.
• 3:00
  SPEAKER_00: I have talked about it here on the podcast and on LinkedIn a little bit.
• 3:04
  SPEAKER_00: This tool gives website owners information about the grounding queries where their content appears and which pages are being cited.
• 3:13
  SPEAKER_00: Now, the new addition with this beta feature topics portfolio takes it a step further.
• 3:20
  SPEAKER_00: Instead of looking at individual grounding queries, Bing groups these queries into broader topics.
• 3:26
  SPEAKER_00: Let's say your company publishes content about cybersecurity.
• 3:30
  SPEAKER_00: Rather than looking at a dozen of individual queries in isolation, you might be able to see the broader topics where your website is appearing in AI-generated answers.
• 3:40
  SPEAKER_00: So within each topic, Bing provides several metrics.
• 3:44
  SPEAKER_00: There's topic size, which estimates the total citations across websites for that particular topic.
• 3:51
  SPEAKER_00: There's sites cited, which gives you an idea of how many websites are appearing.
• 3:57
  SPEAKER_00: And there's a count of distinct grounding queries.
• 4:00
  SPEAKER_00: You can also see where your own website is being cited as well.
• 4:04
  SPEAKER_00: But the feature that I am most interested in is opportunity queries.
• 4:10
  SPEAKER_00: These are queries where other websites are appearing as sources, but yours is not.
• 4:15
  SPEAKER_00: And that is where this starts to become useful for content strategy.
• 4:20
  SPEAKER_00: Because now instead of asking, well, hey, which website, or which keywords not website, which keywords should we write about next?
• 4:27
  SPEAKER_00: You can start asking, what topics are AI engines already using sources for, and where are we missing from that conversation?
• 4:35
  SPEAKER_00: I think that's a different way to think about content gaps.
• 4:38
  SPEAKER_00: Now, there are a few important caveats we need to talk about.
• 4:42
  SPEAKER_00: First, this is a beta feature.
• 4:44
  SPEAKER_00: Bing says the classifications are AI generated, and the numbers are just really an estimate.
• 4:50
  SPEAKER_00: So take these as directional signals, not just exact measurement numbers.
• 4:55
  SPEAKER_00: Second, you need enough data to access the feature.
• 4:58
  SPEAKER_00: So if your website doesn't have sufficient presence, you might not see topics portfolio at all if you're looking into this tool.
• 5:06
  SPEAKER_00: And third, these are grounding queries.
• 5:08
  SPEAKER_00: So they're not necessarily the exact prompts people typed into the chat bot to, you know, get the results that you're looking at.
• 5:15
  SPEAKER_00: So all of that matters, keep all of that in mind.
• 5:18
  SPEAKER_00: But even with those limitations, I think Bing is giving marketers something we've been asking for, which is just a better understanding of how our content participates in AI generated answers.
• 5:29
  SPEAKER_00: And I would really love to see other platforms move into this direction.
• 5:34
  SPEAKER_00: Google Search Console, I am talking about you.
• 5:37
  SPEAKER_00: Because right now, so much AI visibility measurement depends on third-party tools trying to reconstruct what's happening.
• 5:44
  SPEAKER_00: Direct platform data doesn't solve every problem, but it gives us another signal to work with.
• 5:50
  SPEAKER_00: So my advice on what to do with this is pretty simple.
• 5:53
  SPEAKER_00: If you're already using Bing Webmaster Tools, go see whether topics portfolio is available for your website, look at the topics you're getting cited at, look at those opportunities, and then compare that information with what you're seeing in your own AI visibility monitoring.
• 6:09
  SPEAKER_00: I wouldn't use it to make a content strategy decision by itself, just compare.
• 6:15
  SPEAKER_00: But as another source of information, absolutely useful.
• 6:20
  SPEAKER_00: Alright, let's move on to the second story.
• 6:23
  SPEAKER_00: Next up, we've got Sindy Research from Hrefs covered by Search Engine Journal, and this one really caught my attention.
• 6:30
  SPEAKER_00: Href analyzed the websites ChatGPT cited in millions of US responses between March and September.
• 6:37
  SPEAKER_00: And what they found suggests that ChatGPT is citing original sources more often rather than websites that summarize information from just somewhere else.
• 6:50
  SPEAKER_00: One example is PubMed.
• 6:53
  SPEAKER_00: In March, PubMed ranked 64th among the most frequently cited domains in the study, but by September, it scurried on up to 4th.
• 7:04
  SPEAKER_00: Meanwhile, WebMD dropped from 18th to 79th.
• 7:09
  SPEAKER_00: The research also found gains for government agencies, regulators, and websites that publish hands-on product testing.
• 7:19
  SPEAKER_00: Meanwhile, several major health and finance explainer websites, well, they lost ground.
• 7:26
  SPEAKER_00: Now, before we all go running off to declare that Chat DPT has completely changed its citation preferences, I want to point out something important about the methodology here.
• 7:37
  SPEAKER_00: Href compared 2.4 million responses in March with 3.8 million in September.
• 7:44
  SPEAKER_00: Those are not necessarily identical sets of prompts.
• 7:49
  SPEAKER_00: And so some of those changes could reflect differences in what people are asking.
• 7:54
  SPEAKER_00: But the tool also looked at approximately 25,000 prompts that appeared in both periods and the direction of the citation changes held up in that comparison.
• 8:04
  SPEAKER_00: So there's that.
• 8:05
  SPEAKER_00: And I think there's a pretty important lesson here for content marketers.
• 8:09
  SPEAKER_00: If your entire content strategy consists of rewriting information that's already available from more authoritative sources, what exactly are you giving an AI engine that it can't get somewhere else?
• 8:22
  SPEAKER_00: Think about it.
• 8:23
  SPEAKER_00: If you are publishing a blog post that summarizes a government of report, why wouldn't an AI engine just, you know, cite the government report?
• 8:31
  SPEAKER_00: And if you're writing about a scientific study, why wouldn't it cite the actual research?
• 8:38
  SPEAKER_00: This is why I keep coming back to original research first-hand experience and building actual authority around the subjects your brain wants to be known for.
• 8:47
  SPEAKER_00: That doesn't mean that exploratory content is suddenly useless.
• 8:51
  SPEAKER_00: There's still value in that, and there's still value in making complicated subjects understandable.
• 8:57
  SPEAKER_00: There is still value in connecting ideas and adding some context, supporting context to an original source that might not be included in that original source.
• 9:07
  SPEAKER_00: But there needs to be something distinctive about what you're publishing.
• 9:11
  SPEAKER_00: Otherwise, you're competing with the source you got your information from in the first place.
• 9:15
  SPEAKER_00: Now here's another finding from the study that I also want to mention.
• 9:19
  SPEAKER_00: Reddit citations, we're talking about Reddit again, because when are we never not talking about Reddit?
• 9:24
  SPEAKER_00: Reddit citations dropped significantly.
• 9:27
  SPEAKER_00: In March, Reddit appeared as a citation in 19.1% of the US Chat GPT answers that they analyzed, and by September, that answer that's sorry, that number was 2.3%.
• 9:43
  SPEAKER_00: Wikipedia also saw a pretty substantial decline, going from 14.1% to 2.9%.
• 9:50
  SPEAKER_00: Now I find all of that pretty interesting because I have covered some other Reddit research that talks about how ChatGPT uses Reddit in answers now.
• 10:00
  SPEAKER_00: In one of those studies, Reddit appeared in 84 of the 221 sources that ChatGPT retrieved, but it wasn't cited in the final answer.
• 10:10
  SPEAKER_00: So retrieval is not the same thing as citation.
• 10:13
  SPEAKER_00: Just because an AI system looks at a source, that doesn't mean that source is going to appear in the answer.
• 10:20
  SPEAKER_00: And just because Reddit citations are declining doesn't necessarily mean that Reddit has stopped influencing what ChatGPT recommends.
• 10:28
  SPEAKER_00: I still think there is a role for Reddit as an opinion layer, especially when people are asking questions about products, services, or, you know, real world experiences.
• 10:39
  SPEAKER_00: But we need to be careful about treating citation frequency as a complete picture of how a source influences an answer.
• 10:47
  SPEAKER_00: To me, at least, the bigger takeaway for this research is that brands need to give AI engines a reason to use their content.
• 10:54
  SPEAKER_00: You know, things like original data, actual expertise, information that comes directly from your company that can't come from anywhere else.
• 11:03
  SPEAKER_00: Not just another article that says exactly what every other article says.
• 11:08
  SPEAKER_00: Now, if you have been listening to me talk about authority in AI Search for the last year, you probably already know where I stand on this one.
• 11:16
  SPEAKER_00: Alright, the next story is a little wild.
• 11:19
  SPEAKER_00: We talked about this on LinkedIn yesterday, so if you follow me over there, and I hope you will connect with me if you're listening to the podcast.
• 11:27
  SPEAKER_00: This might be a repeat, but I have to talk about it now.
• 11:30
  SPEAKER_00: Nozzle published research on October 10th suggesting that Google has been serving what Nozzle calls poisoned search results to automated rank tracking tools.
• 11:41
  SPEAKER_00: Now, before I get into the findings, I want to be very clear about what we're talking about here.
• 11:46
  SPEAKER_00: The research is about search engine results pages collected by automated tracking systems.
• 11:52
  SPEAKER_00: This is not evidence that ordinary Google users are seeing fake search results.
• 11:58
  SPEAKER_00: Keep that in mind because I think that's pretty important.
• 12:01
  SPEAKER_00: So here is what Nozzle found.
• 12:03
  SPEAKER_00: The company analyzed 27.5 million Google search results across, like the search result pages, we should probably say, across 49 countries.
• 12:14
  SPEAKER_00: Beginning in September, it started to pick up on something a little wonky.
• 12:18
  SPEAKER_00: Some of those results returned to its tracking systems looked pretty normal.
• 12:23
  SPEAKER_00: The pages had the usual Google formatting, there wasn't an error message or a captcha, but the websites appearing in the results were dramatically different from the ones that normally appeared for those searches.
• 12:37
  SPEAKER_00: Nozzle classified about 8% of the results it examined since September 13th as, and I quote, poisoned, compared with a baseline of just 0.17%.
• 12:49
  SPEAKER_00: And on some days that percentage was substantially higher.
• 12:53
  SPEAKER_00: Here's where it gets interesting though.
• 12:56
  SPEAKER_00: For keywords where a tracked website would normally appear in the top 10, that website was missing from the top 10 in nearly 59% of the poison results.
• 13:07
  SPEAKER_00: Imagine opening your ranking report and seeing that a bunch of your most important keywords have suddenly dropped.
• 13:14
  SPEAKER_00: Now, if you were like me, you might panic a little.
• 13:18
  SPEAKER_00: You start looking for those technical issues, and then maybe you might start thinking, oh hey, did Google roll out another spam update?
• 13:26
  SPEAKER_00: And did we get hit by that?
• 13:28
  SPEAKER_00: But then, a couple of days later, your rankings are back.
• 13:33
  SPEAKER_00: Now, according to Nozzle, nearly 78% of these unusual results return to their usual set of websites within three days.
• 13:42
  SPEAKER_00: Nozzle seems to believe that this may be related to Google's efforts to limit automated scraping, including scraping used to collect information for AI systems.
• 13:53
  SPEAKER_00: Now I want to emphasize that this is Nozzle's interpretation of the data.
• 13:58
  SPEAKER_00: We do not have confirmation from Google that this is an intentional anti-scraping tactic.
• 14:04
  SPEAKER_00: But what does this mean for marketers?
• 14:07
  SPEAKER_00: Well, first, it means we need to be careful about reacting to sudden changes in our third-party ranking reports.
• 14:13
  SPEAKER_00: So if you see something wonky, give it time, it might settle.
• 14:17
  SPEAKER_00: But also compare that with your Google Search Console results.
• 14:21
  SPEAKER_00: And there's a broader point here that connects directly to AI visibility measurement.
• 14:27
  SPEAKER_00: A lot of the tools that we use depend on collecting information from platforms that don't necessarily provide direct access to everything that's happening behind the scenes.
• 14:38
  SPEAKER_00: When that collection process changes, the data changes.
• 14:43
  SPEAKER_00: And sometimes that has absolutely nothing to do with whether your website has become more or less visible to actual users.
• 14:51
  SPEAKER_00: That doesn't mean that we should stop using ranking tools, it just means we need to understand what they're measuring and validate unusual results before making major decisions.
• 15:02
  SPEAKER_00: And I talked about this more on LinkedIn.
• 15:05
  SPEAKER_00: If those unusual results are pulling in social media pages or posts more often, I think there's a discussion worth having about what we are actually putting out on social media, comparing that to the larger content strategy.
• 15:22
  SPEAKER_00: Everything is connected.
• 15:24
  SPEAKER_00: I'm gonna say that forever, I think.
• 15:26
  SPEAKER_00: Everything is connected.
• 15:28
  SPEAKER_00: Now, this brings me to our final big story.
• 15:31
  SPEAKER_00: ChatGPT may be breaking AI traffic attribution.
• 15:36
  SPEAKER_00: I think this is gonna be one of the most frustrating ones for anyone trying to prove the value of AI search visibility.
• 15:43
  SPEAKER_00: Search Engine Roundtable reported on October 14th that ChatGPT appears to have stopped passing some of that tracking information that previously helped website owners identify visits coming from ChatGPT.
• 15:56
  SPEAKER_00: Specifically, we're talking about the UTM parameters and referral information.
• 16:01
  SPEAKER_00: Earlier this year, OpenAI began adding to the tracking parameters to some links in ChatGPT.
• 16:08
  SPEAKER_00: That was really helpful because it gave marketers another way to identify visits coming from that particular platform.
• 16:15
  SPEAKER_00: But according to reporting from very sorts, something appears to have changed with ChatGPT's GPT 6.
• 16:23
  SPEAKER_00: Cedric Usler from Adobe, and I probably just pronounced his name wrong, so Cedric, if you were listening, I am so sorry.
• 16:30
  SPEAKER_00: But anyway, Cedric noticed that links were being opened without the previous Chat GPT UTEM parameter and without the referral information that would normally identify where the visit came from.
• 16:42
  SPEAKER_00: So, what happens?
• 16:44
  SPEAKER_00: A person clicks a link in ChatGPT, they land on your website, and now your analytics platform may classify that visit as direct traffic.
• 16:53
  SPEAKER_00: Not chat GPT traffic, direct.
• 16:57
  SPEAKER_00: Now here's the part that I found interesting.
• 17:00
  SPEAKER_00: CedRet looked at approximately 500 pages where ChatGPT had previously accounted for at least 5% of visits.
• 17:09
  SPEAKER_00: Comparing October 9th through the 13th with the previous week, ChatGPT attributed visits fell by about 31,000 per day.
• 17:19
  SPEAKER_00: But direct visits to those same pages increased by approximately 26,000 per day.
• 17:25
  SPEAKER_00: Now, that doesn't prove every missing ChatGPT visit became a direct visit, but it certainly raises questions about whether we're seeing an actual traffic decline or an attribution problem.
• 17:38
  SPEAKER_00: And this is exactly why I have been saying that AI referral traffic cannot be your only measurement of AI search visibility.
• 17:46
  SPEAKER_00: Because here's the problem.
• 17:48
  SPEAKER_00: Let's say you're a marketing director and you have spent the last six months investing in GEO.
• 17:54
  SPEAKER_00: You and your team have improved your content, you've built more authority around your brand, you're showing up more frequently in AI-generated answers, and then you can open your analytics dashboard and see chat GPT traffic falling.
• 18:10
  SPEAKER_00: Does that mean that your strategy stopped working?
• 18:13
  SPEAKER_00: Well, not necessarily.
• 18:15
  SPEAKER_00: But it might mean the platform changed how it sends visitors to your website.
• 18:19
  SPEAKER_00: And if you're only looking at that referral traffic number, you could reach a completely incorrect conclusion about your performance.
• 18:27
  SPEAKER_00: We also don't know whether this change was intentional or how consistently it affects different Chat GPT experiences.
• 18:35
  SPEAKER_00: So, I don't think I would jump to conclusions here, but I would be checking my analytics.
• 18:41
  SPEAKER_00: If you've seen a sudden decline in ChatGPT referrals over the last week, look at your direct traffic over the same period.
• 18:49
  SPEAKER_00: Then go compare those changes with what you're seeing in your AI visibility monitoring.
• 18:54
  SPEAKER_00: Because we have to stop treating one single metric as the whole story.
• 19:00
  SPEAKER_00: A brand can be mentioned in an AI answer without being cited.
• 19:04
  SPEAKER_00: It can be cited without getting that click.
• 19:07
  SPEAKER_00: And apparently, it can get a click without that visit being attributed to the AI platform that sent it.
• 19:14
  SPEAKER_00: That's a pretty messy measurement problem that we have going on here.
• 19:19
  SPEAKER_00: And until we have better reporting directly from the platforms, we're going to have to keep pacing together multiple signals to help us understand what's actually happening.
• 19:29
  SPEAKER_00: Okay, now before we wrap up, one quick update from Yahoo.
• 19:33
  SPEAKER_00: According to Search Engine Watch, Yahoo is testing a search box underneath its AI overview that allows people to continue their searches in Yahoo Scout.
• 19:43
  SPEAKER_00: It's also testing related questions that send users to Scout when clicked.
• 19:47
  SPEAKER_00: Now, this is not a massive announcement on its own, but I do think it's worth mentioning here because we're seeing a similar pattern across various search engines.
• 19:58
  SPEAKER_00: Instead of giving users an answer and then sending them somewhere else, platforms are creating more opportunities to keep the next question inside of their own AI experiences.
• 20:09
  SPEAKER_00: We're seeing this with Google's AI mode.
• 20:11
  SPEAKER_00: Now Yahoo is experimenting with it through Scout.
• 20:15
  SPEAKER_00: And for marketers, it's just another reminder that the customer journey is becoming less linear.
• 20:21
  SPEAKER_00: Someone might start with a traditional search, they read an AI-generated answer, maybe they ask a follow-up question, maybe they compare several options, and they never visit a website until much, much later in the process, which makes understanding how your brand appears throughout that journey even more important.
• 20:43
  SPEAKER_00: So what do we make of all of this?
• 20:46
  SPEAKER_00: Well, Bing is giving us more ways to understand where our content appears in AI-generated answers.
• 20:53
  SPEAKER_00: Hrefs is showing us that sources chat GPT sites can change pretty dramatically over time.
• 21:01
  SPEAKER_00: Nozzle is raising questions about the reliability of data collected from automated search results.
• 21:07
  SPEAKER_00: And ChatGPT may have just made referral attribution a whole heck of a lot messier than what it was.
• 21:14
  SPEAKER_00: I think the lesson here is that we need to get much more comfortable working with imperfect information.
• 21:21
  SPEAKER_00: Not because measurement doesn't matter, it absolutely does, but because there isn't one dashboard, one visibility score, or one traffic report that's going to tell you everything you need to know about your brand's performance in AI search.
• 21:36
  SPEAKER_00: One thing you can do though is understand whether AI engines recognize your brand, whether you're appearing in relevant answers, which sources they're using, and whether those appearances are contributing to actual business outcomes.
• 21:50
  SPEAKER_00: And yeah, you're gonna need to measure those things over time.
• 21:53
  SPEAKER_00: That's just how you start separating meaningful changes from the noise, but we don't have a perfect system for that yet.
• 22:00
  SPEAKER_00: Alright, that's what I've got for you this week.
• 22:02
  SPEAKER_00: If you have enjoyed this episode, make sure you are following Found in AI wherever you listen to your podcast.
• 22:08
  SPEAKER_00: And if you want more of these updates, along with my research and thoughts on what's happening in AI Search, subscribe to the Visibility Report.
• 22:15
  SPEAKER_00: I will put the link in the show notes.
• 22:18
  SPEAKER_00: And one gentle reminder, I am posting a free live training on November 5th at 1 p.m.
• 22:24
  SPEAKER_00: Easter time.
• 22:25
  SPEAKER_00: You can find more information about that in the show notes as well.
• 22:28
  SPEAKER_00: As always, thank you for listening.
• 22:30
  SPEAKER_00: I will see you in the next one.
• 22:32
  SPEAKER_00: Stay visible.