Latest / Found in AI: AI Search Visibility, SEO, & GEO / ChatGPT’s Search Index, llms.txt Confusion & the Rise of AI Shopping

Transcript

ChatGPT’s Search Index, llms.txt Confusion & the Rise of AI Shopping · Found in AI: AI Search Visibility, SEO, & GEO

• 0:41
  SPEAKER_00: Hey, welcome back to Found in AI, the show where we talk about what's changing in AI search, what actually matters, and what marketers need to be paying attention to.
• 0:52
  SPEAKER_00: I have three news stories for you today, and they actually fit together pretty well.
• 0:57
  SPEAKER_00: First, we have new research from Peak AI, and it suggests that Chat GPT has built its own search index.
• 1:04
  SPEAKER_00: So no, apparently optimizing for being is not the magical shortcut to ChatGPT visibility that we've been told that it is.
• 1:13
  SPEAKER_00: Then Common Crawl, they analyzed more than half a million LLMS.txt files and found that, well, some of us may have gotten a little confused about what that file is actually supposed to do.
• 1:26
  SPEAKER_00: And finally, Microsoft is telling businesses to get ready for AI agents to shop on behalf of consumers.
• 1:33
  SPEAKER_00: Just like our good friend Google.
• 1:36
  SPEAKER_00: Hey, I'm Cassie Clark, an AI search visibility consultant.
• 1:40
  SPEAKER_00: Today is Thursday, September 10th.
• 1:42
  SPEAKER_00: Let's just get into it.
• 1:52
  SPEAKER_00: Let's start with Peak AI because this one is really fascinating.
• 1:56
  SPEAKER_00: And if you subscribe to my LinkedIn newsletter, you may have already seen me talk about this on Tuesday of this week.
• 2:03
  SPEAKER_00: Peak AI published research last week showing evidence that ChatGPT has its own search index.
• 2:10
  SPEAKER_00: For a while, one of the common assumptions about ChatGPT search has been that it relies heavily on traditional search engines, particularly Bing.
• 2:20
  SPEAKER_00: That has led to plenty of advice along the lines of, well, if you want to improve your visibility in ChatGPT, just make sure you're ranking and bing, you'll be fine.
• 2:31
  SPEAKER_00: Except Peak's research shows that the retrieval system is much more complicated than that.
• 2:37
  SPEAKER_00: Between May 21st and July 21st, Peak researchers found a field appearing in ChatGPT server-side events called Result Source, and one of the sources shown up was something called Labrador.
• 2:51
  SPEAKER_00: According to Peak, Labrador is OpenAI's own retrieval index.
• 2:56
  SPEAKER_00: Here's where it gets a little bit more interesting.
• 2:59
  SPEAKER_00: It doesn't appear to just be one big index.
• 3:03
  SPEAKER_00: Peak found evidence of separate Labrador indexes for things like the general web, PDFs, YouTube, news, Wikipedia, local information, and a couple other things.
• 3:18
  SPEAKER_00: And the news index is broken down even further, with separate indexes for content published within the last day, the last seven days, and older content, which, as someone who has been talking about freshness as a component for AI visibility for quite some time now, you know the FSA framework that I talk about a lot, that obviously caught my attention.
• 3:40
  SPEAKER_00: Peak says the Labrador Index sourced things like the full content of the page, the crawl date, and the publication date.
• 3:47
  SPEAKER_00: So freshness isn't just some abstract content marketing recommendation here.
• 3:52
  SPEAKER_00: We now have evidence that recency is literally represented within the retrieval infrastructure.
• 3:58
  SPEAKER_00: Now, this does not mean Google and Bing suddenly do not matter.
• 4:04
  SPEAKER_00: Actually, that's one of the most important parts of this research.
• 4:07
  SPEAKER_00: Peak found that ChatGPT is still pulling information from multiple outside providers.
• 4:12
  SPEAKER_00: In their investigation, found evidence connecting ChatGPT retrieval to Google, Microsoft's Web IQ, Yelp, Trif Advisor, and several other sources.
• 4:23
  SPEAKER_00: So please don't let the takeaway be here.
• 4:26
  SPEAKER_00: ChatGPT has its own index, forget Google and me.
• 4:29
  SPEAKER_00: That is absolutely not what the research shows.
• 4:33
  SPEAKER_00: What it shows is that ChatGPT Retrieval is becoming its own ecosystem.
• 4:39
  SPEAKER_00: OpenAI can crawl information itself.
• 4:41
  SPEAKER_00: It can store and cache that information.
• 4:44
  SPEAKER_00: It can use its own indexes, and then it can supplement all of that with information from those outside providers.
• 4:50
  SPEAKER_00: Which makes the idea that you can use Bing Rankings as a proxy for ChatGPT visibility a whole heck of a lot harder to defend.
• 4:59
  SPEAKER_00: I will say this until you are tired of hearing it.
• 5:02
  SPEAKER_00: Being indexed and ranking in traditional search engines still absolutely matters.
• 5:08
  SPEAKER_00: But the better question for AI visibility is becoming can that AI engine itself find, retrieve, understand, and reuse your information?
• 5:18
  SPEAKER_00: And there's one more piece of the Peak research that I want you to remember because it's going to connect directly to the Microsoft story later on in the show.
• 5:27
  SPEAKER_00: Shopping.
• 5:28
  SPEAKER_00: Peak found multiple shopping experiments running inside ChatGPT.
• 5:33
  SPEAKER_00: One was literally called Prefer Index over SERP version 3.
• 5:37
  SPEAKER_00: SERP, of course, meaning search engine result page.
• 5:41
  SPEAKER_00: Peak found that OpenAI was testing its own product index against search results with portions of real ChatGPT traffic.
• 5:49
  SPEAKER_00: And as of September 2nd, Peak says at least five shopping-related experiments were still running.
• 5:56
  SPEAKER_00: So OpenAI isn't just building an index for informational search, it's also actively testing how its own index performs when Chat GPT needs to find and rate products.
• 6:08
  SPEAKER_00: Keep that in the back of your mind because we are absolutely coming back to it.
• 6:12
  SPEAKER_00: Next up, we need to talk about that LLMS.txt file.
• 6:16
  SPEAKER_00: And this story is actually a really good reminder that we should probably stop treating every new technical thing associated with AI as the new robots.txt file.
• 6:27
  SPEAKER_00: Search Engine Journal covered a new analysis from CommonCrawl looking at 584,107 LLMS.txt files.
• 6:37
  SPEAKER_00: And there are some interesting findings.
• 6:40
  SPEAKER_00: 68% of the files Common Crawl analyzed originated from a plugin or template.
• 6:46
  SPEAKER_00: About 22% did not contain any links at all.
• 6:50
  SPEAKER_00: And then CommonCrawl found something even stranger.
• 6:54
  SPEAKER_00: Some websites are apparently using LLMS.txt to tell crawlers whether they're allowed to access the site.
• 7:01
  SPEAKER_00: But there's just one small problem with that.
• 7:04
  SPEAKER_00: That is not what this file does.
• 7:07
  SPEAKER_00: LLMS.txt is a proposed standard that essentially gives AI systems a curated list of content from a website.
• 7:15
  SPEAKER_00: It can help describe which pages you think are the most important.
• 7:18
  SPEAKER_00: It can also give machines a cleaner map of your content, but what it cannot do is grant or deny crawler access.
• 7:26
  SPEAKER_00: That's what robots.txt is for.
• 7:30
  SPEAKER_00: Common crawl even found more than 136,000 robots.txt files being served at the lms.txt path.
• 7:39
  SPEAKER_00: Which is, well, I don't really know what we're doing there, but we are definitely trying to do something.
• 7:45
  SPEAKER_00: They also found 32 LLMS.txt files that appeared to tell Common Crawl's CC bot that it was not allowed to crawl the site.
• 7:54
  SPEAKER_00: Common Crawl checked the actual robots.txt files for those particular sites, and of the 31 that they could actually assess, none completely blocked their bot.
• 8:06
  SPEAKER_00: One site even had an LLMS.txt file saying CC bot was blocked, while its robots.txt file actually permitted the crawler.
• 8:15
  SPEAKER_00: So the website was basically telling the crawler, you are not allowed here.
• 8:19
  SPEAKER_00: No bots allowed.
• 8:21
  SPEAKER_00: In a file the crawler isn't required to listen to, while the file it does actually listen to said, hey, come on in, take a seat.
• 8:30
  SPEAKER_00: Now, there really is a more serious point underneath all of this.
• 8:33
  SPEAKER_00: Common crawl is very careful to say that this analysis cannot tell us whether AI systems actually read the LLMS.txt files.
• 8:43
  SPEAKER_00: And I think that's important because I have seen this presented as almost mandatory for AI visibility.
• 8:49
  SPEAKER_00: We still do not have that evidence supporting that kind of claim.
• 8:52
  SPEAKER_00: And when you put this story immediately after the peak research, I really think it illustrates something much bigger.
• 9:00
  SPEAKER_00: Look at how complicated ChatGPT's actual retrieval infrastructure appears to be.
• 9:05
  SPEAKER_00: We're talking about crawlers, caches, multiple indexes, vertical indexes, outside search providers, licensed data sources, ranking systems, and retrieval experiments.
• 9:18
  SPEAKER_00: And then on the marketing side, we're still looking for one little text file that tells AI what to do.
• 9:24
  SPEAKER_00: That's not how this works.
• 9:26
  SPEAKER_00: I'm not saying don't use the lms.txt file.
• 9:30
  SPEAKER_00: If you want to provide machines with a clean, curated map of important content, fine, use it.
• 9:36
  SPEAKER_00: But please don't confuse making information easier for a machine to understand with controlling how that machine actually retrieves the information.
• 9:44
  SPEAKER_00: Those two things are not the same thing.
• 9:46
  SPEAKER_00: So please don't use lms.txt instead of robots.txt to control crawler access.
• 9:52
  SPEAKER_00: The crawler, I promise you, does not care what your lms.txt file says.
• 9:58
  SPEAKER_00: Now let's bring this back around to shopping.
• 10:01
  SPEAKER_00: I subscribe to Microsoft's product newsletter on LinkedIn, and they recently highlighted their guide for businesses preparing for Identitommer.
• 10:12
  SPEAKER_00: Microsoft's message is pretty straightforward here.
• 10:16
  SPEAKER_00: Businesses aren't just marketing to human shoppers anymore.
• 10:19
  SPEAKER_00: Pretty soon there's going to be an AI assistant or agent sitting between the customer and the business, and that really changes what it means to be discoverable.
• 10:29
  SPEAKER_00: Microsoft describes a shopping journey where someone can tell an AI assistant what they're looking for.
• 10:34
  SPEAKER_00: Maybe, for example, they need a gift under 50 for a niece who likes art.
• 10:39
  SPEAKER_00: Instead of opening 12 browser tabs, comparing the products, reading the reviews, checking prices, looking at social media, narrowing everything down, looking for the deal, the AI assistant can just do that work for you.
• 10:54
  SPEAKER_00: It can weigh the options and then hands the shopper a much smaller consideration set, saving the user time.
• 11:01
  SPEAKER_00: And eventually the agent can complete more of the transaction itself.
• 11:05
  SPEAKER_00: Microsoft says AI was the fastest growing source of retail traffic during last holiday season, growing 693% year over year.
• 11:15
  SPEAKER_00: They also cite Adobe data showing that AI referred visitors convert 42% better than non-AI traffic.
• 11:22
  SPEAKER_00: But the part of this guidance that I found most interesting isn't really the traffic numbers, is what Microsoft says businesses need to do about it.
• 11:32
  SPEAKER_00: Their framework basically comes down to three things.
• 11:36
  SPEAKER_00: One, get discovered.
• 11:38
  SPEAKER_00: Two, get chosen.
• 11:42
  SPEAKER_00: And the first part, getting discovered, starts with product data.
• 11:46
  SPEAKER_00: Microsoft argues that complete structured product data is becoming one of a business's most valuable marketing assets because those AI agents don't shop the way that people do.
• 11:58
  SPEAKER_00: People can respond to a beautiful product photo, clever copy, a brave storytelling, but an AI agent just needs facts.
• 12:06
  SPEAKER_00: And if that information is incomplete or outdated, Microsoft warrants that the product may not simply rank lower.
• 12:14
  SPEAKER_00: It just may not make the recommendation set at all.
• 12:18
  SPEAKER_00: That is a big difference.
• 12:20
  SPEAKER_00: That is a big difference because now we're not just talking about AI visibility, we're also talking about AI readiness.
• 12:26
  SPEAKER_00: Microsoft is already building the pieces of that experience and putting it all together for us.
• 12:32
  SPEAKER_00: They're currently talking about AI ready product feeds through Microsoft Merchant Center.
• 12:37
  SPEAKER_00: They're also talking about advertising across being in co-pilot.
• 12:40
  SPEAKER_00: They're talking about copilot checkouts so shoppers can purchase without leaving the conversation.
• 12:46
  SPEAKER_00: They're talking about brand agents on company websites.
• 12:49
  SPEAKER_00: And they also have AI visibility reporting through Microsoft Clarity.
• 12:53
  SPEAKER_00: So when Microsoft talks about agenda commerce, and when Google talks about it, this isn't some prediction for 2035.
• 13:02
  SPEAKER_00: They are actively working on the infrastructure right now.
• 13:06
  SPEAKER_00: And this is why I wanted to cover these three stories together.
• 13:09
  SPEAKER_00: I think they tell us something much bigger about where AI Search is heading.
• 13:13
  SPEAKER_00: Peak's research shows us that OpenAI is building more of its own retrieval infrastructure.
• 13:19
  SPEAKER_00: Common Crawl's research shows us why we can't reduce AI optimization to one technical file or really one supposed hack.
• 13:27
  SPEAKER_00: And Microsoft, bless them, they are showing us what happens when AI discovery starts turning into AI action.
• 13:33
  SPEAKER_00: The job here simply isn't getting a web page to rank anymore.
• 13:38
  SPEAKER_00: Businesses need to make their information retrievable.
• 13:41
  SPEAKER_00: They need to make it understandable.
• 13:43
  SPEAKER_00: They need to make it trustworthy, and they need to keep it current.
• 13:47
  SPEAKER_00: And as agents become capable of doing more on a user's behalf, that information also needs to become actionable.
• 13:55
  SPEAKER_00: I've talked about there being two layers of AI readiness.
• 13:59
  SPEAKER_00: The first is visibility, we talk about it quite often.
• 14:02
  SPEAKER_00: Can the AI engine find you, understand you, and recommend you?
• 14:06
  SPEAKER_00: The second is AI agent readiness.
• 14:09
  SPEAKER_00: Once an AI engine finds you, can it actually do something with the information that you have provided?
• 14:16
  SPEAKER_00: And we're starting to see the infrastructure for both layers being built at the same time.
• 14:22
  SPEAKER_00: That's why I think businesses that wait for AI search to settle down before doing anything, that's why I think they're gonna have a much harder job later.
• 14:31
  SPEAKER_00: Absolutely everything about AI search optimization and AI readiness is going to change at some point.
• 14:38
  SPEAKER_00: That Labrador index peak documented will change.
• 14:41
  SPEAKER_00: How ChatGPT retrieves information is also going to change.
• 14:45
  SPEAKER_00: Agentic commerce will change, but the direction is becoming much, much harder to ignore.
• 14:52
  SPEAKER_00: These AI engines are moving from answering questions to helping people make decisions to now taking actions based on those decisions.
• 15:00
  SPEAKER_00: And I think businesses need to be ready to participate at every single step.
• 15:05
  SPEAKER_00: That's it for today's Found in AI news update.
• 15:09
  SPEAKER_00: I'll link the peak of AI research, the common call coverage from Search Engine Journal, and Microsoft's Agentic Commerce Guide in the show notes in case you want to read them.
• 15:17
  SPEAKER_00: And if you're wondering whether your business is actually prepared to show up in these AI discovery experiences, that's exactly what I look for in the AI Search Visibility Audit.
• 15:27
  SPEAKER_00: We look at where your brand appears today, what sources and signals are shaping those answers, where the gaps are, and what your team should actually prioritize instead of chasing every new GEO tactic that shows up on LinkedIn or Reddit or wherever you get your information.
• 15:43
  SPEAKER_00: You can learn more at CassieClarkmarketing.com to get started.
• 15:46
  SPEAKER_00: And as always, if this episode helped you make sense of what the heck is actually happening in AI Search this week, follow Foundin AI wherever you're listening.
• 15:54
  SPEAKER_00: I would love it if he would give it a rating.
• 15:57
  SPEAKER_00: I will see you next time.
• 15:58
  SPEAKER_00: Until then, stay visible.