Latest / Wpfavs - Your favourite WordPress news / llms.txt Hype vs. AI Agent Reality

Transcript

llms.txt Hype vs. AI Agent Reality · Wpfavs - Your favourite WordPress news

• 0:13
  So everybody has been talking about Generative Engine Optimization,
• 0:16
  or GEO, like it is the shiny new gold rush.
• 0:20
  Agencies are out here charging thousands of dollars to build this single file,
• 0:24
  llms dot txt, telling site owners it is the secret key to ranking number one on
• 0:29
  ChatGPT or Perplexity.
• 0:31
  Wait, wait, but, but is it not?
• 0:33
  I mean, everyone is saying that file is basically the new sitemap for AI crawlers.
• 0:37
  Well, that is the hype.
• 0:39
  But if you look at actual server data, out of over five hundred million LLM crawler
• 0:43
  visits analyzed in 2026, search bots like GPTBot or PerplexityBot fetched that file
• 0:50
  exactly four hundred and eight times.
• 0:53
  Four hundred and eight times out of five hundred million!
• 0:55
  Four hundred and eight times!
• 0:57
  That is, uh, that is practically zero.
• 0:59
  It is a rounding error!
• 1:00
  Exactly.
• 1:01
  And SE Ranking ran a study on three hundred thousand domains and found absolutely
• 1:06
  zero statistical correlation between having an llms dot txt file and getting cited
• 1:12
  more often in AI search responses.
• 1:14
  In fact, John Mueller over at Google outright called it the modern equivalent of
• 1:19
  meta keywords.
• 1:20
  Ouch.
• 1:20
  So search engine engineers are saying, look, our crawlers just read the raw HTML
• 1:24
  directly, stop overcomplicating it.
• 1:26
  Right.
• 1:28
  But, but here is where people get confused, because the file is not totally useless.
• 1:33
  It is just being completely sold for the wrong job.
• 1:36
  It is not for search engines.
• 1:38
  It is for Business to Agent routing.
• 1:40
  Right, right, exactly!
• 1:42
  When you use developer tools like Cursor or Claude Code, or LangChain with their
• 1:47
  mcpdoc server, those coding agents fetch the Markdown file directly.
• 1:51
  Why?
• 1:52
  Because they do not want to parse two megabytes of HTML headers,
• 1:56
  scripts, and navigation menus.
• 1:58
  They just want clean, raw documentation.
• 2:01
  It cuts token overhead by ten times!
• 2:03
  Ah, so it is, it is an agent navigation file for actual task execution,
• 2:08
  not a marketing gimmick for search ranking.
• 2:10
  Precisely.
• 2:11
  But even if you do want to set it up for agents, WordPress site owners are shooting
• 2:15
  themselves in the foot.
• 2:17
  Plugins like Yoast SEO and Rank Math added these one click toggles to generate the
• 2:21
  file, which is great.
• 2:24
  But then site owners leave their robots dot txt file configured to block user agents
• 2:29
  like ClaudeBot or GPTBot from accessing the root directory in the first place.
• 2:34
  They block the very bots at the door while hanging a welcome sign on the window!
• 2:38
  I, I see this all the time.
• 2:39
  And that really brings us to the core issue of how we build WordPress sites today.
• 2:44
  Most themes are designed for human eyeballs, packed with heavy page builders,
• 2:48
  DOM clutter, popups, and scripts that completely exhaust an AI agent's context
• 2:53
  window.
• 2:53
  That is why we built the free AI Content Reader at GetAISO,
• 2:57
  to strip away all CSS and JavaScript and show webmasters what an AI assistant
• 3:02
  actually sees when it visits their page.
• 3:04
  And when you strip away the pretty visual layer, a lot of sites look like a jumbled
• 3:09
  mess of navigation links and consent banners before you even get to the actual
• 3:13
  content, right?
• 3:14
  Mm, completely.
• 3:16
  It is pure noise to an LLM.
• 3:18
  But when you clean up that site architecture, incredible things happen.
• 3:22
  Benjamin Intal, founder of Stackable and WP Interactions, shared Ahentic AI
• 3:28
  workspace experiment recently.
• 3:30
  And it shows how an AI agent can understand an entire WordPress site,
• 3:34
  help launch it, and orchestrate plugin setups locally.
• 3:37
  Wait, with, with no external servers?
• 3:39
  No third party MCP bridge?
• 3:41
  None at all!
• 3:42
  Just pure WordPress, clean site architecture, and the core WordPress APIs.
• 3:46
  The agent just talks directly to the site.
• 3:48
  That is awesome.
• 3:49
  But, okay, how do site builders or agencies actually measure if agents are visiting?
• 3:55
  Because Google Analytics does not show Claude Code or custom terminal agents
• 4:00
  browsing your pages.
• 4:01
  You have to look at server logs.
• 4:03
  Traditional analytics scripts rely on JavaScript executing in a browser,
• 4:07
  which headless AI agents rarely run.
• 4:10
  So you have to monitor your CDN access logs for direct hits to llms dot txt,
• 4:15
  or even embed single use honeypot URLs inside your Markdown indexes to track when an
• 4:21
  agent follows a link.
• 4:22
  That honeypot trick is brilliant.
• 4:24
  Moving forward into 2026, site builders have to treat raw Markdown delivery and
• 4:30
  structured JSON LD schema with the exact same priority as visual theme design.
• 4:35
  If an agent cannot read your site cleanly, your site does not exist to the automated
• 4:40
  web.
• 4:40
  So build for human eyes on the front, but keep clean Markdown and clear APIs open
• 4:45
  for the bots in the back.
• 4:47
  Sounds like a solid plan.
• 4:49
  Good chatting, guys!
• 4:50
  Talk soon!