Latest / Found, Trusted, Booked / How to Tell If Answer Engines Have Indexed Your Content
Transcript
- You publish good content, but you have no way to know whether answer engines like ChatGPT, Perplexity, or Claude have actually found it. Search Console tells you about Google. It tells you nothing about anyone else. That gap matters because a prospect might ask an AI assistant about your firm or your field, and if the assistant has not indexed your content, you will not appear in the answer, no matter how good the page is. The question is: how do you find out what answer engines actually have? In my latest piece, I wrote about the four places you can check when Search Console comes up empty, and what each one can and cannot prove. Let me walk you through this because the answer is not as straightforward as you might hope. There is no dashboard from OpenAI or Perplexity or any of the major operators that says "Here is your indexing status with us." So you have to read the evidence yourself. And the evidence is scattered across files and logs and traffic data that already exist on your server, you just have to know where to look and what it means. Start with your robots.txt file. Most people know this one sits at the root of your site and tells crawlers what they are and are not allowed to access. Here is the thing that trips people up: a single robots.txt can have different rules for different crawlers. You might give OpenAI's crawler permission to read your whole site, but your general rule for all other crawlers blocks a specific folder. If that happens, only OpenAI gets through, and you will not see anything wrong when you look at the file. You have to match each crawler to its rule. But here is the catch: robots.txt is just a request. A crawler can ignore it. So the next step is your firewall and bot-protection rules. Some content delivery networks and web application firewalls can block bots they do not recognize, even if robots.txt says they are welcome. Googlebot usually gets through because it is so common. But another crawler might get refused at the firewall level. You have to ask your host whether those rules exist and what response codes they return to each crawler. Once you know the crawler is getting through, your server logs show whether it actually showed up. You can filter the logs for the user agent strings each operator publishes. Count the requests per crawler, per page, and per month. One warning: user agent strings can be faked, so verify the source. Google publishes the IP address ranges its crawlers use, and most operators do the same. Match the requests against the real IP ranges to confirm the bot is genuine. Those three checks tell you whether a crawler can reach your content. What they do not tell you is whether it understood it. That is why you should load a key page yourself without running scripts and check whether the article text is in the first response. Some crawlers run JavaScript and some do not. If your content only appears after scripts run, a crawler that does not execute scripts will see an empty shell. You have to check each operator's documentation to know how they handle this. Then look at referral traffic. Google Analytics has an AI Assistant channel that captures visits from ChatGPT, Perplexity, Copilot, and others. But here is what matters: a visit only shows up if someone clicks a link in the assistant's answer and the referrer is passed along. If an assistant cites your page and nobody clicks it, you will not see a referral. So zero visits does not mean zero citations. It means zero clicks. The final step is to ask the assistants yourself. Pose the questions a prospect in your field might ask, and record whether your firm or your content appears in the answer. Run this test a few times because answers vary. This is sampling, not a ranking, but it shows you what is actually happening today. So: your content might not be reaching answer engines, and you need to check four places Search Console does not cover, then interpret what each check actually proves. Read the full piece linked in the show notes for a detailed audit template and a worked example from a wealth management firm, so you can run the same process for your own site. This episode was produced by the ChoiceDigital™ Engine — reviewed for accuracy, editorial quality, and compliance, then approved by Dr. Adrian McMillian, whose voice you have been hearing as an AI-generated replica.