Latest / Marketing Automation in 2026 / Licensing and Robots for Maximum Inclusion: How to Welcome AI Crawlers

Transcript

Licensing and Robots for Maximum Inclusion: How to Welcome AI Crawlers · Marketing Automation in 2026

• 0:00
  SPEAKER_00: Licensing and robots for maximum inclusion.
• 0:03
  SPEAKER_00: How to welcome artificial intelligence crawlers.
• 0:06
  SPEAKER_00: Updated August 25th, 2026.
• 0:10
  SPEAKER_00: Websites now have more than one way to reach an audience.
• 0:14
  SPEAKER_00: A page may be found through Google Search, summarized by ChatGPT, cited by Perplexity, used in CloudSearch, displayed through Apple services, or collected by a public web archive.
• 0:26
  SPEAKER_00: These systems do not all use the same crawler or follow the same rules.
• 0:30
  SPEAKER_00: A website that blocks GPTbot may still appear in ChatGPT search if it allows OAI SearchBot.
• 0:37
  SPEAKER_00: A website that allows AppleBot can remain visible in Apple Search while blocking AppleBot extended from artificial intelligence model training.
• 0:45
  SPEAKER_00: Google's Google Extended is not a normal crawler at all.
• 0:49
  SPEAKER_00: It is a robots.text control token.
• 0:52
  SPEAKER_00: The best policy is therefore not simply allow artificial intelligence or block artificial intelligence.
• 0:58
  SPEAKER_00: It is to create separate permissions for search and discovery, user requested retrieval, model development and training, public datasets, sensitive private or restricted material.
• 1:09
  SPEAKER_00: This article provides a current crawler inventory, robots.txt examples, meta tag guidance, internet protocol address verification methods, Creative Commons licensing advice, legal boilerplate, and a risk-benefit matrix, the four controls every publisher should understand.
• 1:26
  SPEAKER_00: Robots.txt controls requested crawling.
• 1:29
  SPEAKER_00: A robots.txt file tells compliant automated clients which pages they may request.
• 1:35
  SPEAKER_00: It is useful for managing crawler traffic and expressing a publisher's preferences.
• 1:39
  SPEAKER_00: However, robots.txt is not an access control system.
• 1:44
  SPEAKER_00: The robots exclusion protocol states that its rules are not access authorization.
• 1:49
  SPEAKER_00: Google also warns that a blocked address can still appear in search results if other pages link to it.
• 1:55
  SPEAKER_00: Use passwords, authentication, or server-side access controls for confidential information.
• 2:01
  SPEAKER_00: Meta tags control indexing and snippets.
• 2:04
  SPEAKER_00: Robots meta tags and the Xrobots tag response header can control whether a page is indexed or whether a search engine may show a snippet.
• 2:12
  SPEAKER_00: These controls are normally visible only after a crawler has been allowed to fetch the page.
• 2:16
  SPEAKER_00: If robots.txt blocks the page first, the crawler may never see the meta tag.
• 2:22
  SPEAKER_00: Network controls verify the crawler.
• 2:24
  SPEAKER_00: A user agent name is easy to copy.
• 2:27
  SPEAKER_00: An attacker can send a request claiming to be GPTbot, Googlebot, or PerplexityBot.
• 2:32
  SPEAKER_00: A stronger approach combines the claimed user agent, a published internet protocol address range, reverse domain name system verification, forward domain name system verification, rate limits, and request monitoring.
• 2:45
  SPEAKER_00: A license grants reuse rights.
• 2:47
  SPEAKER_00: Robots.txt says what a crawler is asked to do.
• 2:50
  SPEAKER_00: A license says what people or organizations may legally do with material when copyright permission is required.
• 2:57
  SPEAKER_00: A permissive license can reduce legal uncertainty, but it does not guarantee that a crawler will visit the page, that a model will use it, or that an assistant will cite it.
• 3:08
  SPEAKER_00: Inventory of Important Crawlers.
• 3:10
  SPEAKER_00: The following inventory was checked against provider documentation available on August 25th, 2026.
• 3:16
  SPEAKER_00: User agent names, purposes, and internet protocol address ranges can change, so production systems should use the provider's current documentation and live address feeds.
• 3:25
  SPEAKER_00: For the full table, please open this article on autopod.co.
• 3:30
  SPEAKER_00: OpenAI documents its search, training, advertising, and user-triggered crawlers as separate controls.
• 3:36
  SPEAKER_00: Its documentation specifically recommends allowing OAI SearchBot for ChatGPT search while using GPTBot separately for training preferences.
• 3:45
  SPEAKER_00: Google similarly separates GoogleBot, Google Other, and Google Extended.
• 3:50
  SPEAKER_00: Google Extended does not have its own HTTP request user agent, and blocking it does not remove a page from Google search.
• 3:57
  SPEAKER_00: Anthropic documents separate roles for CloudBot, CloudSearchBot, and CloudUser.
• 4:02
  SPEAKER_00: Anthropic also states that its bots follow robots.txt and support the non-standard crawl delay directive.
• 4:09
  SPEAKER_00: Perplexity distinguishes between automatic search crawling and user requested fetching.
• 4:14
  SPEAKER_00: Its current documentation says that PerplexityBot respects robots.txt, while PerplexityUser generally does not because it responds to a user request.
• 4:23
  SPEAKER_00: Apple's current documentation makes the same search versus training distinction.
• 4:27
  SPEAKER_00: AppleBot supports discovery, while AppleBot extended lets publishers control training use without removing pages from Apple search.
• 4:35
  SPEAKER_00: Recommended robots.txt configurations place robots.txt at the root of each host, such as rules apply to the specific host, protocol, and port where the file is served.
• 4:46
  SPEAKER_00: A separate subdomain may need its own file.
• 4:49
  SPEAKER_00: Configuration one, maximum inclusion.
• 4:52
  SPEAKER_00: Use this when public editorial content may be found, summarized, cited, indexed, and collected by compliant crawlers.
• 4:58
  SPEAKER_00: This allows named crawlers including OpenAI, Google, Anthropic, Perplexity, Apple, Common Crawl, Microsoft, and Amazon, unless another specific rule blocks them.
• 5:08
  SPEAKER_00: Do not place a blanket rule such as user agent table disallow below this configuration.
• 5:13
  SPEAKER_00: A later or more specific group may change how a crawler interprets the file.
• 5:18
  SPEAKER_00: Configuration 2.
• 5:19
  SPEAKER_00: Allow search but block model development crawlers.
• 5:22
  SPEAKER_00: This is often the best compromise for publishers that want citations and search traffic but do not want to signal permission for model development collection.
• 5:30
  SPEAKER_00: This configuration leaves OAI SearchBot, Cloud Searchbot, PerplexityBot, Googlebot, and AppleBot covered by the wildcard group.
• 5:38
  SPEAKER_00: It also keeps search and training permissions separate.
• 5:40
  SPEAKER_00: For Apple, blocking AppleBot extended while allowing AppleBot preserves discovery through Apple services.
• 5:47
  SPEAKER_00: For Google, blocking Google Extended does not block ordinary Google search.
• 5:52
  SPEAKER_00: Configuration 3.
• 5:53
  SPEAKER_00: Explicitly allow selected search crawlers.
• 5:56
  SPEAKER_00: If an existing robot's text file blocks all crawlers, add separate groups for the search crawlers you want to welcome.
• 6:03
  SPEAKER_00: When using specific groups, repeat the sensitive path rules inside each group.
• 6:08
  SPEAKER_00: Some crawlers use the most specific matching group rather than combining it with the wildcard group.
• 6:14
  SPEAKER_00: Bang documents this behavior directly, and Google provides separate guidance on crawler-specific groups.
• 6:20
  SPEAKER_00: Important robots.
• 6:22
  SPEAKER_00: A crawler can ignore robots text.
• 6:25
  SPEAKER_00: A malicious crawler can pretend to be a trusted crawler.
• 6:28
  SPEAKER_00: A blocked page can still be known by its title or web address.
• 6:32
  SPEAKER_00: Robots text does not protect passwords, private files, customer records, or confidential application programming interfaces.
• 6:40
  SPEAKER_00: A crawl delay directive is not part of the core robots exclusion protocol and is not supported by every crawler.
• 6:47
  SPEAKER_00: Anthropic and Bing document support, while Apple says AppleBot does not follow crawl delay.
• 6:52
  SPEAKER_00: Meta tags and HTTP headers.
• 6:55
  SPEAKER_00: Public page example.
• 6:56
  SPEAKER_00: For a public article, use a clear title, author, canonical web address, publication date, and modification date.
• 7:03
  SPEAKER_00: The MacSnippet directive is primarily documented for Google.
• 7:06
  SPEAKER_00: Do not assume that every artificial intelligence crawler supports every meta directive.
• 7:11
  SPEAKER_00: Private or sensitive page example.
• 7:14
  SPEAKER_00: Use this for pages that should not appear in search results.
• 7:17
  SPEAKER_00: The page must remain crawlable long enough for the crawler to see the directive.
• 7:21
  SPEAKER_00: If the page must truly remain private, use authentication or password protection instead.
• 7:27
  SPEAKER_00: Hide only sensitive sections from search snippets.
• 7:30
  SPEAKER_00: Google supports data no snippet for specific parts of an HTML page.
• 7:34
  SPEAKER_00: This is useful for contact details, internal notes, or user-generated information.
• 7:39
  SPEAKER_00: It is not a universal instruction for every artificial intelligence system.
• 7:43
  SPEAKER_00: Use the Xrobots tag header for files.
• 7:46
  SPEAKER_00: Meta tags cannot be placed inside a portable document file, image, or video file.
• 7:51
  SPEAKER_00: Use an HTTP response header instead.
• 7:54
  SPEAKER_00: For a page that should remain searchable, but should not supply text for snippets or artificial intelligence answers use.
• 8:01
  SPEAKER_00: Google documents XRobots tag for non-HTML resources, and Apple supports it for AppleBot as well.
• 8:07
  SPEAKER_00: Apple specific controls.
• 8:09
  SPEAKER_00: Apple supports Apple says no snippet prevents the page from being used as additional context and current content when Apple models generate output.
• 8:18
  SPEAKER_00: The page may still remain discoverable through Apple Search, Spotlight, Siri, and Safari.
• 8:23
  SPEAKER_00: For paywalled pages, Apple also supports structured data using.
• 8:28
  SPEAKER_00: Apple says such pages may remain eligible for search results, but will not be used as additional context for artificial intelligence answers.
• 8:37
  SPEAKER_00: How to verify crawler internet protocol addresses.
• 8:40
  SPEAKER_00: Why user agent matching is not enough.
• 8:43
  SPEAKER_00: A rule such as this is weak.
• 8:45
  SPEAKER_00: Anyone can send that text.
• 8:47
  SPEAKER_00: A better rule is do not trust an X-Forded foreheader unless it comes from a proxy or content delivery network that your organization controls.
• 8:55
  SPEAKER_00: Provider verification methods.
• 8:57
  SPEAKER_00: For the full table, please open this article on Autopod.co.
• 9:02
  SPEAKER_00: Google, Apple, Common Crawl, OpenAI, Anthropic, and Perplexity all publish provider-specific methods or address feeds.
• 9:10
  SPEAKER_00: These lists can change, so a scheduled update process is safer than a permanent manually copied list.
• 9:15
  SPEAKER_00: A simple firewall design might look like this.
• 9:18
  SPEAKER_00: Do not apply a broad allow rule to an entire cloud provider.
• 9:22
  SPEAKER_00: A legitimate crawler may use cloud infrastructure, but most traffic from that cloud provider is not necessarily the crawler.
• 9:28
  SPEAKER_00: How open licensing influences inclusion.
• 9:31
  SPEAKER_00: CC by 4.0 in plain language.
• 9:34
  SPEAKER_00: The Creative Commons Attribution 4.0 International License, commonly called CC by 4.0, allows people to copy and redistribute the work, adapt, translate, remix, and build upon it, use it commercially.
• 9:47
  SPEAKER_00: The main conditions are give appropriate credit, link to the license, indicate whether changes were made, do not suggest that the original creator endorses the reuse, do not add legal or technical restrictions that prevent uses allowed by the license.
• 10:00
  SPEAKER_00: Creative Commons also warns that the license may not cover privacy rights, publicity rights, moral rights, trademark rights, patent rights, or third-party material.
• 10:10
  SPEAKER_00: A license is not a crawler invitation by itself.
• 10:13
  SPEAKER_00: Putting CC by 4.0 on a page does not guarantee that an organization will crawl it.
• 10:18
  SPEAKER_00: A crawler may still be blocked by robots.txt, a content delivery network, a web application firewall, rate limiting, a JavaScript challenge, a login wall, a poor server response, an inaccessible sitemap.
• 10:32
  SPEAKER_00: Conversely, allowing a crawler does not automatically grant every copyright permission needed for every later use.
• 10:38
  SPEAKER_00: Robots.txt and licensing should be designed together.
• 10:41
  SPEAKER_00: YCC BY can support broader inclusion.
• 10:44
  SPEAKER_00: A clear permissive license may make a publisher's policy easier for data teams, search systems, and assistant operators to understand.
• 10:51
  SPEAKER_00: It can reduce uncertainty about copying, adaptation, commercial use, translation, and redistribution when those activities require copyright permission.
• 11:00
  SPEAKER_00: However, Creative Commons explains that artificial intelligence training is legally complex.
• 11:05
  SPEAKER_00: A restrictive license is not always an effective way to prevent training, because some training uses may be permitted by copyright exceptions or limitations.
• 11:14
  SPEAKER_00: Creative Commons also notes that license conditions can be difficult to apply to machine training and model outputs.
• 11:21
  SPEAKER_00: The practical lesson is use CC BY when you genuinely want broad, lawful reuse.
• 11:27
  SPEAKER_00: Do not use a restrictive Creative Commons license as a guaranteed artificial intelligence training opt-out.
• 11:34
  SPEAKER_00: Attribution improves source clarity.
• 11:36
  SPEAKER_00: Creative Commons recommends the TASEL method, title, author, source license.
• 11:42
  SPEAKER_00: For example, licensing in robots for maximum inclusion, example publishing, licensed under Creative Commons Attribution 4.0 International.
• 11:51
  SPEAKER_00: Changes made.
• 11:52
  SPEAKER_00: Updated crawler inventory.
• 11:54
  SPEAKER_00: Clear attribution does not force every assistant to cite a page.
• 11:58
  SPEAKER_00: It does make correct citation easier when a system extracts the page's title, author, source, and licensing information.
• 12:06
  SPEAKER_00: Add structured article information.
• 12:09
  SPEAKER_00: Use visible information and structured data together.
• 12:12
  SPEAKER_00: Google recommends clear author information, author pages, publication dates, modification dates, and stable canonical pages.
• 12:20
  SPEAKER_00: These signals can help search systems understand who created the material and which version is authoritative.
• 12:26
  SPEAKER_00: They do not guarantee a ranking, inclusion, or citation.
• 12:31
  SPEAKER_00: Legal boilerplate for an open, citation-friendly site.
• 12:35
  SPEAKER_00: The following is sample language, not legal advice.
• 12:38
  SPEAKER_00: Have counsel adapt it to your jurisdiction, ownership structure, privacy obligations, and third party content.
• 12:45
  SPEAKER_00: CC by 4.0 licensing statement.
• 12:48
  SPEAKER_00: The phrase including artificial intelligence systems clarifies the publisher's intent.
• 12:53
  SPEAKER_00: It should not be used to pretend that the publisher owns rights to material created by someone else.
• 12:58
  SPEAKER_00: Brand privacy and third party rights notice.
• 13:01
  SPEAKER_00: This language is important because Creative Commons licenses do not automatically grant every type of legal right connected to a page.
• 13:09
  SPEAKER_00: Search and assistant citation guidance.
• 13:12
  SPEAKER_00: If you want search visibility but do not want to grant a broad training license, do not place the CC BY Statement and the All Rights Reserve Statement over the same material.
• 13:22
  SPEAKER_00: Clearly identify which license applies to which content.
• 13:26
  SPEAKER_00: Risk benefit matrix for open licensing.
• 13:29
  SPEAKER_00: Copyright law and artificial intelligence training rules differ by country and remain unsettled.
• 13:34
  SPEAKER_00: The United States Copyright Office continues to examine these issues, while Creative Commons describes artificial intelligence training as fact-specific and legally complex.
• 13:45
  SPEAKER_00: For the full table, please open this article on Autopod.co.
• 13:49
  SPEAKER_00: The most balanced strategy for many organizations is CC BY 4.0.
• 13:54
  SPEAKER_00: Zero for original public editorial text.
• 13:56
  SPEAKER_00: Separate labels for third-party material.
• 13:59
  SPEAKER_00: No public, personal, or confidential information.
• 14:02
  SPEAKER_00: Allow search and retrieval crawlers.
• 14:04
  SPEAKER_00: Allow model development crawlers only if the organization truly accepts that use.
• 14:08
  SPEAKER_00: Use clear attribution and canonical links.
• 14:11
  SPEAKER_00: Protect trademarks through a separate brand notice.
• 14:13
  SPEAKER_00: A practical implementation checklist.
• 14:15
  SPEAKER_00: Content and licensing.
• 14:17
  SPEAKER_00: Identify who owns each page, image, chart, video, and quotation.
• 14:21
  SPEAKER_00: License only material for which your organization controls the rights.
• 14:24
  SPEAKER_00: Mark third-party materials separately.
• 14:27
  SPEAKER_00: Add a visible license statement.
• 14:28
  SPEAKER_00: Provide a preferred citation using title, author, source, and license.
• 14:33
  SPEAKER_00: Keep logos, trademarks, personal data, and confidential information outside the license scope.
• 14:38
  SPEAKER_00: Robots.txt.
• 14:40
  SPEAKER_00: Create a public robots.txt file at the root of every host.
• 14:44
  SPEAKER_00: Allow search crawlers separately from training crawlers.
• 14:47
  SPEAKER_00: Block administrative, account, checkout, private, and internal application paths.
• 14:51
  SPEAKER_00: Do not rely on robots.txt for security.
• 14:54
  SPEAKER_00: Test the file after every major website deployment.
• 14:57
  SPEAKER_00: Meta tags.
• 14:58
  SPEAKER_00: Use canonical links.
• 15:00
  SPEAKER_00: Add author, publication date, and modification date.
• 15:03
  SPEAKER_00: Use noindex for pages that should not appear in search.
• 15:06
  SPEAKER_00: Use no snippet or data no snippet for sensitive excerpts where supported.
• 15:10
  SPEAKER_00: Use Xrobots tag for portable document files, images, and other non-HTML resources.
• 15:16
  SPEAKER_00: Remember that a crawler must be able to fetch a page before it can read its meta tags.
• 15:20
  SPEAKER_00: Network security.
• 15:26
  SPEAKER_00: Verify trusted crawlers using official address feeds or domain name system methods.
• 15:31
  SPEAKER_00: Refresh address feeds automatically.
• 15:33
  SPEAKER_00: Combine user agent and source address checks.
• 15:36
  SPEAKER_00: Rate limit verified crawlers rather than giving them unrestricted access.
• 15:40
  SPEAKER_00: Challenge or block requests that claim to be trusted crawlers but fail verification.
• 15:44
  SPEAKER_00: Measurement.
• 15:45
  SPEAKER_00: Track.
• 15:46
  SPEAKER_00: Visits from artificial intelligence search services.
• 15:49
  SPEAKER_00: Referrals to the original page.
• 15:52
  SPEAKER_00: Server load by crawler.
• 15:54
  SPEAKER_00: Requests returning 403, 404, or 429, crawler access to unintended paths.
• 16:00
  SPEAKER_00: Whether current articles are being found after publication.
• 16:03
  SPEAKER_00: Conclusion.
• 16:04
  SPEAKER_00: Maximum inclusion requires selective openness, not a single blanket permission.
• 16:08
  SPEAKER_00: Use robots.txt to separate search, user retrieval, training, and public dataset crawling.
• 16:14
  SPEAKER_00: Use meta tags and HTTP headers to manage indexing and snippets.
• 16:18
  SPEAKER_00: Use authentication for anything private.
• 16:20
  SPEAKER_00: Verify crawler internet protocol addresses rather than trusting user agent names alone.
• 16:25
  SPEAKER_00: A permissive license such as CC by 4.0 can make broad reuse easier when you genuinely want it.
• 16:32
  SPEAKER_00: Clear attribution information.
• 16:33
  SPEAKER_00: Title, author, source, license, canonical page, and update date can also make it easier for search systems and assistants to identify and cite the correct source.
• 16:44
  SPEAKER_00: The strongest publishing policy is therefore.
• 16:47
  SPEAKER_00: Open the public content you want discovered, clearly license the material you want reused, mark the material you do not own, protect private information at the server level, and give every crawler a separate reviewable permission.
• 17:00
  SPEAKER_00: All links to sources are available in the text version of this article.
• 17:04
  SPEAKER_00: You can find the full article at autopod.co.