Latest / ELECTE's Podcast: AI Frontiers / AI Traffic: Monetizing the Crawl

Transcript

AI Traffic: Monetizing the Crawl · ELECTE's Podcast: AI Frontiers

• 0:00
  SPEAKER_00: The problem When AI breaks the pact with publishers, the Google model, a balance that worked for over 20 years, the web has operated on an unwritten pact.
• 0:11
  SPEAKER_00: Google and other search engines indexed publishers' content for free, displaying snippets and summaries on their pages, but in return they sent massive traffic to the original sites.
• 0:21
  SPEAKER_00: Publishers accepted this use of their content because they received visibility, readers, and as a result, advertising revenue.
• 0:30
  SPEAKER_00: Google News perfected this model.
• 0:32
  SPEAKER_00: It aggregates news from thousands of sources, displays headlines and short excerpts, but each click takes the user to the original site.
• 0:41
  SPEAKER_00: A win-win balance that supported the digital information ecosystem.
• 0:46
  SPEAKER_00: AI breaks the balance.
• 0:48
  SPEAKER_00: Imagine you run a news site and discover that every day thousands of AI bots scan your articles, use the content to feed their models, and serve complete answers to users without ever sending traffic back to your site.
• 1:01
  SPEAKER_00: AI that doesn't generate links and doesn't generate clicks, it responds directly.
• 1:06
  SPEAKER_00: Welcome to the reality of 2025.
• 1:09
  SPEAKER_00: According to Cloudflare data, by mid-2025, 80% of AI crawling activity will be for model training, while referrals to publishers, especially from Google, are declining dramatically.
• 1:22
  SPEAKER_00: The model that has worked for 20 years is crumbling.
• 1:25
  SPEAKER_00: Google itself, with its AI overviews, is sending less and less traffic to the original sites.
• 1:32
  SPEAKER_00: The crawl to refer ratio shows shocking numbers.
• 1:35
  SPEAKER_00: Anthropic has a ratio of 38,000 scans for every visitor sent to the site, while OpenAI reaches$1,700 v1.
• 1:44
  SPEAKER_00: But what if bots could become paying customers instead of just free consumers of content?
• 1:48
  SPEAKER_00: It's time for a new deal.
• 1:50
  SPEAKER_00: If AI can no longer guarantee return traffic as Google did, it must at least pay for access to content.
• 1:57
  SPEAKER_00: The solution pay-per-crawl and AI micropayments Cloudflare leads the way.
• 2:04
  SPEAKER_00: In July 2025, Cloudflare announced that it had become the first internet infrastructure company to block AI crawlers by default, simultaneously introducing the pay-per-crawl system.
• 2:16
  SPEAKER_00: What's new?
• 2:17
  SPEAKER_00: Every new domain that registers with Cloudflare is now explicitly asked if it wants to allow AI crawlers to access its content, but the real game changer is the marketplace where publishers can request compensation from AI companies every time one of their pages is scanned.
• 2:32
  SPEAKER_00: No longer just allow or block, but a third option, charge.
• 2:37
  SPEAKER_00: Tollbit.
• 2:38
  SPEAKER_00: The pioneer of AI micropayments even before Cloudflare, Tolbit had begun building this infrastructure.
• 2:44
  SPEAKER_00: The startup, which recently raised$24 million in Series A funding, has created a platform where AI bots can pay websites directly to use their content.
• 2:54
  SPEAKER_00: Time and AdWeek are among the first customers experimenting with this model, discovering that AI represents a new category of customers with specific needs.
• 3:03
  SPEAKER_00: The numbers.
• 3:04
  SPEAKER_00: How much can you really earn?
• 3:07
  SPEAKER_00: Tolbit's projections.
• 3:09
  SPEAKER_00: According to a detailed analysis, Tolbit could already be generating approximately$71 million per year for its publishing partners.
• 3:18
  SPEAKER_00: Here's how exponential traffic growth.
• 3:21
  SPEAKER_00: AI bot traffic to Tolbit client sites has doubled every quarter Q4 2024.
• 3:26
  SPEAKER_00: 2.75 million daily bot visits Q1 2025.
• 3:31
  SPEAKER_00: 6.5 million visits Q2 2025 projection.
• 3:34
  SPEAKER_00: 13 million visits using a typical CPM of$15 for content sites.
• 3:39
  SPEAKER_00: This translates to$195,000 per day in potential revenue for the Tolbit ecosystem or over$71 million per year.
• 3:47
  SPEAKER_00: The pricing model tollbit suggests two pricing tiers.
• 3:51
  SPEAKER_00: Summarization rate for bots that want to summarize content syndication rate, much higher for bots that want to display the full article.
• 4:00
  SPEAKER_00: The difference can be substantial, making this a particularly attractive model for premium and specialized content.
• 4:06
  SPEAKER_00: Real data from the field of Tolbit analysis of 160 websites revealed that AI companies scanned an average of 2 million times in Q4 2024.
• 4:16
  SPEAKER_00: Each page was scanned an average of seven times.
• 4:19
  SPEAKER_00: Toshit Panigrahi, co-founder of Tolbit, explains The bot traffic generated by these AI platforms is almost equivalent to the bot traffic of search engines that have been around for 20 years, which is incredible.
• 4:32
  SPEAKER_00: The evolving ecosystem Datadome.
• 4:37
  SPEAKER_00: Datadome has partnered with both Tolbit and other platforms to offer an integrated solution.
• 4:43
  SPEAKER_00: Real-time protection from malicious bots, and monetization of compliant ones.
• 4:48
  SPEAKER_00: Their data shows that AI traffic has tripled in just six months from 2, 6% to 8, 2% of all verified bot traffic.
• 4:58
  SPEAKER_00: Skyfire, the AI payment network.
• 5:01
  SPEAKER_00: Skyfire represents a different approach, building an entire payment network for autonomous AI agents.
• 5:07
  SPEAKER_00: With$8,5 million raised in partnerships with companies such as Datadome, Skyfire aims to become the standard payment infrastructure for the AI economy, the impact for publishers, opportunities and challenges, the concrete benefits, new source of revenue, diversification beyond traditional advertising, granular control, ability to choose which bots to allow access and at what price, complete visibility, detailed dashboards on who accesses content and how often, direct compensation, immediate payments without advertising intermediaries, the realistic challenges, limited adoption, not all AI companies will agree to pay to pricing complexity, difficult to determine the correct value for each type of content.
• 5:55
  SPEAKER_00: Enforcement, many bots will continue to ignore the rules from 3-3% in 2024 to 13% in March 2025 for scale required.
• 6:05
  SPEAKER_00: Micropayments only become significant with high volumes realistic forecasts for publishers.
• 6:11
  SPEAKER_00: The convenience threshold, blocking is not always worth it.
• 6:16
  SPEAKER_00: Before analyzing the numbers, it is important to note that paper crawl is not convenient for everyone.
• 6:22
  SPEAKER_00: For sites with limited traffic, allowing free access to AI bots can be strategically more advantageous.
• 6:29
  SPEAKER_00: It increases the discoverability of content in AI systems, builds relationships with emerging platforms, and positions the site as AI friendly for future opportunities.
• 6:40
  SPEAKER_00: Direct monetization only becomes attractive when volumes justify the management complexity and implementation costs.
• 6:49
  SPEAKER_00: For small sites, 10K visits jocks month.
• 6:52
  SPEAKER_00: Estimated bot traffic, 500 to 1,000 AI hits jock month potential revenue,$5 to 50 month, depending on CPM and acceptance rate.
• 7:02
  SPEAKER_00: Impact marginal, often better to remain open for visibility recommendation.
• 7:07
  SPEAKER_00: Monitor and evaluate strategic free access for medium sites, 100k visits month.
• 7:12
  SPEAKER_00: Estimated bot traffic, 5,000 to 15,000 AI hits month potential revenue,$75 to$750 month impact threshold where monetization begins to make sense.
• 7:25
  SPEAKER_00: Recommendation.
• 7:51
  SPEAKER_00: Chandra Sekar, CEO of Stack Overflow, sums it up perfectly.
• 7:56
  SPEAKER_00: Community platforms that power LLMs should be compensated for their contributions so they can reinvest in their communities.
• 8:04
  SPEAKER_00: With AI to AI commerce projected to reach$46 billion over the next three years, publishers who position themselves early in this ecosystem could find themselves with a significant competitive advantage.
• 8:16
  SPEAKER_00: Conclusions.
• 8:17
  SPEAKER_00: Is it time to act?
• 8:19
  SPEAKER_00: Pay per crawl is no longer science fiction, but operational reality.
• 8:23
  SPEAKER_00: For publishers, the question is not whether this model will take hold, but if and when they can adapt to capture its value.
• 8:31
  SPEAKER_00: The uncomfortable truth, not everyone should monetize right away.
• 8:35
  SPEAKER_00: For sites with limited traffic, 50k visits month, maintaining free access to AI bots may be more strategically advantageous, building visibility and relationships for future opportunities.
• 8:47
  SPEAKER_00: The breakeven point for direct monetization is generally reached with significant traffic volumes.
• 8:53
  SPEAKER_00: Projections show attractive potential revenues for medium and large publishers, especially for quality content and substantial traffic.
• 9:01
  SPEAKER_00: However, success will depend on the ability to assess your breakeven point, volumes versus management complexity.
• 9:11
  SPEAKER_00: Choose the right strategy, free access, total block or monetization.
• 9:17
  SPEAKER_00: Choose the right platform, cloud flare, toll bit, or integrated solutions.
• 9:22
  SPEAKER_00: Set competitive but profitable pricing.
• 9:26
  SPEAKER_00: Maintain content quality that justifies micropayments.
• 9:30
  SPEAKER_00: Monitor and optimize the approach constantly for publishers with substantial traffic who are willing to experiment.
• 9:36
  SPEAKER_00: 2025 could be the year when AI ceases to be just a cost and finally becomes a business opportunity.
• 9:42
  SPEAKER_00: For everyone else, it may be time to lay the groundwork to capture this opportunity in the future.
• 9:47
  SPEAKER_00: Main sources: Cloudflare blog, AI Crawler Control, Tollbit Series.
• 9:53
  SPEAKER_00: A Announcement, MIT Technology Review, Cloudflare AI Blocking Datadome Partnership Announcement, Skyfire TechCrunch Coverage, Share the Newsletter.
• 10:03
  SPEAKER_00: Welcome to Electies Newsletter, English.
• 10:06
  SPEAKER_00: This newsletter explores the fascinating world of how companies are using AI to change the way they work.
• 10:12
  SPEAKER_00: It shares interesting stories and discoveries about artificial intelligence in business, like how companies are using AI to make smarter decisions, what new AI tools are emerging, and how these changes affect our everyday lives.
• 10:24
  SPEAKER_00: You don't need to be a tech expert to enjoy it.
• 10:27
  SPEAKER_00: It's written for anyone curious about how AI is shaping the future of business and work.
• 10:32
  SPEAKER_00: Whether you're interested in learning about the latest AI breakthroughs, understanding how companies are becoming more innovative, or just want to stay informed about tech trends.
• 10:42
  SPEAKER_00: This newsletter breaks it all down in an engaging, easy to understand way.
• 10:47
  SPEAKER_00: It's like having a friendly guide who keeps you in the loop about the most interesting developments in business technology.
• 10:54
  SPEAKER_00: Without getting too technical or complicated, subscribe now, subscribe to get full access to the newsletter and publication archives.