Artificial intelligence is transforming how the web is consumed. For over two decades, publishers optimized towards human consumption - the holy metrics being views, clicks, and impressions. Now, an increasing share of their content is being consumed by machines, or AI crawlers, that read, index, and train on the content of the open web.
Unlike traditional search engines, which drive referral traffic, AI crawlers extract value without generating pageviews or monetization opportunities. Publishers face growing infrastructure costs from crawler activity yet have little visibility or control over how often or deeply their sites are crawled.
Despite that, less than 1 percent of publishers have any kind of commercial agreement with AI platforms today. While a handful of large media groups, such as The New York Times, Axel Springer, and News Corp, have deals with AI platforms to license their editorial content, the large majority of publishers remain uncompensated, even though their pages hold measurable training value for large language models.
AI technology companies face a different problem: scaling direct relationships with thousands of publishers is logistically (and often legally) impossible, as each site has different usage rights, pricing expectations, and technical setups. The result is an inefficient market, wherein publishers struggle to fully monetize their content, AI platforms cannot allocate their budgets efficiently, and the open web lacks a fair, scalable exchange mechanism for content access.
That’s the gap BidSwitch has set out to close.
Why Real-Time Bidding (RTB) is the right foundation for crawl access
Real-Time Bidding (RTB), the means by which advertising inventory is bought and sold on a per-impression basis via instantaneous programmatic auction, has already solved this problem once before. Fifteen years ago, digital advertising faced a similar fragmentation challenge, with thousands of publishers struggling to monetize the millions of ad slots that were made available with the boom of online advertising, with no scalable way for buyers and sellers to transact in real time.
From this chaos, the OpenRTB protocol was formed, and the programmatic ecosystem was built around it, standardizing how value flows between SSPs, DSPs, and exchanges. Today, those pipes connect hundreds of thousands of publishers, demand partners, and supply partners, with billions of bid requests traveling through this infrastructure daily, carrying pricing, consent, and policy signals.
Crawl access, at its core, is just a new transaction type for programmatic. The page replaces the ad slot and the crawl request replaces the bid request, with everything else – from reporting to payment flows to auction flows – already in existence.
By extending the RTB infrastructure to crawl monetization, publishers gain an immediate path to price and control access to their content, while AI platforms can pay for content they access, in a scalable and transparent way, without having to build yet another one-off licensing API.
BidSwitch RTB protocol extension for crawl transactions
BidSwitch has introduced support for crawl transactions via the BidSwitch RTB Protocol, enabling publishers to expose crawl requests via their BidSwitch-integrated SSPs, allowing AI crawlers to bid for page-level access. Crawl transactions would then follow the same RTB pattern as typical programmatic inventory:
- The SSP issues a Bid Request with a minimum crawl price (bidfloor) and crawl identifier (ext.crawlid).
- The DSP (AI platform) responds with Accept (price) or NoBid.
- If accepted, the crawler obtains access to the page, the crawl proceeds, and a Crawl Impression is logged via burl.
By reusing existing OpenRTB infrastructure, this protocol extension creates a standardized, scalable crawl monetization framework, enabling publishers to define fair floors for crawl access and AI platforms to extend paid crawl access beyond existing direct licensing deals.
Case study: Monetizing AI crawlers via OpenRTB with Raptive and SmartMedia Technologies
To validate this concept, BidSwitch partnered with Raptive, one of the largest publisher networks, and SmartMedia Technologies, an AI innovator, to pilot real-time crawl monetization.
Raptive’s partner site InspiredTaste.net allowed page-level access to two AI crawlers (AmazonBot and TRADRBot, SmartMedia’s proprietary AI crawler) via the Cloudflare bot management panel. A custom Cloudflare snippet intercepted all incoming crawler requests and forwarded them in OpenRTB format to the BidSwitch Test Crawl DSP endpoint. The Crawl DSP evaluated each crawl request based on the publisher-defined floor price and the crawler’s budget policy and pacing rules, following the same RTB request and response pattern as typical programmatic auctions. Accepted crawls were logged as impressions, while declined requests were blocked. All transactions were logged centrally within the BidSwitch Test DSP.
Throughout the limited test, a total of 3,418 requests were logged between both AI crawlers, with 51.4% (1,757) of requests accepted and logged as a bid, and 48.6% (1,661) requests declined and logged as nobid. Of accepted bids, 99.1% were successful crawls, generating estimated publisher revenue of $174.
The pilot demonstrated that AI traffic can be monetized in real-time and at scale using existing RTB mechanics, enabling publishers to define their own floor prices and control enforcement and AI platforms to manage campaigns through a crawl DSP layer.
“This test was an excellent proof of concept to show that the technology required to implement a pay-per-crawl model can work in practice today. It's an exciting precedent that helps to lay the groundwork for what content monetization can look like in the future.” Paul Bannister, Chief Strategy Officer, Raptive
“It was an honor to be part of this early step, which demonstrates that a pay-per-crawl content monetization model can work. We are very excited to see how this progresses in the future.” Adam Gallagher, Co-Founder, Inspired Taste
“By automating data collection with TRADRBot, we’re unlocking a new level of intelligence and scalability across our Web3 loyalty platform. Our PPCr* via RTB model delivers real-time insights in a more sustainable, efficient way — without the friction of additional publisher contracts.” Tyler Moebius, Chief Executive Officer, SmartMedia Technologies
Launching the Dynamic Content Ledger
Building on this success, BidSwitch is launching the Dynamic Content Ledger (DCL), a new programmatic layer purpose-built to connect AI platforms and publishers.
DCL extends OpenRTB support to crawl transactions, allowing AI crawlers to allocate their budgets dynamically, based on content value; publishers to express the value of their content with adaptive floor pricing; and both sides to benefit from transparent, logged, and auditable transactions.
DCL provides fair and scalable access to high-value content for AI companies without the hassle of managing thousands of one-to-one licensing contracts. Publishers can now leverage DCL to participate in the AI economy through their existing SSP connections, transforming the open web into a transparent, compensated data layer.
Building the future
RTB has connected the open web for over 15 years, and now is the time for it to connect the next layer of the web. By standardizing crawl monetization, the industry can quickly move toward a future where every crawl is logged and priced transparently. Every publisher, regardless of size or technical capabilities, should be able to participate, and AI platforms should be able to access content ethically and at scale.
The Dynamic Content Ledger is the foundation of that future, bringing fairness, scalability, and real-time economics to the AI era of the open web.