If open web publishers want a market for crawl access, the first thing they need is control.
AI crawlers are already extracting, summarizing, and consuming publisher content inside AI interfaces, and – from what we’re seeing at BidSwitch – they rarely send traffic back in return. At the same time, BidSwitch exchange data shows that only about 27% of publishers are blocking major AI crawlers today.
That number says a lot.
The industry conversation around AI and publisher rights may be accelerating, but the operational reality is still early. Most publishers haven’t fully locked down crawler access yet. And as long as content remains broadly available for free, crawlers have little reason to adopt paid access models later.
In many cases, the user gets an answer or summary without ever visiting the original source. That creates a familiar imbalance: publishers fund and produce the content, while other platforms capture more of the value around its use.
Our position is that the open web needs a fairer, more scalable way for content access to be priced and transacted. That’s one of the biggest reasons we announced the Dynamic Content Ledger (DCL) in November 2025.
From a distance, blocking AI crawlers sounds simple. You add a few lines to robots.txt, publish, and move on.
In practice, it’s rarely that clean.
Yes, robots.txt matters – but it’s only one layer. Publishers that want real control also need enforcement beyond robots.txt, especially when they want to manage access across different crawler types and infrastructure environments.
That usually means a mix of:
Clear rules around which crawlers are allowed and which are not
Properly configured robots.txt directives
Enforcement through CDN or WAF tools such as Cloudflare, Fastly, Akamai, or AWS WAF
Visibility into who is crawling, how often, and which URLs are being hit
That’s where things tend to stall. Not because publishers don’t care, but because crawler control sits at the intersection of policy, operations, ad tech, and site infrastructure. It’s easy to agree it matters. It’s harder to get it done.
There’s a broader market question behind all of this.
If publisher content remains broadly accessible for free, why would crawlers ever move toward paid access models? Markets don’t form around assets that can be taken without friction – they form when access has structure.
The broader industry is increasingly aligning around stricter control of AI crawlers. For example, in his blog, How blocking AI bots paves the way for fair compensation, Paul Bannister, Raptive CSO, argues that blocking AI training bots is a necessary first step toward establishing fair compensation models for publishers.
At BidSwitch, we see it the same way. Crawler control is foundational, not optional.
But before publishers can monetize crawl access, they need the ability to define it. And that means moving from passive exposure to active permissioning. It also means treating crawl access less like a background technical detail and more like a monetizable supply path in its own right.
There’s also nuance to that access. Not every publisher wants to shut the gates completely, and, frankly, that’s not the only option. Some publishers may want to block aggressively now. Others may prefer selective access. Others still may want to understand the exposure first, then decide. Those are all reasonable positions.
But whatever policy a publisher chooses, one thing is clear: unrestricted access weakens leverage.
No control, no scarcity. No scarcity, no market.
The Dynamic Content Ledger is a hands-on support initiative from BidSwitch designed to help publishers put practical crawler controls in place. The focus is straightforward: help publishers reduce uncertainty, answer technical questions, and move from passive exposure to active control.
That can include support with:
robots.txt configuration
CDN and WAF enforcement
Visibility into crawler traffic patterns
Understanding which parts of a site are most exposed
Preparing for future crawl monetization models
There’s nothing too flashy about this work, but that’s the point.
Before any market matures, the plumbing has to work. BidSwitch has spent years building transaction infrastructure for digital advertising. Applying that operational mindset here means treating crawler access less like a vague industry issue and more like a supply-side control problem that can be solved.
This is a very good moment for publishers to get specific, asking questions like:
Do you know which AI crawlers are accessing your content today?
Do you know whether your current controls are advisory, enforceable, or somewhere in between.
Do you know which pages or sections of your site are most exposed?
And if a real market for paid crawl access develops, will you be ready to participate from a position of control?
Not too long ago, questions like these were hypothetical. Today, they’re operational.
Our view at BidSwitch is simple: publishers shouldn’t have to choose between doing nothing and waiting for the market to sort itself out. There’s practical work that can be done now, and that work matters. The more control publishers have today, the more options they’ll have tomorrow.
If you’re a publisher and want help understanding your crawler exposure, configuring enforcement, or preparing for crawl monetization, talk to the BidSwitch team via bidswitch@bidswitch.com.