Content Cache
Control your CDN costs through Content Cache. You allow all bots to use the cache, or explicitly allow or block bots individually.
Content Cache is a routing mode that serves cached HTML to whitelisted bots without charging them. It creates a sanctioned, cost-neutral access path for AI agents while protecting publishers from overwhelming bot traffic.
To use this feature to its fullest extent, start by redirecting bots from your main site to your agent site (tollbit subdomain); you can view how to do this in your CDN via our integrations page. AI bots would continue to be able to fetch your content, but it would be doing so through your agent site instead of your human site. TollBit intelligently routes the request based on specified preferences. Simply whitelist agents and their crawling automatically becomes sanctioned through the cache.
Select a TollBit property, and under Web Content select Content cache on the top of the page. You may either select all AI bots to use content cache or select specific bots by selecting them in the table or searching/adding a string.
In summary, a redrected bot can fall in 2 categories:
- Whitelisted → Check cache → Serve from cache or fetch once and cache
- Not whitelisted → Require payment token or block
Note that Content Cache is not a permanent archive. Pages are stored depending on whether they are requested by bots. The TTL is set to 5 minutes for non article pages (home page, hub pages, etc), and longer for articles, after which the cached content is automatically evicted.
Updated 2 days ago