Creative Commons Backs Pay-to-Crawl Technology for AI Web Crawlers
Earlier this year, Creative Commons (CC) announced a framework for an open AI ecosystem. Now, the nonprofit organization is showing support for “pay-to-crawl” technology, which involves automating compensation for website content accessed by machines such as AI web crawlers.
CC is widely recognized for its role in the licensing movement that enables creators to share their work while retaining copyright. In a recent blog post, the organization expressed cautious support for pay-to-crawl systems, highlighting the potential benefits for websites to sustain content creation and sharing while managing substitutive uses.
The concept of pay-to-crawl, championed by companies like Cloudflare, revolves around charging AI bots each time they scrape a site to gather content for model training and updates. This approach differs from the traditional practice of allowing web crawlers to freely index website content for search engine inclusion.
With the rise of AI technology, the landscape has shifted, impacting publishers as AI-driven interactions often bypass the need for users to visit the original sources. This shift has led to a decline in search traffic for publishers, prompting the exploration of pay-to-crawl as a potential solution to help them recover lost revenue.
While major content deals have been struck between prominent companies and AI providers, smaller web publishers may benefit from a pay-to-crawl system without engaging in individual negotiations. However, CC emphasized the importance of responsible implementation of pay-to-crawl systems to prevent power concentration on the web and ensure continued access to content for various stakeholders.
Cloudflare is not the only company investing in pay-to-crawl technology. Microsoft is developing an AI marketplace for publishers, while startups like ProRata.ai and TollBit are also entering this space. Additionally, the RSL Collective introduced a new standard called Really Simple Licensing (RSL), which has gained support from industry players like Cloudflare, Akamai, and Fastly.
CC has endorsed the RSL standard as part of its broader project, CC signals, which aims to develop technology and tools for the AI era. By supporting responsible pay-to-crawl initiatives and open standards like RSL, CC is positioning itself at the forefront of shaping the future of content accessibility in the age of AI.
