Cloud service provider Cloudflare has introduced a new, no-cost tool designed to prevent bots from scraping data from websites hosted on its platform for AI model training.
While some AI companies like Google, OpenAI, and Apple provide ways for website owners to block their bots through the robots.txt file, Cloudflare highlights that not all AI scrapers adhere to these rules. The company expressed concern on its blog that certain AI firms may continuously adapt to bypass detection measures to access content.
To tackle this issue, Cloudflare has refined its bot detection models by analyzing AI bot and crawler traffic. These models assess various factors, including attempts by AI bots to disguise their activity as legitimate web browsing.
"When bad actors attempt to crawl websites at scale, they generally use tools and frameworks that we are able to fingerprint," Cloudflare notes. Their advanced models can flag suspicious traffic from evasive AI bots.
Cloudflare has also provided a form for website owners to report suspected AI bot activity and plans to maintain a manual blacklist of AI bots over time.
AI bots on the rise
The surge in demand for training data due to the generative AI boom has made the problem of AI bots more pressing. Many websites have started blocking AI scrapers. Studies show that around 26% of the top 1,000 websites have blocked OpenAI's bot, and over 600 news publishers have taken similar actions.
Despite these efforts, blocking AI bots is not foolproof. Some companies ignore standard bot exclusion protocols to gain an edge in the competitive AI landscape. For instance, AI search engine Perplexity was accused of mimicking legitimate users to scrape content, and both OpenAI and Anthropic have reportedly bypassed robots.txt rules.TollBit, a content licensing startup, recently informed publishers that many AI agents disregard the robots.txt standard.
Tools like Cloudflare's could provide a solution by accurately identifying and blocking stealthy AI bots. Still, this may not address the broader issue of publishers potentially losing referral traffic from AI tools like Google’s AI Overviews if they block specific AI crawlers.
