Good news for websites hosted on CloudFlare. They have improved one step further to block AI bots.
They recently launched a toggle that shouts “Get out of here, you bots” but is labelled ‘AI Scrapers and Crawlers’.

This deserves appreciation. AI bots are everywhere, trying to access your content, and then without your permission, the companies that leverages these bots, use your content to train their models. Well, many companies try to make us believe, that they are not using our content without permission and are following rules. However, not all are honest with us. Every now and then you hear news, organizations suing AI pioneers on stealing their data.

Behind the scenes, the bots, what they do is that they follow the robot.txt files on websites, to see what they are allowed to scrape. Even then, most of the customers, as mentioned by Cloudflare, are not so okay with this too, because they just want to block the AI bots no matter what. So, being a customer-oriented firm, Cloudflare developed this feature. The best part – its available to everyone, even those on free-tier.

According to their report, Bytespider, operated by Bytedance (yeah, it owns TikTok), is at the top spot when you see the number of requests made by AI bots per day. The Chinese company uses this data to train their models including Doubao, said to be a ChatGPT rival. Amazonbot follows next, with its intent to index content for Alexa. (By the way how is Alexa catching up in this AI world? Long time, no see).

Not just that, Bytespider accessed around 40% of websites protected by Cloudflare. Of course, that also means, they are getting blocked the maximum. Interestingly, GPTBot, by OpenAI is second. I could not find Amazonbot in this list.

Cloudflare assures that their trained bot detection models are capable of identifying the bots that disguise themselves as real web browsers. This they achieve by analyzing traffic patterns and by assigning scores – which for instance, anything below 30, they could tell that these are bots.
Apart from that, they have forms where the Cloudflare customers can report any unusual activity.

This is a significant step especially when there are companies that, like I said earlier, have difficulty following the rules. They feel they are left behind if they don’t break rules. We do need resources like Cloudflare’s that could help prevent such issues

^{Featured image include photo by fcleetus and Cloudlare turnstile icon from Cloudlare developers site}