Cloudflare, a cloud-based internet infrastructure company, recently released its 2025 internet trends report, which can offer insights into how traditional search and browsing behaviour is evolving as AI bots are slowly becoming the go-to resource, not only in everyday use but also in professional workflows.
To assess AI’s impact on internet search and bot crawling behaviour, MediaNama examined key data points from Cloudflare’s report. The following points highlight the most significant aspects of the report, which are further explained in detail along with their relevance.
Some Key Takeaways from the Cloudflare report:
- Anthropic showed the sharpest crawl-to-refer imbalance, peaking near 500,000:1, highlighting how AI product design can prioritize data extraction and on-platform engagement over directing users to original sources.
- Dual-purpose crawling has become standard among Big Tech, with Googlebot and Bingbot used for both search indexing and AI training, despite Cloudflare’s call for single-purpose bots.
- AI model training continued to dominate crawler traffic in 2025, reaching up to eight times the volume of search crawling and 32 times that of user-triggered crawling, driven largely by GPTBot. At the same time, user-action AI bots grew the fastest, with crawling tied to user queries increasing more than 15-fold over the year.
- Google’s bots remained the largest traffic source on Cloudflare’s network for a third straight year, followed by OpenAI among verified AI bots, while Bingbot recorded lower but steady activity.
- AI crawling concentrated heavily in retail and software, with retail alone accounting for more than 25% of all crawling activity.
- Bot-led traffic skewed toward smaller jurisdictions, while India recorded relatively low bot-led activity compared to the US, China and Russia.
- India remains mobile-first, with nearly 65% of traffic coming from mobile devices, far higher than major developed markets, like the US and Russia, etc.
- Satellite internet traffic surged globally, with Starlink traffic more than doubling in 2025 as it expanded into new regions.
The Crawl-to-Refer Ratio of AI Bots
Crawl-to-refer ratio measures how much automated crawling by AI and search platforms translates into actual human traffic being sent back to the original websites. The metric can be inherently volatile as it fluctuates with daily changes in crawling and browsing behaviour, but it provides a useful lens to assess how different platforms are interacting with the open web– which various AI companies use as a defence for their unethical practices.
Among the popular AI companies, Anthropic showed the most extreme imbalance, with crawl-to-refer ratios peaking near 500,000:1 early in the year before settling at still-elevated levels between 25,000:1 and 100,000:1, driven largely by minimal referral traffic.
It means that for every one visit sent back to an originally crawled source website,…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
