A new analysis of 858,457 sites hosted on the Duda platform shows how AI crawlers are interacting with websites at scale. The data offers a clearer view of how crawling activity is growing and what SEOs and businesses should do to increase traffic from AI search.
AI Crawling Has Already Reached Scale
AI crawling is growing quickly, with more requests tied to real-time answers and most of that activity coming from a single provider. The data creates a pattern that shows which sites are being crawled and more importantly, why.
Year-Over-Year Growth In LLM Referrals
LLM referral traffic has increased sharply over the past year, with multiple platforms showing meaningful gains from very different starting points.
AI Referral Traffic Patterns
- Total LLM referrals: 93,484 to 161,469 (+72.7%)
- ChatGPT: 81,652 to 136,095 (+66.7%)
- Claude: 106 to 2,488 (23x growth)
- Copilot: 22 to 9,560 (from near-zero)
- Perplexity: 11,533 to 13,157 (+14.1%)
Growth is not happening evenly, but across the board, referral traffic from AI systems is increasing. That makes AI-generated discovery a growing source of traffic, not a marginal one.
Crawlers Are Increasingly Fetching Content To Ground Answers
AI crawlers are no longer used primarily for indexing, with most activity now tied to retrieving content in real time to generate answers for users.
Most crawling is now happening in response to user queries rather than for building an index, which changes how content is accessed and used.
- User Fetch (real-time answers): 56.9% of all crawler activity, driven almost entirely by ChatGPT
- Training (model learning): 28.8%, split across GPTBot and other model crawlers
- Discovery (content indexing): 14.3%, distributed across multiple systems
- ChatGPT User Fetch volume: ~39.8 million visits
The trends are largely driven by ChatGPT, which is responsible for nearly all real-time retrieval activity. That means the move toward answer-based crawling is not evenly distributed, but concentrated in one platform shaping how content is accessed. This trend may change with Google’s new Google-Agent crawler.
Market Concentration In AI Crawling
AI crawler activity is heavily concentrated, with OpenAI responsible for the vast majority of requests, reflecting its position as the primary tool users rely on to find and retrieve information.
- OpenAI: 55.8 million visits (81.0%)
- Anthropic (Claude): 11.5 million (16.6%)
- Perplexity: 1.3 million (1.8%)
- Google (Gemini): 380,000 (0.6%)
Most AI crawling activity comes from OpenAI, which aligns with ChatGPT’s role as a primary tool for finding and retrieving information. Claude follows at a much smaller share, suggesting a different usage pattern, while the rest of the market accounts for a minimal portion of crawler activity.
Scale And What That Actually Means
AI crawling is already operating across a large portion of the web, reaching hundreds of thousands of sites and generating tens of millions of requests in a single month.
More than half of all…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
