Everything looked normal in the SEO data. Google Search Console, traffic, and indexing — no red flags. Then I opened Scrunch, our AI citation monitoring tool, and looked at platform-by-platform presence for searchinfluence.com over the prior 30 days:
- Google AI Mode: 37.8%
- Copilot: 22.2%
- Google Gemini: 16.3%
- ChatGPT: 9.6%
- Perplexity: 7.8%
- Claude: 0.0%
- Meta AI: 0.0%
Two platforms at zero. Every crawler reads the same site, so content quality and topical authority can’t account for that gap. They’re identical for every platform on the list.
What varies is access — whether each platform’s crawler is allowed in. Nothing else explains how Google AI Mode hits 37.8% while Claude lands at 0%. So I opened the logs.
What 7 days of Cloudflare logs showed
Seven days of Cloudflare data (April 4-10) for searchinfluence.com revealed 29,099 bot requests, 65.8% of them AI bots. Here’s a per-bot share of those requests rate-limited (HTTP 429, “too many requests”), broken out by bot user-agent (UA, the identifier each request sends):
- Amazonbot: 51% rate-limited
- ClaudeBot: 29%
- GPTBot: 29%
- Bytespider: 61% blocked (different mechanism: 403/5xx, not 429)
- ChatGPT-User: 0%
- PerplexityBot: 0%
The split isn’t random. Training crawlers, the ones that pull whole sites in big bursts, get throttled. User-facing crawlers, the ones that fire human-paced requests during a live user query, don’t.
For context: Cloudflare’s Q1 2026 crawl-to-referral analysis shows ClaudeBot makes 20,583 crawl requests for every referral it sends back.
- GPTBot: 1,255 to 1.
- Perplexity: 111 to 1.
- Google: 5 to 1.
AI training crawlers take far more than they give back, so it makes sense that hosting infrastructure has started fighting back. Whether that’s the right fight for your site is a separate question.
The 429s in our logs were being passed through Cloudflare with a cache status of dynamic or bypass. So I wrote them off as downstream of Cloudflare, must be a web application firewall (WAF) or security plugin. That assumption sent me down a multi-hour rabbit hole through the wrong layers.
Your customers search everywhere. Make sure your brand shows up.
The SEO toolkit you know, plus the AI visibility data you need.
Start Free Trial
Get started with
Where we looked first, and why we were wrong
Suspect 1: Solid Security’s HackRepair default ban list
A WordPress security plugin we use for hardening, with a built-in bot UA blocklist. Toggled it off, ran a 24-hour before/after on per-bot 429 counts. No change.
Two bots even spiked higher in the post-toggle window. Coincidental crawl bursts, not a regression. Ruled out.
Suspect 2: Solid Security’s other firewall subsystems
24,538 firewall log entries over 30 days. Every single one was a /wp-login.php…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
