How AI search responds when you make a site more crawlable

Crawl waste – basically, cluttering your crawlable content with tons of non-indexed pages – is one of the most insidious SEO pitfalls.

We know (because we do it all the time for my agency’s clients) that reducing the number of non-indexed pages and highlighting richer, authoritative content is a factor in improving rankings and visibility in traditional SERP listings.

But how does it impact AI search visibility? And were the results different by platform? 

Thanks to some recent cleanup and data-crunching (we used Profound to measure changes in AI search visibility) for a client in the childcare vertical, we’ve got some directional answers and takeaways that I’ll lay out in this article. 

Yes, it’s just one website, but with AI moving at light speed, having promising results that need further validation is a valuable first step.

Crawl waste cleanup: What we did

Sitemaps should only feature canonical versions of pages that a brand wants indexed in search results.

The childcare client’s site had 2,000 indexed pages and ~45,000 pages that were not being indexed – including 4,000 duplicate and redirected URLs incorrectly listed in the sitemap. (A sitemap should only contain canonical URLs you want Google to find and index.) 

The site also contained low-quality directory listings, all of which pointed to poor health.

We optimized the sitemap and indexing strategy by reducing the number of duplicate and redirected URLs in the sitemap from more than 4,000 to fewer than 200. 

This cleanup improved crawl efficiency, helping search engines and AI systems focus on the client’s high-quality, intent-aligned content.

Dig deeper. Crawl efficacy: How to level up crawl optimization

How the AI search platforms responded

The topline:

• This initiative lifted visibility across every AI search platform – up to 10% in some cases – pushing overall visibility above 90% for Perplexity, AI Mode, AI Overviews, Gemini, Grok, and Meta AI. (We calculate visibility by dividing the number of times the brand appeared in prompt results by the total number of relevant prompts.)
• The one exception was ChatGPT, which saw a 4.1% drop. My take is that ChatGPT can be slower to absorb the latest online information unless users enable web access and browsing. Because the cleanup involved major indexing and technical changes, ChatGPT may have lagged compared to faster adapters like Perplexity and Google.

The entire Google suite of AI tools responded strongly to the cleanup. 

That result – while just one example – aligns with Google’s repeated message that what benefits traditional SEO also benefits AI search.

Even with ChatGPT’s decline, overall visibility still grew by 3.5%. 

It isn’t a massive jump, but it signals an upward trajectory that helps the brand gain ground on competitors. 

In…