Google’s John Mueller answered a question about the curious circumstance of Search Console reporting thousands of URLs as indexed despite being blocked by robots.txt. Mueller helped explain how this happens and what to do about it.
Content Indexed Despite Being Blocked By Robots.txt
A Redditor asked for advice because Google Search Console was reporting more than 51,000 pages under the status “Indexed, though blocked by robots.txt.” The affected URLs were primarily WooCommerce product URLs containing add-to-cart URL parameters like “?add-to-cart=”.
Because the issue appeared suddenly, the site owner questioned whether the robots.txt rules themselves were responsible for creating the problem. They also wanted to know whether removing the rules would help Google process the canonical signals and eliminate the reported URLs from Search Console.
The person asked:
“I have WooCommerce site and suddenly since past month we are facing this issue: “Indexed, though blocked by robots.txt”
there are total “Affected pages 51K pages”
in the end of url I see mostly ?page&post_type=product&product=slug&add-to-cart=98063,
After inspecting those urls I found they have index tag setup and robots.txt had
* Disallow: /*?add-to-cart=
* Disallow: /*?*add-to-cart=I removed those two rules from robots.txt and hoping those pages fixed cause they have canonical set to correct product, will that fix issue?
or should I also setup noindex rules? will that cause us our crawl budget? it is pretty big woocommerce site, let me know guys your thoughts if someone has experience fixing such issue? and what will be the right method without preventing our SEO or functionality loss.”
Google Says Add-To-Cart URLs Don’t Need To Be Indexed
Mueller responded that the add-to-cart URLs do not need to be indexed and that blocking them through robots.txt is an acceptable approach.
He explained that even when Google reports those URLs as indexed, they are unlikely to appear in normal search results because they are blocked by robots.txt. According to Mueller, users generally do not search for those URLs directly, making them poor candidates for search visibility.
John Mueller responded:
“You don’t need the add-to-cart URLs indexed. Blocking them with robots.txt is fine. Even if they get “indexed” since they’re blocked by robots.txt, it’s unlikely that they’ll be shown in search (unless you do specific queries for those URLs, which users don’t do).”
I’m kind of on the fence about what Mueller said about “robots.txt” making it “unlikely” that the URLs will be shown in Search. The reason is because robots.txt does not prevent a web page from showing in Google Search. It just prevents Googlebot from crawling those pages. So technically, that’s not quite correct and I’m a little surprised Mueller would say that.
Noindex Is Probably Not A Solution
One of the Redditors who responded to that question suggested the solution of adding a noindex…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
