Anthropic updated its crawler documentation this week, clarifying how its Claude bots access websites and how you can block them.
- Anthropic’s document explains what each bot does, how it affects AI training and search visibility, and how to opt out through robots.txt.
Why we care. If you publish or own content, you want control over how AI systems use it. Anthropic separates training crawlers, user-triggered fetches, and search indexing. Blocking one bot doesn’t block the others. Each choice carries different visibility and training trade-offs.
The robots. Anthropic uses three separate user agents:
- ClaudeBot collects public web content that may be used to train and improve Anthropic’s generative AI models. If you block ClaudeBot in robots.txt, Anthropic said it will exclude your site’s future content from AI training datasets.
- Claude-User retrieves content when a user asks Claude a question that requires access to a webpage. If you block Claude-User, Anthropic can’t fetch your pages in response to user queries. The company says this may reduce your visibility in user-directed search responses.
- Claude-SearchBot crawls content to improve the quality and relevance of Claude’s search results. If you block Claude-SearchBot, Anthropic won’t index your content for search optimization, which may reduce visibility and accuracy in Claude-powered search answers.
How to block them. The bots respect standard robots.txt directives, including “Disallow” rules and the non-standard “Crawl-delay” extension, Anthropic said. To block a bot across your entire site:
User-agent: ClaudeBot
Disallow: /
- You must add directives for each bot and each subdomain you want to restrict.
- IP blocking may not work reliably because its bots use public cloud provider IP addresses, Anthropic said. Blocking those ranges could prevent the bot from accessing robots.txt. The company doesn’t publish IP ranges.
The document. Does Anthropic crawl data from the web, and how can site owners block the crawler?
Search Engine Land is owned by Semrush. We remain committed to providing high-quality coverage of marketing topics. Unless otherwise noted, this page’s content was written by either an employee or a paid contractor of Semrush Inc.
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
