Today’s question looks beyond the typical traffic-driving goals of AI visibility to the value those large language models provide a website owner, and asks:
“AI crawlers are visiting my website increasingly often, but I can’t tell whether they provide any value. Should I allow them, block them, or treat different AI crawlers differently? How can I measure whether their activity leads to citations, referral traffic, or conversions before making that decision?”
Many SEOs don’t realize the cost of having bots visit their site. Recently, with the proliferation of AI bots, the costs of allowing anyone and everyone to access your content are becoming an expensive business.
Types Of AI Crawlers
First, let’s look at the different types of bots that visit a website.
Common bots that will be visiting a website regularly include those we want to have access to our site, for example, search engine bots. These aren’t the only bots, but they are often some of the most prolific consumers of bandwidth. Alongside search bots, there will be tools. These can include bots from uptime monitors, search and analytics tools, and security and vulnerability scanners.
Overall, website owners have to decide whether the bots visiting their site should be allowed to continue or if they pose more harm than good. Examples of bots that site managers often block are those that are trying to scrape product information to feed another website’s database, or malicious bots looking for login vulnerabilities. Whether or not to block these bots is a fairly easy decision – they pose a risk to the intellectual property of the brand or the safety of the website.
AI bots might actually fall somewhere in between these “good” and “bad” bots.
AI Training Bots
These bots, for example, OpenAI’s GPTBot, are scouring the web for information to feed the AI training models. They are helping to create the knowledge base that the LLMs are learning from, including entities and how they relate to each other.
For many website owners, these are the most controversial AI crawlers. Their primary purpose is not to send traffic back to your site, but to “read” and collect information that may be used to train and improve models. In some cases, that content may later be used to answer user questions without generating a visit to the original source. This makes it harder to draw a direct line between the crawler’s activity and business value.
Search Indexing Bots
These bots, OpenAI’s OAI-SearchBot, for example, are reviewing pages and collecting information to surface and link websites in LLM “search results,” not to train foundation models.
These are often easier to justify allowing because their purpose is closer to that of a traditional search engine. If they are indexing your content so that it can be cited in AI-generated answers, they have a more obvious route to creating visibility, referral traffic, and brand awareness.
User-Triggered Fetches
These bots,…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
