Google’s John Mueller and Martin Splitt talked about LLMs.txt and markdown, with Mueller offering a surprising fact about the original purpose of LLMs.txt and also explaining why the proposed standards are have severe shortcomings.
What Discovery Is And Why It Matters
In the context of information retrieval (search), discovery is about a search engine discovering that a specific web page exists. Discovery is a part of the overall search engine architecture.
Search Engine Architecture:
- Discovery
Discovering the URL (adding it to the crawl). - Crawling
Downloading and parsing the content. - Indexing
The process of analyzing the raw data and storing it in a structured database optimized for retrieval. - Ranking
The part that everyone’s interested in. - Serving
This is the last step which is serving the ranked web pages in the search results.
The above is a simplified overview of what search is and Discovery is the very first part of the process that eventually ends with ranking and serving links to websites.
The takeaway here is that Discovery is a critical part of getting a web page queued for crawling, indexed, ranked, and eventually shown in the search results. Without Discovery a web page is invisible.
Now here is why this is important: Discovery is not a part of the proposed LLMs.txt standard. use
Original Intent Of LLMs.txt
John Mueller said that he met one of the people responsible for creating the LLMs.txt proposal and said that the creator explained that LLMs.txt was never about making a site discoverable, it was never meant to be a part of that process.
This is an important point because many site owners are spending time, money, and effort generating LLMs.txt for the purpose of getting discovered and ranked in LLMs. That means that the reason people are using LLMs.txt is in conflict with the actual purpose of LLMs.txt, which has nothing to do with Discovery.
Mueller explained:
“So I talked with, I think, one of the people who created that proposal a while back. And the idea was really not to create something that makes it easier for search engines or LLM systems to discover all of your content, but almost more that if an LLM already knows about your site and wants to find out what else is here, then that might be an approach.
And I think the aspect of using this as a way to optimize for Discovery by AI systems or Discovery by search systems, that doesn’t make any sense at all.”
Mueller next explained that many people are using LLMs.txt in the hope of aiding the process of Discovery despite the fact that’s not the purpose of LLMs.txt.
He then pivoted to the fact that LLMs.txt are inherently untrustworthy because it’s a site owner saying what their site’s content is about, which may or may not match what’s in the actual HTML.
He continued:
“Because it’s basically you’re telling these systems, like, I have the best website ever. And here are all of the pages that everyone must go to. And you must buy all of my products or…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
