Google’s John Mueller argued that LLM systems can’t use files like llms.txt to decide which websites to surface for a given query.
He made the comments on a recent episode of Search Off the Record, the podcast from Google’s Search Relations team.
His comment points to a broader signal problem, not just intentional gaming. Even a well-written llms.txt file is still self-reported information from the site that wants to be chosen.
For discovery, Mueller pointed back to normal HTML pages and internal links.
What Mueller Said
The conversation started with a question about whether publishers should convert websites to Markdown for LLMs. Mueller and co-host Martin Splitt agreed that HTML is still the foundation for crawling and discovery.
The discussion got specific when Mueller turned to llms.txt. He described the discovery use case as a dead end:
“It’s basically you’re telling these systems, like, I have the best website ever. And here are all of the pages that everyone must go to. And you must buy all of my products or whatever you put in there. So in LLM system, it basically, by design, can’t trust what is here as a way of differentiating between different websites.”
His argument comes down to differentiating. If sites use llms.txt to promote themselves, the files can make similar claims. An LLM deciding which site best answers a query still needs another way to differentiate between them.
What ‘By Design’ Might Mean
“By design” could mean two different things, and Mueller didn’t clarify which.
One reading is architectural. LLM systems evaluate web content and can’t use self-reported files when picking sources.
The other reading treats it as a signal problem. Self-reported signals lose value when everyone provides them. Meta keywords stopped working for the same reason. Every site stuffed them, and search engines couldn’t extract a useful ranking signal.
Both readings reach the same conclusion on discovery. But they imply different things about whether the limitation could change over time.
Where Mueller Sees A Role
Mueller didn’t reject all uses of llms.txt. He carved out one case where it could help:
“If someone is already on your website, maybe some kind of automated system is helpful.”
He used the example of an agent trying to buy a photograph from a specific site. The LLM would visit the site and look for instructions on how to complete the purchase.
The argument splits discovery from navigation. llms.txt can’t help an LLM choose which site to visit. But it could help once the agent is already there, like a store directory for someone who already walked in.
Beyond The Gaming Argument
Mueller has called building Markdown pages for bots “a stupid idea”. He’s also compared llms.txt to the keywords meta tag.
SEJ’s Roger Montti wrote that llms.txt is “inherently untrustworthy” because nothing stops site owners from adding self-serving content. SE Ranking’s analysis of 300,000 domains found no…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
