On March 20, 2026, Google quietly added a new entry to its official list of web fetchers. Not a crawler. Not a training bot. An agent.
Google-Agent is the user agent string for AI systems running on Google infrastructure that browse websites on behalf of users. When someone asks an AI assistant to research a product, fill out a form, or compare options across websites, Google-Agent is the thing that actually visits the page. Project Mariner, Google’s experimental AI browsing tool, is the first product using it.
This is not Googlebot. Googlebot crawls the web continuously, indexing pages for search. Google-Agent only shows up when a human asks it to. That distinction changes everything about how it operates.
Robots.txt Does Not Apply
Google classifies Google-Agent as a user-triggered fetcher. The category includes tools like Google Read Aloud (text-to-speech), NotebookLM (document analysis), and Feedfetcher (RSS). All of them share one property: a human initiated the request. Google’s position is that user-triggered fetchers “generally ignore robots.txt rules” because the fetch was requested by a person.
The logic: If you type a URL into Chrome, the browser fetches the page regardless of what robots.txt says. Google-Agent operates on the same principle. The agent is the user’s proxy, not an autonomous crawler.
This is a meaningful departure from how OpenAI and Anthropic handle similar traffic. ChatGPT-User and Claude-User both function as user-triggered fetchers, but they respect robots.txt directives. If you block ChatGPT-User in robots.txt, ChatGPT won’t fetch your page when a user asks it to browse. Google made a different call.
Website owners who relied on robots.txt as a universal access control mechanism now have a gap. If you need to restrict access from Google-Agent, you’ll need server-side authentication or access controls. The same tools you’d use to block a human visitor.
Cryptographic Identity: Web Bot Auth
The more significant development is buried in a single line of Google’s documentation: Google-Agent is experimenting with the web-bot-auth protocol using the identity https://agent.bot.goog.
Web Bot Auth is an IETF draft standard that works like a digital passport for bots. Each agent holds a private key, publishes its public key in a directory, and cryptographically signs every HTTP request. The website verifies the signature and knows, with cryptographic certainty, that the visitor is who it claims to be.
User agent strings can be spoofed by anyone. Web Bot Auth cannot. Google adopting this protocol, even experimentally, signals where agent identity is heading. Akamai, Cloudflare, and Amazon (AgentCore Browser) already support it. Google brings the critical mass.
This matters because the web is about to have an identity problem. As agent traffic increases, websites need to distinguish between legitimate AI agents acting on behalf of real users and scrapers pretending to be agents. IP verification…
Source link
Disclaimer
We strive to uphold the highest ethical standards in all of our reporting and coverage. We blogs.grocliq.com want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.
Website Upgradation is going on for any glitch kindly connect at [email protected]
