Google-Extended
Google · robots.txt control token
What Google says it does
“training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy)”
The facts
| Operator | |
| robots.txt token | Google-Extended |
| Appears in access logs | Never. It is a control token, not a crawler; pages are fetched under other user agents. |
| User-agent string | Not published by the operator |
| robots.txt | Is itself a robots.txt control. Google's own words: Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity. |
| IP list | Not published |
| Documentation | https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers, read 4 Aug 2026 |
Allow or block it in robots.txt
To block Google-Extended entirely:
User-agent: Google-Extended Disallow: /
To allow it everywhere:
User-agent: Google-Extended Allow: /
Verify a request is really Google-Extended
It never appears in an access log. Pages are fetched by Google's normal crawlers; this token only controls what the fetched content may be used for.
If you are grepping your logs for Google-Extended you will not find it, and that is expected. Disallowing it in robots.txt is the entire mechanism.
Observed on this property
Never, by design. It never appears in an access log. Pages are fetched by Google's normal crawlers; this token only controls what the fetched content may be used for.