# Google-Extended

Google's robots.txt control token: what it does, robots.txt handling per the operator, and observed behaviour on this property.

- Operator: Google
- Purpose: robots.txt control token | operator's words: "training future generations of Gemini models that power Gemini Apps and Vertex AI API for Gemini and for grounding (providing content from the Google Search index to the model at prompt time to improve factuality and relevancy)"
- robots.txt token: `Google-Extended`
- Appears in access logs: never (control token only; pages are fetched under other user agents)
- User-agent: not published by the operator
- robots.txt: Is itself a robots.txt control. Google's own words: Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity.
- IP list: not published
- Verification: It never appears in an access log. Pages are fetched by Google's normal crawlers; this token only controls what the fetched content may be used for.
- Documentation: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers (read 2026-08-04)
- Note: If you are grepping your logs for Google-Extended you will not find it, and that is expected. Disallowing it in robots.txt is the entire mechanism.

Block it:
```
User-agent: Google-Extended
Disallow: /
```

Observed on this property (logged since 2026-08-04):
- Never appears in logs by design.

---

Source: https://agentvisibilitytools.com/crawlers/google-extended/ · HTML version of this page.
Full index: https://agentvisibilitytools.com/agents.md · JSON: https://agentvisibilitytools.com/api/tools
Free to quote with attribution to The Agent Visibility Directory, https://agentvisibilitytools.com.