Meta-ExternalFetcher
Meta · User-requested fetches
What Meta says it does
“fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities”
The facts
| Operator | Meta |
| robots.txt token | meta-externalfetcher |
| Appears in access logs | Yes |
| User-agent string | meta-externalfetcher/1.1 |
| robots.txt | May bypass robots.txt. Meta states it may bypass robots.txt because it performs fetches that were requested by the user. |
| IP list | Not published |
| Documentation | https://developers.facebook.com/docs/sharing/webmasters/web-crawlers, read 4 Aug 2026 |
Allow or block it in robots.txt
To block meta-externalfetcher entirely:
User-agent: meta-externalfetcher Disallow: /
To allow it everywhere:
User-agent: meta-externalfetcher Allow: /
Meta's stated position: may bypass robots.txt. Meta states it may bypass robots.txt because it performs fetches that were requested by the user. Blocking it reliably means blocking at the server, not in robots.txt.
Verify a request is really Meta-ExternalFetcher
Meta publishes no IP list for it on the crawler page.
Observed on this property
Not observed here yet. This site logs every request from 15 named AI agents; logging began 4 Aug 2026. When Meta-ExternalFetcher first fetches a page, this section fills in from the log.
Also run by Meta
- Meta-ExternalAgent | Trains AI models; indexing