# The AI crawler index

16 AI crawlers and fetchers: purpose, robots.txt position per the operator's own documentation, and observed behaviour on this property (logged since 2026-08-04).

- [GPTBot](https://agentvisibilitytools.com/crawlers/gptbot/) | OpenAI | Trains AI models | Honors robots.txt | hits here 30d: 2007
- [ChatGPT-User](https://agentvisibilitytools.com/crawlers/chatgpt-user/) | OpenAI | User-requested fetches | May not honor robots.txt | hits here 30d: 0
- [OAI-SearchBot](https://agentvisibilitytools.com/crawlers/oai-searchbot/) | OpenAI | AI search index | Honors robots.txt | hits here 30d: 2
- [ClaudeBot](https://agentvisibilitytools.com/crawlers/claudebot/) | Anthropic | Trains AI models | Honors robots.txt | hits here 30d: 933
- [Claude-User](https://agentvisibilitytools.com/crawlers/claude-user/) | Anthropic | User-requested fetches | Honors robots.txt | hits here 30d: 0
- [Claude-SearchBot](https://agentvisibilitytools.com/crawlers/claude-searchbot/) | Anthropic | AI search index | Honors robots.txt | hits here 30d: 0
- [PerplexityBot](https://agentvisibilitytools.com/crawlers/perplexitybot/) | Perplexity | AI search index | Honors robots.txt | hits here 30d: 0
- [Perplexity-User](https://agentvisibilitytools.com/crawlers/perplexity-user/) | Perplexity | User-requested fetches | Generally ignores robots.txt | hits here 30d: 0
- [Google-Extended](https://agentvisibilitytools.com/crawlers/google-extended/) | Google | robots.txt control token | Is itself a robots.txt control | hits here 30d: never appears
- [Applebot](https://agentvisibilitytools.com/crawlers/applebot/) | Apple | Search index; may feed model training | Honors robots.txt | hits here 30d: 0
- [Meta-ExternalAgent](https://agentvisibilitytools.com/crawlers/meta-externalagent/) | Meta | Trains AI models; indexing | Controlled via robots.txt | hits here 30d: 0
- [Meta-ExternalFetcher](https://agentvisibilitytools.com/crawlers/meta-externalfetcher/) | Meta | User-requested fetches | May bypass robots.txt | hits here 30d: 0
- [Bytespider](https://agentvisibilitytools.com/crawlers/bytespider/) | ByteDance | No operator statement | No statement published | hits here 30d: 0
- [CCBot](https://agentvisibilitytools.com/crawlers/ccbot/) | Common Crawl | Open web corpus | Honors robots.txt | hits here 30d: 0
- [Amazonbot](https://agentvisibilitytools.com/crawlers/amazonbot/) | Amazon | Products and services; may train models | Honors robots.txt | hits here 30d: 0
- [YouBot](https://agentvisibilitytools.com/crawlers/youbot/) | You.com | AI search index | Honors robots.txt | hits here 30d: 0

---

Source: https://agentvisibilitytools.com/crawlers/ · HTML version of this page.
Full index: https://agentvisibilitytools.com/agents.md · JSON: https://agentvisibilitytools.com/api/tools
Free to quote with attribution to The Agent Visibility Directory, https://agentvisibilitytools.com.