Claude-SearchBot
Anthropic · AI search index
What Anthropic says it does
“Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.”
The facts
| Operator | Anthropic |
| robots.txt token | Claude-SearchBot |
| Appears in access logs | Yes |
| User-agent string | Not published by the operator |
| robots.txt | Honors robots.txt. Anthropic states its crawlers respect 'do not crawl' signals by honoring industry standard directives in robots.txt. |
| IP list | https://claude.com/crawling/bots.json |
| Documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, read 4 Aug 2026 |
Allow or block it in robots.txt
To block Claude-SearchBot entirely:
User-agent: Claude-SearchBot Disallow: /
To allow it everywhere:
User-agent: Claude-SearchBot Allow: /
Verify a request is really Claude-SearchBot
Match the requesting IP against the published list, shared across Anthropic's three agents.
Observed on this property
| First seen here | 26 Aug 2026 |
| Most recent | 13 Sep 2026 |
| Requests, last 30 days | 74 |
| Requests, all time | 74 |
| robots.txt fetches | 0 |
User-agent string it sent here:
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:133.18) Gecko/20100101 Firefox/133.18; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot
Most requested paths:
/admin/.env(2)/api/config.json(2)/config/aws.json(2)/id_dsa/(2)/proxy/(2)/amplifyconfiguration.json(1)/api/graphql(1)/api-keys.json(1)
Also run by Anthropic
- ClaudeBot | Trains AI models
- Claude-User | User-requested fetches