Claude-SearchBot
Anthropic · AI search index
What Anthropic says it does
“Claude-SearchBot navigates the web to improve search result quality for users. It analyzes online content specifically to enhance the relevance and accuracy of search responses.”
The facts
| Operator | Anthropic |
| robots.txt token | Claude-SearchBot |
| Appears in access logs | Yes |
| User-agent string | Not published by the operator |
| robots.txt | Honors robots.txt. Anthropic states its crawlers respect 'do not crawl' signals by honoring industry standard directives in robots.txt. |
| IP list | https://claude.com/crawling/bots.json |
| Documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, read 4 Aug 2026 |
Allow or block it in robots.txt
To block Claude-SearchBot entirely:
User-agent: Claude-SearchBot Disallow: /
To allow it everywhere:
User-agent: Claude-SearchBot Allow: /
Verify a request is really Claude-SearchBot
Match the requesting IP against the published list, shared across Anthropic's three agents.
Observed on this property
Not observed here yet. This site logs every request from 15 named AI agents; logging began 4 Aug 2026. When Claude-SearchBot first fetches a page, this section fills in from the log.
Also run by Anthropic
- ClaudeBot | Trains AI models
- Claude-User | User-requested fetches