ClaudeBot
Anthropic · Trains AI models
What Anthropic says it does
“ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training.”
The facts
| Operator | Anthropic |
| robots.txt token | ClaudeBot |
| Appears in access logs | Yes |
| User-agent string | Not published by the operator |
| robots.txt | Honors robots.txt. Anthropic states its crawlers respect 'do not crawl' signals by honoring industry standard directives in robots.txt. |
| IP list | https://claude.com/crawling/bots.json |
| Documentation | https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler, read 4 Aug 2026 |
Allow or block it in robots.txt
To block ClaudeBot entirely:
User-agent: ClaudeBot Disallow: /
To allow it everywhere:
User-agent: ClaudeBot Allow: /
Verify a request is really ClaudeBot
Match the requesting IP against the published list, shared across Anthropic's three agents.
Anthropic publishes the robots.txt token but no full user-agent string. The string shown in access logs is recorded in the observed section below once this site has logged one.
Observed on this property
| First seen here | 4 Aug 2026 |
| Most recent | 4 Aug 2026 |
| Requests, last 30 days | 314 |
| Requests, all time | 314 |
| robots.txt fetches | 2 |
User-agent string it sent here:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)
Most requested paths:
/api/tools(2)/robots.txt(2)/sitemap.xml(2)/(1)/about/(1)/agents.md(1)/ai-visibility-tools-under-100/(1)/alternatives/aiclicks/(1)
Also run by Anthropic
- Claude-User | User-requested fetches
- Claude-SearchBot | AI search index