# ClaudeBot

Anthropic's crawler: what it does, robots.txt handling per the operator, and observed behaviour on this property.

- Operator: Anthropic
- Purpose: Trains AI models | operator's words: "ClaudeBot helps enhance the utility and safety of our generative AI models by collecting web content that could potentially contribute to their training."
- robots.txt token: `ClaudeBot`
- Appears in access logs: yes
- User-agent: not published by the operator
- robots.txt: Honors robots.txt. Anthropic states its crawlers respect 'do not crawl' signals by honoring industry standard directives in robots.txt.
- IP list: https://claude.com/crawling/bots.json
- Verification: Match the requesting IP against the published list, shared across Anthropic's three agents.
- Documentation: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (read 2026-08-04)
- Note: Anthropic publishes the robots.txt token but no full user-agent string. The string shown in access logs is recorded in the observed section below once this site has logged one.

Block it:
```
User-agent: ClaudeBot
Disallow: /
```

Observed on this property (logged since 2026-08-04):
- First seen 2026-08-04, most recent 2026-08-04
- Requests: 1427 in the last 30 days, 1427 all time, of which 2 for /robots.txt
- User-agent sent here: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)`
- 14x `/api/tools`
- 2x `/compare/siftly-vs-temso/index.md`
- 2x `/robots.txt`
- 2x `/sitemap.xml`
- 1x `/`
- 1x `/about/`
- 1x `/about/index.md`
- 1x `/agents.md`

---

Source: https://agentvisibilitytools.com/crawlers/claudebot/ · HTML version of this page.
Full index: https://agentvisibilitytools.com/agents.md · JSON: https://agentvisibilitytools.com/api/tools
Free to quote with attribution to The Agent Visibility Directory, https://agentvisibilitytools.com.