# CCBot

Common Crawl's crawler: what it does, robots.txt handling per the operator, and observed behaviour on this property.

- Operator: Common Crawl
- Purpose: Open web corpus | operator's words: "Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone."
- robots.txt token: `CCBot`
- Appears in access logs: yes
- User-agent: `CCBot/2.0 (https://commoncrawl.org/faq/)`
- robots.txt: Honors robots.txt. Common Crawl's page gives the exact robots.txt lines to block it, which is the mechanism it commits to honoring.
- IP list: https://index.commoncrawl.org/ccbot.json
- Verification: Reverse DNS resolves inside *.crawl.commoncrawl.org, or match the IP against the published list.
- Documentation: https://commoncrawl.org/ccbot (read 2026-08-04)
- Note: The corpus is public, so blocking or allowing CCBot decides whether a site is in the dataset anyone downstream can build on. That reach is why it appears in AI-crawler discussions despite being older than all of them.

Block it:
```
User-agent: CCBot
Disallow: /
```

Observed on this property (logged since 2026-08-04):
- Not observed here yet.

---

Source: https://agentvisibilitytools.com/crawlers/ccbot/ · HTML version of this page.
Full index: https://agentvisibilitytools.com/agents.md · JSON: https://agentvisibilitytools.com/api/tools
Free to quote with attribution to The Agent Visibility Directory, https://agentvisibilitytools.com.