CCBot
Common Crawl · Open web corpus
What Common Crawl says it does
“Common Crawl is a non-profit foundation founded with the goal of democratizing access to web information by producing and maintaining an open repository of web crawl data that is universally accessible and analyzable by anyone.”
The facts
| Operator | Common Crawl |
| robots.txt token | CCBot |
| Appears in access logs | Yes |
| User-agent string | CCBot/2.0 (https://commoncrawl.org/faq/) |
| robots.txt | Honors robots.txt. Common Crawl's page gives the exact robots.txt lines to block it, which is the mechanism it commits to honoring. |
| IP list | https://index.commoncrawl.org/ccbot.json |
| Documentation | https://commoncrawl.org/ccbot, read 4 Aug 2026 |
Allow or block it in robots.txt
To block CCBot entirely:
User-agent: CCBot Disallow: /
To allow it everywhere:
User-agent: CCBot Allow: /
Verify a request is really CCBot
Reverse DNS resolves inside *.crawl.commoncrawl.org, or match the IP against the published list.
The corpus is public, so blocking or allowing CCBot decides whether a site is in the dataset anyone downstream can build on. That reach is why it appears in AI-crawler discussions despite being older than all of them.
Observed on this property
Not observed here yet. This site logs every request from 15 named AI agents; logging began 4 Aug 2026. When CCBot first fetches a page, this section fills in from the log.