Common Crawl review: the raw web dataset behind the best AI datasets
Answer in brief: Common Crawl is the raw material behind most open LLM pretraining datasets. It holds petabytes of web data collected regularly since 2008, with new crawls throughout 2026 up to CC-MAIN-2026-43. Access through Amazon's hosting is free. FineWeb, FineWeb-2, Dolma's web portion and LAION-BVD's video URLs were all built from it. Pick it when you want the freshest web data and full control over filtering. Pick FineWeb or Dolma when you want that work already done.
Best for: teams building their own web-scale pretraining corpus from the newest crawls.
Independent research from Common Crawl's official site and the AWS registry, checked 2026-11-09. Not hands-on testing.
What it does for AI datasets
The corpus contains raw web page data, metadata extracts and text extracts. It is stored on Amazon Web Services' Public Data Sets and on several academic cloud platforms. You can run analysis jobs on it inside AWS or download it in whole or in part. To find specific pages, use the URL Index and CDXJ Index. Common Crawl also publishes web graphs, crawl and graph statistics, an errata page and an opt-out ledger. The AWS registry describes the corpus as over 300 billion web pages.
The datasets built from it, by task
- English LLM pretraining: FineWeb and FineWeb-Edu, cleaned and deduplicated from Common Crawl dumps.
- Multilingual pretraining: FineWeb-2, split by language and script.
- Mixed-source pretraining: Dolma, whose largest source is Common Crawl processed with the Dolma pipeline.
- Video and multimodal: LAION-BVD started from Common Crawl WAT files to extract 1.3B video URLs.
Who it's built for
Pretraining teams with data-engineering capacity, researchers studying the web, and anyone who needs crawls newer than the cutoffs of ready-made corpora. FineWeb's newest per-crawl subset is CC-MAIN-2025-26, and Dolma's sources mostly stop in 2023.
Where it shines
- Freshness. Ten crawls are listed for 2026 so far, from CC-MAIN-2026-04 to CC-MAIN-2026-43.
- Scale. Petabytes, going back to 2008.
- Tooling. Indexes, web graphs, statistics, example projects and a Get Started guide.
- Transparency. A public opt-out ledger and errata page.
Honest drawbacks
- It's raw. You need extraction, language ID, quality filtering, deduplication, PII handling and safety processing before training.
- Processing petabytes takes serious compute and storage.
- Common Crawl's terms of use apply, and rights in the underlying pages vary.
Pricing and licenses
Access to the corpus hosted by Amazon is free; you pay for your own compute and any egress. Use is governed by Common Crawl's terms of use, and corpora built from it, such as FineWeb, also point to those terms.
Best alternatives
- FineWeb: Common Crawl with the filtering already done.
- Dolma: a documented mix of web, papers, code and books.
- Registry of Open Data on AWS: other cloud-scale datasets next to Common Crawl.
FAQ
Is Common Crawl free?
Yes. Access to the corpus hosted by Amazon is free. You pay for the compute you use to process it.
What is the latest Common Crawl crawl?
As of 2026-11-09, the newest crawl listed is CC-MAIN-2026-43.
Should I use Common Crawl or FineWeb?
Use FineWeb if you want cleaned, deduplicated English text ready to train on. Use Common Crawl if you need newer crawls, other languages or your own filtering.
Sources
- Common Crawl overview
- Common Crawl latest crawl
- Common Crawl terms of use
- Registry of Open Data on AWS
- LAION-BVD announcement
← Back to the ranked list of the best AI datasets
Named on this page as a provider? Join the network for a linked profile, Visit buttons and featured placement.