Common Crawl vs FineWeb: Should You Train on the Raw Web or a Filtered Corpus?
Answer in brief: Common Crawl is the raw material. It is a nonprofit archive of more than 300 billion web pages collected since 2008, free to download, with new crawls throughout the year. The latest is CC-MAIN-2026-43. FineWeb is that raw material already refined: an English-only corpus of more than 18.5 trillion tokens built from Common Crawl dumps from 2013 to June 2025, then extracted, language-filtered, quality-filtered and deduplicated. Choose FineWeb if you want to start training now. Choose Common Crawl if you need crawls newer than mid-2025, languages other than English, or your own filtering recipe. This is independent research from official documentation.
Top pick for most teams: FineWeb. It saves you the months of extraction, filtering and deduplication that raw Common Crawl requires. Top pick for custom pipelines: Common Crawl. It is the freshest and most complete web source available.
Head-to-head at a glance
| Common Crawl | FineWeb | |
|---|---|---|
| What you get | Raw pages (WARC), metadata (WAT), extracted text (WET), URL/CDXJ indexes, web graphs | Clean extracted English text with metadata, in Parquet on the Hub |
| Coverage | All languages, 2008 to present | English, Common Crawl dumps from 2013 to June 2025 |
| Newest data | CC-MAIN-2026-43 | CC-MAIN-2025-26 (v1.4.0) |
| Ready to train? | No. You filter, deduplicate and clean it yourself | Yes. Filtering and MinHash deduplication are already done |
| Access | HTTPS via data.commoncrawl.org (no AWS account); S3 at s3://commoncrawl/ (authenticated users); selected crawls mirrored on Hugging Face | Hugging Face datasets library or direct download; 10B, 100B and 350B samples |
| License | Common Crawl Terms of Use; rights in the underlying pages vary | ODC-By 1.0, plus Common Crawl Terms of Use |
| Cost | Free; you pay for compute and egress | Free; you pay for storage and compute |
Common Crawl: the whole web, on your terms
Nothing beats Common Crawl for coverage and freshness. It is hosted in the AWS Open Data Sponsorship Program in us-east-1, so you can process it right next to the data, and anyone can download it over plain HTTPS without an AWS account. You get the URL index, the CDXJ index and web graphs, which let you pull specific domains instead of whole crawls. There is also a public opt-out ledger. The catch is real: the raw data needs serious engineering for extraction, language identification, quality filtering, deduplication, safety and rights review before it is fit for training.
FineWeb: Common Crawl, already cleaned up
FineWeb is Hugging Face's published answer to how you turn Common Crawl into good pretraining data. Built with the open datatrove library, the pipeline filters URLs, extracts text with Trafilatura, keeps English text with a fastText score of at least 0.65, applies Gopher, C4 and custom quality filters, and runs MinHash deduplication on each dump. Email addresses and IP addresses are anonymized, and there is a form for requesting removal of personal data. The totals table lists 18,527B tokens across about 50.4TB, and the paper documents the ablations behind each step.
When to pick which
- Start training this week: FineWeb. Run ablations on sample-10BT (27.6GB) first.
- Need 2026 web data: Common Crawl. FineWeb's newest dump is from mid-2025.
- Multilingual: FineWeb-2 for filtered text in hundreds of languages, or Common Crawl for full control. RedPajama-V2 covers English, German, French, Spanish and Italian with quality signals you can filter on.
- Your own filters or domain-targeted pulls: Common Crawl and its indexes.
- Licensed, proprietary or human-labeled data: that is outside the open web. See Best AI training data providers.
Pricing model
Both are free to access. With Common Crawl the costs are compute, storage and any egress outside us-east-1. With FineWeb they are storage for up to about 50TB, plus training compute. Our access costs guide goes into more detail.
FAQ
Is FineWeb just Common Crawl?
It is built entirely from Common Crawl, but heavily processed: text extraction, English filtering, quality filters, deduplication and anonymization of personal data. You get far less noise per token.
Do I need an AWS account to download Common Crawl?
No. HTTPS download from data.commoncrawl.org works without one. Access through the S3 API requires an authenticated AWS user.
Which is better for a non-English model?
Not FineWeb, which is English only. Use FineWeb-2 or filter Common Crawl yourself.
Keep exploring
- Back to the ranked list of the best AI datasets
- Common Crawl review · FineWeb review
- FineWeb vs Dolma · Alternatives to Common Crawl · How to choose an AI dataset
Sources
- Common Crawl
- Common Crawl: Get Started
- Common Crawl on the Registry of Open Data on AWS
- FineWeb dataset card
- FineWeb paper
- RedPajama-V2 dataset card
Named on this page as a provider? Join the network for a linked profile, Visit buttons and featured placement.