The Best Alternatives to Common Crawl for LLM Pretraining Data in 2026
Answer in brief: You rarely need to start from raw Common Crawl. For English, FineWeb gives you 18.5T+ filtered, deduplicated tokens. FineWeb-2 covers hundreds of languages. Dolma mixes web text with papers, code and books. RedPajama-V2 gives you 30B documents with quality signals so you can filter them your own way. For code, use The Stack v2. All are free. This is independent research from each dataset's official card.
Top pick: FineWeb. It is Common Crawl with the hard work already done: Trafilatura extraction, English filtering, Gopher, C4 and custom quality filters, and MinHash deduplication, across dumps from 2013 to 2025, under ODC-By.
Common Crawl alternatives compared
| Dataset | Size | Languages and sources | Processing | License |
|---|---|---|---|---|
| FineWeb | 18,527B GPT-2 tokens | English web, 2013 to June 2025 | Fully filtered and deduplicated | ODC-By 1.0 |
| FineWeb-Edu | Built on FineWeb | English, with educational-quality score columns | FineWeb plus an educational classifier score | See card |
| FineWeb-2 | Multilingual | Hundreds of language codes | Filtered and deduplicated for each language | ODC-By |
| Dolma | About 3T tokens | Web, academic papers, code, books, Wikipedia, Reddit, StackExchange | Curated mixed recipe; open toolkit | ODC-BY |
| RedPajama-V2 | 100B+ documents; 30B with quality signals; about 20B after deduplication | English, German, French, Spanish, Italian from 84 Common Crawl snapshots | CCNet processing plus precomputed quality signals; you choose the filters | Common Crawl Terms of Use |
| The Stack v2 | 67.5TB full; about 900B training tokens | Source code in 600+ languages (Software Heritage) | Deduplicated; license detected per file | Original file licenses; gated |
FineWeb and FineWeb-Edu: the English default
FineWeb's paper documents an ablation behind every filtering step, and its 10B, 100B and 350B samples make experiments cheap. FineWeb-Edu adds educational scores (score and int_score columns) for teams that want textbook-like data.
FineWeb-2: go multilingual without the pipeline
Building quality filters for hundreds of languages is hard. FineWeb-2 publishes that work under ODC-By, organized by language code, so you can take only what you need.
Dolma: more than the web
Dolma adds sources that web crawls underrepresent, such as peS2o and arXiv papers, Project Gutenberg books, OpenWebMath and code, and AI2 trained OLMo on it. The newest version is from April 2024.
RedPajama-V2: your filters, precomputed signals
RedPajama-V2 is for teams that want control without reprocessing raw crawls. It ships quality signals for 30B documents across 84 snapshots, so filtering becomes a query instead of a compute job.
The Stack v2: the code corpus web data lacks
FineWeb's card itself recommends pairing it with The Stack v2 for code. It has more than 3B files in 658 languages, StarCoder2 was trained on v2.0.1, and v2.2.0 removed opt-outs made before 2026-07-29. Access is gated, and bulk downloads go through Software Heritage.
When raw Common Crawl still wins
- You need crawls from after mid-2025 (the latest is CC-MAIN-2026-43).
- You want specific domains through the URL or CDXJ indexes, or the web graphs.
- Your filtering recipe is your competitive edge.
Pricing model
All the corpora here are free to download. Costs are storage (FineWeb alone is about 50TB) and compute. Beyond open data, licensed text from publishers and vendors is covered on our sibling site, Best AI training data providers.
FAQ
What is the best multilingual alternative to Common Crawl?
FineWeb-2, which covers hundreds of languages under ODC-By. RedPajama-V2 covers five European languages with quality signals.
Are these alternatives built from Common Crawl?
Mostly. FineWeb, FineWeb-2 and RedPajama-V2 are derived from Common Crawl, and Dolma includes it alongside non-web sources. The Stack v2 comes from Software Heritage instead.
Can I opt out of The Stack v2?
Yes. BigCode offers an 'Am I In The Stack?' tool, and new releases remove opt-outs that have been processed.
Keep exploring
- The full ranked list of the best AI datasets
- Common Crawl review · FineWeb review · Dolma review
- Common Crawl vs FineWeb · FineWeb vs Dolma
Sources
- Common Crawl
- FineWeb dataset card
- FineWeb-Edu dataset card
- FineWeb-2 dataset card
- Dolma dataset card
- RedPajama-V2 dataset card
- The Stack v2 dataset card
Named on this page as a provider? Join the network for a linked profile, Visit buttons and featured placement.