FineWeb vs Dolma: Which Open LLM Pretraining Dataset Should You Use in 2026?
Answer in brief: Choose FineWeb if you want the largest, freshest, ready-filtered English web corpus. It has more than 18.5 trillion GPT-2 tokens drawn from every Common Crawl dump since 2013, through June 2025, under ODC-By 1.0. Choose Dolma if you want a mixed-source recipe of about 3 trillion tokens covering web pages, academic papers, code, books and encyclopedia text. Dolma has already trained AI2's OLMo models and ships with an open curation toolkit. Both are free. This page is independent research based on each dataset's official card and paper. We have not run training experiments of our own.
Top pick: FineWeb. For most teams pretraining an English LLM in 2026, FineWeb's scale, recency (dumps through mid-2025) and ready-made samples (10B, 100B and 350B tokens) make it the fastest way to good web data.
The 30-second scorecard
| FineWeb | Dolma | |
|---|---|---|
| Publisher | Hugging Face (HuggingFaceFW) | Allen Institute for AI (AI2) |
| Size | 18,527B GPT-2 tokens, about 50.4TB | About 3T tokens; v1.7 is 4.5TB gzipped |
| Sources | English web only (Common Crawl, 2013 to 2025) | Web, peS2o/arXiv papers, code, Gutenberg books, Wikipedia, Reddit, StackExchange and more |
| Freshness | v1.4.0 added CC-MAIN-2025-05 through 2025-26 | v1.7 released April 2024; cutoff around October 2023 |
| License | ODC-By 1.0, plus Common Crawl Terms of Use | ODC-BY (since 2024-04-15), plus the licenses of its sources |
| Small samples | sample-10BT (27.6GB), 100BT, 350BT | v1.6-sample, 16.4GB (about 10B tokens) |
| Tooling | datatrove pipeline | Dolma toolkit on GitHub |
| Paper | arXiv 2406.17557 (NeurIPS 2024 Datasets and Benchmarks track) | arXiv 2402.00159 |
FineWeb: scale and freshness, already filtered
FineWeb takes care of the hardest part of web pretraining for you. Its pipeline runs URL filtering, extracts text with Trafilatura, keeps only text with a fastText English score of at least 0.65, applies Gopher, C4 and custom quality filters, and deduplicates each dump with MinHash. Email addresses and IP addresses are anonymized, and there is a form for requesting removal of personal data. Each crawl is its own subset, so you can mix by time period or start with the 10B-token sample on a single machine. The card says plainly that code content is limited and recommends pairing FineWeb with The Stack v2. Need more than English? The sibling dataset FineWeb-2 covers hundreds of languages under ODC-By. FineWeb-Edu adds educational-quality scores.
Dolma: a complete recipe you can reproduce
Dolma's selling point is breadth and openness. Version 1.7 combines Common Crawl, RefinedWeb, C4, StarCoder code, Reddit, peS2o and arXiv papers, StackExchange, Flan, CC News, OpenWebMath, Algebraic Stack, Project Gutenberg, MegaWika and Wikipedia. AI2 used 1.715T tokens of it to train OLMo-7B-v1.7. Earlier versions (v1.5, v1.6, v1) are still available, so you can reproduce published OLMo checkpoints exactly. The open-source toolkit lets you rerun or adapt the curation, and there is a form for removing personal data. On the Hub, 244 models are tagged as trained on Dolma. The trade-off is age: the newest version dates from April 2024.
When to pick which
- Pick FineWeb if you need the most English web tokens, data from 2024 to 2025, or quick ablations on fixed-size samples.
- Pick Dolma if you want academic, code, book and encyclopedia sources in one mix, or need to reproduce OLMo research.
- Use both if you are building a serious open model. Many teams use filtered web text as the backbone and add curated non-web sources. Add The Stack v2 for code.
- Need licensed or human-labeled data instead of open corpora? See our sibling guide, Best AI training data providers.
Pricing model
Both datasets are free to download from the Hugging Face Hub. You pay only for storage, bandwidth and compute. At full size, FineWeb is about 50TB and Dolma v1.7 is 4.5TB compressed, so budget accordingly. See our dataset access costs guide.
Licensing in plain English
Both use the Open Data Commons Attribution license for the collection. That license does not clear rights in every underlying web page or document. FineWeb is also subject to Common Crawl's Terms of Use, and Dolma users are bound by the licenses of its sources. Read each card before you use either one commercially.
FAQ
Is FineWeb bigger than Dolma?
Yes. FineWeb lists 18,527B GPT-2 tokens, and Dolma is about 3T tokens. Dolma, however, covers more kinds of source than FineWeb's English web text.
Which is more current?
FineWeb. Version 1.4.0 added Common Crawl dumps from January to June 2025. Dolma v1.7 came out in April 2024, with a cutoff around October 2023.
Can I try them without downloading terabytes?
Yes. FineWeb has a 10B-token sample (27.6GB), and Dolma has a 16.4GB sample of about 10B tokens.
Keep exploring
- The full ranked list of the best AI datasets
- FineWeb review · Dolma review · Common Crawl review
- Common Crawl vs FineWeb · Alternatives to Common Crawl · How to choose an AI dataset
Sources
- FineWeb dataset card
- FineWeb paper (arXiv 2406.17557)
- FineWeb-2 dataset card
- Dolma dataset card
- Dolma paper (arXiv 2402.00159)
- Dolma toolkit (GitHub)
Named on this page as a provider? Join the network for a linked profile, Visit buttons and featured placement.