Dolma review: the best AI dataset for reproducible LLM pretraining

Answer in brief: Dolma, from the Allen Institute for AI, is the most thoroughly documented open pretraining mix you can download. It contains 3 trillion tokens of web content, academic papers, code, books and encyclopedia material, and Ai2 used it to train its OLMo models. It is licensed under ODC-BY and comes with a datasheet, a paper, open curation tools and token counts for every source. The catch is age: the newest version, v1.7, came out in April 2024, and its sources stop at or before October 2023.

Best for: researchers who need a reproducible, multi-source pretraining corpus with full documentation.

Independent research from the official Dolma dataset card and paper, checked 2026-11-09. Not hands-on testing.

What it gives you for pretraining

Dolma v1.7 combines Dolma's own Common Crawl extraction, RefinedWeb, StarCoder code, C4, Reddit, peS2o scientific papers, arXiv, StackExchange, Flan, CC News, OpenWebMath, Algebraic Stack, Project Gutenberg, MegaWika and Wikipedia/Wikibooks. The card's v1.7 statistics table lists 2,308.5 billion OLMo tokens in total. After sampling, 1.715 trillion tokens were used to train OLMo-7B-v1.7.

Versions to pick from

• v1.7 (default, 2024-04-15): 4.5TB gzip, with new sources, more quality filtering and fuzzy deduplication.
• v1.6 (2024-01-31): 5.4TB, plus a 16.4GB sample of about 10 billion tokens for exploration.
• v1.5 (2023-10-31): 6.4TB, used to train OLMo-1B, plus a 2.9TB sample of about 1.9T tokens.
• v1 (2023-08-18): the first release, 6.0TB.

Who it's built for

Academic and industry researchers who need to say exactly what went into a model, and teams that want code, math and science in the mix, not just web text.

Where it shines

• Documentation. A datasheet, an arXiv paper and per-source cutoff dates and processing notes.
• Reproducible. Open-source curation tools on GitHub show how to rebuild it.
• Proven. Hugging Face lists 244 models trained on it.
• Privacy process. A form for asking Ai2 to remove documents containing someone's personal data.

Honest drawbacks

• Stale for 2026: no new version since April 2024.
• Big: 4.5TB compressed for v1.7.
• The Dataset Viewer is disabled, and downloading means fetching URL lists with wget.
• ODC-BY binds you to the license terms of every original source as well.

Pricing and licenses

Free download. Released under ODC-BY since April 2024; Ai2 notes that you are also bound by the terms of the original data sources.

Best alternatives

• FineWeb: a larger English web corpus with newer crawls.
• Common Crawl: build your own corpus from the latest crawls.
• Hugging Face Hub: find code and instruction datasets to pair with Dolma.

FAQ

How many tokens are in Dolma?

Ai2 describes Dolma as 3 trillion tokens. The v1.7 table lists 2,308.5 billion OLMo tokens before sampling.

What license is Dolma under?

ODC-BY, and users are also bound by the terms of the original data sources.

Is there a small Dolma sample?

Yes. The v1.6 sample is 16.4GB, with roughly 10 billion tokens.

Sources

• Dolma dataset card
• Dolma paper (arXiv)
• Dolma toolkit on GitHub
• Ai2: Dolma moves to ODC-BY

← Back to the ranked list of the best AI datasets

Named on this page as a provider? Join the network for a linked profile, Visit buttons and featured placement.