UltraData-Code is a complete implementation of the UltraData L0-L4 tiered data management framework. It covers four code data states from L0 through L3, with each level corresponding to a distinct construction stage. The pipeline starts from approximately 192 million public GitHub repositories at L0. This release provides UltraData-Code-L2 (~400B tokens) and UltraData-Code-L3 (~150B tokens) across 11 programming languages.
L0: Repository archival. We preserve the latest revision on the default branch of approximately 192 million public GitHub repositories, together with code, directory structure, file relationships, and provenance metadata. This archive provides complete raw data and traceability for subsequent processing.
L1: Standardized natural code. Starting from L0, we perform large-scale filtering, cleaning, format normalization, and near-duplicate deduplication. General rules remove oversized files, invalid paths, unsupported text, and obvious anomalies. Checks tailored to each language and file type then normalize and clean the remaining content. For deduplication, files are partitioned by extension. MinHash signatures are generated and LSH retrieves near-duplicate candidates. Similarity edges define connected components. For groups containing multiple near-duplicates, the highest ranked 50% are retained. Files with no detected near duplicate are all kept. The result is a standardized natural code corpus.
L2: Algorithmically relevant code. Across 11 programming languages, we select approximately 400B tokens of algorithmically relevant code from L1. We develop a language-adaptive selection framework that combines file role supervision with language-specific heuristic cues to learn algorithmic relevance beyond explicit ALGO files. The framework reuses precomputed semantic embeddings across role, relevance, and quality models, and applies role and quality constraints during selection. Under controlled 10B-token continual pre-training of a 1B model, L2 improves over L1 by 7.80 and 5.13 points on EvalPlus and MultiPL-E, and over Stack-Edu by 4.37 and 3.05 points.
L3: Task-oriented synthesis. We introduce a structured synthesis protocol that uses ALGO files at L2 as seeds and generates aligned task, analysis, solution, and test fields grounded in the source implementation. L3 covers the same 11 programming languages and contains approximately 150B tokens. Under controlled 10B-token continual pre-training of a 1B model, replacing half of the L2 training tokens with L3 further improves EvalPlus and MultiPL-E by 8.42 and 8.07 points over L2-only training.
Abstract: Code generation has become a core capability of large language models (LLMs), and code data is central to the pre-training process that develops it. As obtaining public code of sufficient quality becomes more difficult, further progress cannot rely on corpus expansion alone. Tiered data management instead applies increasingly targeted processing to smaller data pools, producing data states suited to different training objectives. Following this perspective, we present UltraData-Code as four connected data states spanning repository archival at L0, standardized natural code at L1, algorithmic selection at L2, and task-oriented synthesis at L3. L0 preserves repository data and provenance for reprocessing, while L1 applies scalable filtering, normalization, and near-deduplication for broad pre-training. L2 selects high-quality algorithmically relevant files using language-adaptive relevance signals and quality constraints, yielding approximately 400B tokens of UltraData-Code-L2 across 11 programming languages for focused continual pre-training. L3 converts selected implementations into structured programming exercises, yielding approximately 150B tokens of UltraData-Code-L3 across the same 11 languages for task-oriented mid-training or supervised fine-tuning (SFT). Under controlled 10B-token continual pre-training of a 1B model, L2 improves pass@1 over L1 by 7.80 percentage points (pp) on EvalPlus and 5.13pp on MultiPL-E. Replacing half of the L2 training tokens with L3 yields a further gain of 8.42pp and 8.07pp on the two benchmarks, respectively. Both gains increase when the training budget scales to 100B tokens. We have released our datasets and classifiers to support further research on LLMs.
The 10B-token comparisons use the same 1B foundation model, training settings, decontamination, and evaluation protocol.
Each row represents one selected source file. The fields are:
uuid: unique file identifier.repo_name: source repository name.relative_path: file path relative to the repository.content: source code text.category: predicted file role, such as ALGO, WEB, TOOL, DATA, TEST, CONFIG.algo_rel_score: algorithmic relevance score in [0, 1].quality_score: code quality score in [0, 10].Each row is a programming exercise grounded in a real implementation:
uuid: unique file identifier.content: serialization containing task and solution.content_format: serialization format for content.raw_content: original generated record before serialization.task: generated standalone problem statement.analysis: algorithm, edge cases, and complexity discussion.solution: generated self-contained reference implementation.test: generated test candidates.full_content: serialization containing all generated fields.full_content_format: serialization format for full_content.Thanks for their awesome work! Open source contributions make UltraData-Code possible! 🙌
This project is released under the Apache 2.0 license. UltraData-Code is built from code in multiple public repositories (L2) and from model-generated task records derived from those files (L3). Users must also comply with the LICENSE of each source repository. Apache 2.0 does not override those terms.
Public availability of a repository is not a grant of redistribution, commercial use, or training rights. Rights holders may request removal via the contact channel on the dataset page.
The dataset should not contain plaintext secrets or unauthorized personal data. Please report sensitive content or takedown requests through the same channel.
No unauthorized unchanged redistribution: Without prior written permission from the original authors (or this organization), any institution, organization, or third-party platform is strictly prohibited from directly reposting, mirroring, re-hosting, or commercially repackaging and republishing any artifacts of this project in any form.
If you find UltraData-Code useful in your research, please consider citing:
@misc{ultradata_code,
title = {{UltraData-Code}: From Raw Repositories to Algorithmically Dense and Task-Oriented Code Data},
author = {Chengying Tu and Hengyu Zhao and Shuaikang Xue and Zhongming Qu and Jihao Zhou and Xinle Lin and Junshao Guo and Zixuan Fu and Qiang Ma and Jie Zhou and Chaojun Xiao and Hongfei Yan and Yudong Wang and Xu Han and Zhiyuan Liu and Maosong Sun},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/datasets/openbmb/UltraData-Code}}
}