Dataset Summary
Into the Unknown tests whether AI agents can build on a scientist's work and write a follow-up paper. Each of its 15 tasks gives an agent a published (or recently submitted) paper and a follow-up research direction proposed by the paper's own author; the agent must carry out the research and write a follow-up paper that is honest, complete, and meets the scientist's bar.
This dataset holds the 15 tasks and everything nine agents produced on them: 135 submissions, with each agent's paper, full trajectory, and the files it wrote. Code, rubrics and graders are on GitHub; the paper is on arXiv.
Supported Tasks and Leaderboards
The task is open-ended research: given the paper and the direction, produce the follow-up paper. The leaderboard shows every agent's results and every rubric verdict, and you can submit your agent's paper to be evaluated.
Subsets
| subset | content |
|---|---|
benchmark/ | The 15 tasks: paper<i>/task_spec.md, the original paper (paper.pdf, its text paper.md, figures images/) and index.csv. A task's code and data are downloaded by prepare_tasks.py in the GitHub repository. |
outcomes/ | The papers the agents wrote, <agent>/paper<i>.pdf (the proposal/report.pdf the judge read): 131 PDFs, since four GLM-5.3 runs produced none. |
trajectories/ | Each run's <agent>/paper<i>.json (run metadata and every agent event), plus the harnesses' own logs (*.harness_logs.tar; DeepSeek-V4-Pro's full conversation is here) and the runners' console logs (*.runner_logs.tar). |
file_systems/ | Each submission's files as the judge saw them: proposal/ and the rest of the working directory (workdir_outside_proposal/), without environments, caches or the final transcript. view.tar.part-* with a SHA-256 manifest.json; nine runs add a supplement.tar of files the judge's copy lacked. |
Agents (<agent>): opus-4.8, opus-5, opus-5.5, fable-5, sol-5.6, astra-6, deepseek-v4-pro, kimi-k3, glm-5.3.
Dataset Structure
An example of a benchmark datum is as follows:
paper: (str) - The task id, P1 to P15.
group: (str) - A: published 2024 or earlier; B: submitted 2026.
title: (str) - The original paper's title.
domain, venue, lab: (str) - Its field, venue and the laboratory of its authors.
citations: (int) - Citations as of September 14, 2026.
original_paper: (str) - A link to the original paper.
follow_up_direction: (str) - The research direction the paper's author proposed.
task_spec, paper_pdf, paper_md: (str) - Paths to the task specification and the original paper in this dataset.
outcomes/index.csv and trajectories/index.csv list one row per submission (agent, paper, file, and run metadata).
Evaluation
Run an agent and judge its submission with the GitHub repository:
bash src/solver/run.sh <agent> <paper_id>
python src/grader/run_codex_judge.py prepare --agents <agent> # then rescope, launch
To unpack one submission's files:
huggingface-cli download prometheus-eval/into-the-unknown --repo-type dataset \
--include "file_systems/astra-6/paper3/*" --local-dir .
cd file_systems/astra-6/paper3 && cat view.tar.part-* > view.tar && tar -xf view.tar
Sample Usage
Load the benchmark subset directly with the datasets library:
from datasets import load_dataset
tasks = load_dataset('prometheus-eval/into-the-unknown', 'benchmark', split='train')
A few notes specific to this dataset:
outcomes/fable-5/paper8.pdf has a real person's name, affiliation and e-mail blacked out on its title page; every other PDF is byte-identical to the submission.file_systems/opus-5.5/paper12, two R help pages (qsreg.html, spam.html) differ only in case from other pages; the archive has the originals, the others are in supplement.tar, and manifest.json lists both under checked_against_judge_record.sha256_mismatch.Citation
If you find our work helpful, please consider citing it:
@article{kim2026into,
title={Into the Unknown: Benchmarking AI agents on open scientific research problems},
author={Kim, Seungone and Geng, Jiayi and Lee, Young-Jun and Liu, Bo and Suk, Juyoung and Lubbers, Nicholas and Wood, Mitchell and Noack, Marcus and Yao, Yong-Xin and Shi, Yuan and others},
journal={arXiv preprint arXiv:2610.19159},
year={2026}
}
License
Task specifications, rubrics and documentation are released under the Apache-2.0 license of the GitHub repository. The original papers in benchmark/ remain under their authors' and publishers' terms and are included for research use; agent outputs may contain third-party code and data that keep their own licenses.