PaperBench: Evaluating AI's Ability to Replicate AI Research
What this work does
Agents implement research papers from scratch and are graded against hierarchical rubrics, including execution and result-match requirements after clean-environment reproduction.
Connection to discovery
Useful for instrumenting long-horizon research dependencies and measuring reproducible work rather than a plausible report alone.
Evidence
Inspected rubric aggregation, fresh-VM execution and the distinction between code-development and full replication evaluation.
Scope and limitations
Replication supplies the target method and findings. Automated rubric judging is fallible; using an evaluation rubric for training would alter the task and risk leakage.
THE RESEARCH CONVERSATION
Discussion & reactions
Share a question, an insight, or a replication experience. Sign in with GitHub to join the conversation.