Into the Unknown: Benchmarking AI agents on open scientific research problems
What this work does
Fifteen author-proposed, unpublished follow-up directions supply original papers, code, datasets, and task specifications. Agents submit a follow-up paper, code, and execution logs. Expert-verified direction-specific and integrity rubrics are graded by an agentic judge, separating integrity, completeness, and sufficiency. The main metric requires all three strict gates.
Connection to discovery
A concrete evaluation protocol for pursuing open scientific problems without a reference solution, with executable artifacts and evidence traceability. The primary contribution is a benchmark and evaluation method, so it belongs in Discovery Evaluation.
Evidence
The authors evaluate nine agents on 15 tasks (135 submissions); none passes all strict gates and 114 fail integrity. Table 2 reports node-level and paper-level outcomes. Appendix A compares rubric judgments with domain-expert criticisms; agreement is strongest for satisfying the specified direction and weaker for broader scientific quality and impact.
Scope and limitations
Only 15 computational follow-up tasks from selected domains; humans supply the research direction, constraints, and success conditions. No new physical experiments are required. Automated judging and expert-validated rubrics test necessary conditions, not comprehensive novelty, scientific merit, autonomous question generation, or general research ability. Results are author-reported, not independently reproduced.
THE RESEARCH CONVERSATION
Discussion & reactions
Share a question, an insight, or a replication experience. Sign in with GitHub to join the conversation.