RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts
What this work does
Evaluates agents and human experts on seven bounded AI R&D tasks with quantitative scoring and explicit time/compute conditions.
Connection to discovery
Shows why long single trajectories, repeated short attempts and total cost must be compared under a common budget.
Evidence
Inspected best-score-versus-budget comparisons and concrete scoring functions for kernels, scaling experiments and model optimization.
Scope and limitations
Seven selected tasks and specific human recruitment conditions limit generalization. Best-of-k uses privileged selection unless its scoring cost/access is accounted for.
THE RESEARCH CONVERSATION
Discussion & reactions
Share a question, an insight, or a replication experience. Sign in with GitHub to join the conversation.