Are We Measuring Scientific Intelligence? Rethinking the Evaluation of AI Scientists

What this work does

Tests whether AI-scientist answers actually follow supplied evidence by withdrawing or reversing evidence in benchmark data. High ordinary accuracy fell sharply under an evidence-grounded metric, showing that prior knowledge and shortcuts can masquerade as scientific reasoning.