Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents
What this work does
CANOPY combines broader rollout exploration with reference-policy regularization to preserve useful outcome-reward variation and limit policy drift.
Connection to discovery
An empirical counterpoint to assuming long tasks necessarily require a process reward: first test coverage and reward-group signal.
Evidence
AppWorld comparisons and a software-repair transfer test examine outcome-only learning, difficulty and inference budgets.
Scope and limitations
Results rely on verifiable digital tasks. The method does not create a reliable verifier for unknown scientific conclusions, and larger sampling budgets change the compute comparison.
THE RESEARCH CONVERSATION
Discussion & reactions
Share a question, an insight, or a replication experience. Sign in with GitHub to join the conversation.