Policy Invariance under Reward Transformations Theory and Application to Reward Shaping
What this work does
Characterizes potential-based shaping with the discount-consistent difference of successor and current potentials, preserving optimal policies under the stated MDP conditions.
Connection to discovery
A starting constraint when adding dense progress feedback to a sparse discovery objective.
Evidence
The proof separates sufficient invariance from worst-case necessity, while grid-world examples illustrate learning-speed effects and problematic nonpotential rewards.
Scope and limitations
Terminal-state and discount conditions matter. Adding an arbitrary novelty or learned progress bonus is not covered by the theorem. The original Stanford copy has broken text encoding; the Berkeley author copy was used for body review.
THE RESEARCH CONVERSATION
Discussion & reactions
Share a question, an insight, or a replication experience. Sign in with GitHub to join the conversation.