Policy-Invariant Reward Shaping from LLM Feedback: A Framework for Hybrid RL Agents
Christophe D. Hounwanou, John Emeka Eze, Yaé U. Gaba
Abstract
Combining large language models with reinforcement learning is increasingly explored, yet the theoretical status of LLM-derived reward signals is often left implicit. We formalize the hybrid LLM-planner and RL-controller architecture as a Goal-Augmented Markov Decision Process and show that when the LLM per-state progress score is used as a bounded potential function, the resulting shaping term preserves the optimal policy set even when the LLM scores are inaccurate. This guarantee is stronger than what general LLM-as-reward approaches provide. We verify the result numerically on a small MDP under four potential configurations, including an adversarial one scaled to twenty times the base reward magnitude.
Topics
Classified with taxonomy v2 on Wed, 2 Sept 2026.