Online KL-Regularized Reinforcement Learning with Function Approximation under Misspecification
Haoyang Hong, Zichen Wang, Quanquan Gu, Huazheng Wang
cs.LG
Abstract
We study KL-regularized contextual bandits and episodic reinforcement learning (RL) under general function approximation with model misspecification. Existing guarantees rely on realizability and therefore do not extend to misspecified models, where classical regret bounds may fail. This work introduces KL misspecification formulations for contextual bandits and episodic RL and analyzes regression-based algorithms with Gibbs policy updates. High-probability KL-regret guarantees with explicit misspecification terms are established, recovering the standard realizable KL-regularized setting as a special case.
Topics
Classified with taxonomy v2 on Sat, 5 Sept 2026.