A Diffusion Approximation for Temporal-Difference Learning with Linear Features under Markovian Noise
M. Forzo, E. Monzio Compagnoni, A. Russo, A. Pacchiano
Abstract
Temporal difference (TD) learning with linear function approximation is a core method for policy evaluation. Its classical continuous-time description is an ordinary differential equation (ODE), which captures the asymptotic mean dynamics but neglects stochastic fluctuations determining the error floor. We introduce a stochastic differential equation (SDE) approximation for linear TD(0) under Markovian noise. The resulting model distinguishes the contraction dynamics governed by the projected Bellman operator from the influence of Markovian sampling. As a consequence, the model explains the constant-stepsize error floor through the interaction between Markovian long-run covariance and the contraction geometry of the projected Bellman operator.
Topics
Classified with taxonomy v2 on Wed, 2 Sept 2026.