Gradient-free learning of a closed-loop wall controller for turbulent drag reduction

Closed-loop wall control learnt by multi-agent reinforcement learning can lower skin-friction drag in turbulent channels, but these gradient-based policies are trained on small periodic boxes and exhibit reduced performance when carried over to a larger domain. We recently showed that such policies are also prone to saturated bang-bang actuations that collapse into standing streamwise waves whose scale is set by the computational box rather than by the near-wall cycle, and proposed architectural fixes that avoid these degeneracies. Here, we employ Evolution Strategy (ES) to optimise a recurrent closed-loop controller directly on a large turbulent channel at $\mathit{Re}_τ\simeq180$, evaluating policy performance over full flow episodes using an energy-aware criterion and processing candidate policies in parallel. To our knowledge, this is the first application of an evolution strategy to the control of a turbulent flow. The ES controller reduces the skin friction by about $26\%$, exceeding the gradient-based multi-agent controller of Cavallazzi et al. (2026), GRU-MARL, trained on a minimal box ($17\%$), and marginally exceeding classic opposition control (OC, $22\%$). A wall-normal decomposition of the friction, Reynolds-stress profiles and anisotropy invariants show that the ES and opposition-controlled flows follow separate trajectories through the buffer layer, reaching comparable drag reduction by different reorganisations of the near-wall turbulence. In particular, the ES actuation correlates predominantly with the streamwise velocity fluctuations rather than with the wall-normal velocity that classical OC targets.

Reward hacking in physical reinforcement learning revealed by turbulent drag reduction

Reinforcement-learning controllers optimise specified rewards, but in physical systems those rewards often capture only part of the true control objective. Three mechanisms through which this mismatch can produce apparent success without physical improvement are identified: incomplete accounting that omits relevant costs, constraint enforcement outside the policy that corrupts credit assignment, and observations that fail to resolve the relevant dynamics. All three are demonstrated in active drag reduction of wall-bounded turbulence, where the conservation constraint and full energy budget can be measured directly. A memoryless learnt policy reports drag reduction while raising total dissipation, collapsing to non-physical flow configurations. A recurrent multi-agent controller with the zero-mean projection embedded in the actor, temporal memory matched to the relevant timescales, and an actuation cost that bounds the wall power delivers a physically consistent control. Progress in physical reinforcement learning requires the reward, constraints, observations and evaluation metrics to represent unequivocally the physical objective.