Towards Scaling Reinforcement Learning to Massive Populations: Learning Mean-Field Representations

Modern multi-agent systems are increasingly deployed at scale over large populations of agents in settings such as ad-auctions, traffic routing, and recommendation systems. The dominant approach in such settings is to optimize each agent's policy independently, treating the other agents as part of a fixed single-agent environment rather than modeling the population dynamics. In many large-population systems, the dynamics depend on an aggregate summary of the population rather than the identity of any individual. Mean-field RL exploits such structure, providing a principled framework that models each agent's environment as an explicit function of the population distribution. However, in large state-action spaces or high-dimensional control problems, modeling the population distribution is itself intractable. How can we design a scalable framework for high-dimensional control problems with large populations? This work explores this question from the perspective of representation learning. We introduce a mean-field RL framework in which the rewards and transition dynamics depend on the population only through an unknown low-dimensional aggregate statistic. We then study this framework in the offline setting and design a provable approach that learns a near-optimal policy by learning a low-dimensional representation. Motivated by real-life supply-chain optimization problems, we design a one-step routing game to test the hypothesis that learning a low-dimensional population representation improves reward prediction and Nash gap estimation relative to baselines that don't exploit this structure. We show that under a fixed neural-network parameter count and optimization budget, learning a low-dimensional population representation improves reward prediction and the equilibrium quality of the resulting policies.

Graph Feedback Controls Consensus and Clique Formation in Open-Weight Language-Model Populations

Multi-agent language-model (LM) systems often determine which agents communicate, yet routing is usually treated as an implementation detail. We ask whether routing itself determines whether a population converges on a shared convention or fragments into persistent cliques. We study open-weight agents spanning 1.1B-32B parameters in a controlled naming game, tracking both emitted labels and full first-token preference distributions over the allowed labels. Similarity-based routing can isolate emerging conventions and sustain fragmentation even when every agent interacts in every round. Matched controls show that this effect is not explained solely by uneven participation or model-family-specific score preferences: random rematching and policies that connect disagreeing groups improve coordination when partner-label history is retained, but not when it is absent. Exposure alone is nevertheless insufficient, as some mixed-model populations remain divided despite frequent cross-family interaction, although the same models coordinate homogeneously. Trajectory and controlled-history analyses further distinguish reaching consensus from maintaining it. Finally, ARC-Challenge and MMLU experiments show that routing changes how correct and incorrect answers propagate without reliably improving accuracy. These results establish the runtime interaction graph as a causal design variable whose effects depend jointly on memory, model response, and population composition.

Cellular Predictions on the Move: What about Data?

Mobile cellular load forecasting is native to network resource optimization and delivery of services with reliability, latency and quality guarantees. The mainstream of machine learning research in the area is focused primarily on developing powerful learning structures for improved prediction accuracy. The data used for forecasting traditionally belong to the cellular domain and at most contain exogenous information about the surroundings of the base stations. We approach the prediction task from the perspective of data as a vital component of any data learning process. We hypothesize that substantial improvements could be achieved when the data inform on the processes that create the cellular load. Specifically, we propose to characterize the population dynamics -- the potential number of cellular traffic sources and their mobility -- in addition to employing historical time series of mobile data traffic. We validate our hypothesis for the rarely examined highway scenario. Comprehensive experiments show forecasting improvements on the order of $60\%$ due to the use of these data alone.