How AI settled the complexity of the oldest SGD algorithm
Michał Dereziński, Xiaoyu Dong
cs.LG cs.AI math.NA math.OC stat.ML
Abstract
In 1937, Stefan Kaczmarz proposed a simple algorithm for solving systems of linear equations. This algorithm turned out to be the earliest known example of stochastic gradient descent, a ubiquitous computing paradigm that drives the training of modern AI models such as ChatGPT and Gemini. Now, those AI models have joined forces to discover the worst-case complexity of the Kaczmarz algorithm. This paper tells the story of how it happened.
Topics
Classified with taxonomy v2 on Sat, 5 Sept 2026.