Toward a First-Principles Update Geometry for the Language-Model Head
Aditya Somasundaram
cs.LG
Abstract
We study the language-model head and softmax as a single module, deriving an update geometry from their composition rather than from the weight matrix in isolation. Under Hilbert's projective distance, the maximum change caused by an update $S$ over $\left|\left|{h}\right|\right|_2\le H$ is $H\max_{i<j}\left|\left|{s_i-s_j}\right|\right|_2$, which is $H$ times the Euclidean diameter of its token rows. Motivated by Muon's singular-value conditioning, we propose maximizing the smallest row separation while constraining this diameter, producing an approximate-equidistance problem when $V\gg d$.
Topics
Classified with taxonomy v2 on Sat, 5 Sept 2026.