Is SwiGLU's Open Positive Tail Necessary? Evidence from Closed-Tail Gating with MemGLU
Yuting Ge, Pengju Yang, Mingkai Nie
Abstract
We test whether decoder-only language-model FFNs require SwiGLU's open positive tail. We introduce MemGLU as a closed-tail comparator derived from a memristive branch geometry. Across paired 9M and 30M pretraining runs with three seeds, MemGLU remains within about 0.1% of SwiGLU in validation NLL. Trained SwiGLU checkpoints are sensitive to positive-tail suppression, while mechanism diagnostics show that the two models use their gates differently despite similar losses. These results suggest that models adapt to the gate geometry available during pretraining. At the tested scales, SwiGLU's open positive tail is not necessary for decoder-only language-model FFNs.
Topics
Classified with taxonomy v2 on Sat, 5 Sept 2026.