Paper · 2025 · arXiv

Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application

Nischal Mainali, with L. Teixeira

We derive exact stochastic gradient dynamics for a linear transformer, which reveal staged learning and conservation laws, and then use the same measurements to study nonlinear models.