Stochastic Attention via Langevin Dynamics on the Modern Hopfield Energy
发表于 2026年3月6日上传于 2026年3月10日
This paper shows that a standard attention head implements gradient descent on the modern Hopfield log-sum-exp energy and that applying Langevin dynamics to that energy yields a training-free stochastic attention sampler controlled by a single inverse-temperature parameter β; the sampler interpolates between exact retrieval and open-ended generation, admits convergence guarantees due to the Hopfield energy’s analytic structure, and—via an SNR-based temperature rule—yields substantially higher novelty and diversity than learned baselines on image and financial pattern tasks .