Spectral Bias: Why Neural Networks Learn Low Frequencies First

Train a neural network on a wiggly target function and watch what happens, epoch by epoch. The network does not learn the whole function at once. It first fits a smooth, coarse approximation, and only later starts to capture the sharp wiggles, spikes, and fine detail. This is not a bug in your optimiser or a quirk of one architecture. It is a systematic preference known as spectral bias, also called the frequency principle. Understanding spectral bias changes how you think about training, generalisation, and failure modes. It explains why networks generalise surprisingly well on smooth data, why they struggle with high-frequency detail, and why some tricks that look like regularisation are really just frequency control in disguise. The intuition in one picture Imagine trying to sketch a coastline. You would start with the broad shape of the coast, then add bays, then inlets, then rocky detail. A neural network does something similar, but in frequency space rather than in g