Some Math behind Neural Tangent Kernel

Lilian Weng published a blog post in September 2022 titled "Some Math behind Neural Tangent Kernel" on Lil'Log, presenting a math-intensive deep dive into the neural tangent kernel (NTK) introduced by Jacot et al. in 2018.

The post focuses on a small number of core papers rather than a broad literature review. Weng notes that later works modifying or expanding the theory are not covered.

According to the post, neural networks are over-parameterized and can often fit data with near-zero training loss and decent generalization, with optimization consistently producing similarly good outcomes even when parameters exceed training data points. NTK is described as a kernel explaining the evolution of neural networks during training via gradient descent, offering insights into why sufficiently wide networks consistently converge to a global minimum when minimizing an empirical loss.

The post covers background material including vector-to-vector derivatives, differential equations, the Central Limit Theorem, Taylor expansion, kernel methods, and Gaussian processes, then proceeds to the NTK definition, infinite-width networks, the connection with Gaussian processes, deterministic NTK, linearized models, and lazy training.

It references Lee & Bahri et al. (2018) on deep neural networks as Gaussian processes, Chizat et al. (2019) on lazy training, Lee & Xiao et al. (2019) on wide networks evolving as linear models, and Arora et al. (2019) for a proof requiring only sufficiently large minimum width rather than all hidden layers being infinitely wide.

Weng states the goal is to present all the math behind NTK in a clear, easy-to-follow format and invites readers to report mistakes for correction.