Shape, Symmetries, and Structure: The Changing Role of Mathematics in Machine Learning Research

The Gradient published an article arguing that mathematics remains as relevant as ever to machine learning research, though its role is evolving. According to the article, the past decade has seen mathematically principled architecture research yield only marginal improvements, while compute-intensive, engineering-first efforts that scale to larger training sets and model parameter counts produce capabilities that existing theory did not predict. The article states that this shift has prompted speculation about mathematics' diminished role and that mathematics will have to share the stage with perspectives such as biology and the social sciences.

The article counters that mathematics is simply changing how it contributes. It says mathematics may move from primarily providing theoretical guarantees on model performance toward post-hoc explanations of empirical phenomena observed in model training and performance, a role it compares to physics. It also says mathematical intuition may shift from guiding handcrafted features or architectural details at a granular level to higher-level choices such as matching architecture to task structure or data symmetries. It notes the translation-equivariant convolutional neural network is over 40 years old.

The article says the shift toward scale has broadened the mathematics applicable to machine learning, with domains such as topology, algebra and geometry joining probability theory, analysis and linear algebra. It describes tools including intrinsic dimension, used to describe dataset complexity and applied to detecting adversarial examples, AI-generated content and hallucinations; curvature, used to analyze loss landscapes, the 'edge of stability', and adversarial vulnerability; and topology, including homology, applied to how networks process data, predict early-stopping times, and inspire generalizations of graph neural networks. It also cites linear mode connectivity and the linear representation hypothesis as progress. The article does not report new experimental results.