Applied Math Seminar: Michael Murray (UCLA)

Title: Towards Understanding the Success of First Order Methods in Training Mildly Overparameterized Networks

Abstract: For most problems of interest the loss landscape of a neural network is non-convex and contains a plethora of spurious critical points. Despite this first order methods such as SGD and Adam are in practice remarkably successful at finding optimal, or at the least near optimal, minimizers of the loss. In recent years the Neural Tangent Kernel has proven a powerful tool in explaining this phenomena and for providing guarantees for highly overparameterized networks. However, for mildly overparameterized networks (where width scales linearithmically in the sample size) where richer feature learning can occur an explanation is lacking. In this talk I will present recent results on the loss landscape of two-layer mildly overparameterized ReLU networks. Our approach involves bounding the dimension of the sets of local and global minima using the rank of the Jacobian of the parameterization map. Using results on random binary matrices, we show most activation patterns correspond to parameter regions with no bad differentiable local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank to many regions having deficient rank depending on the amount of overparameterization.