MASCLE Machine Learning Seminar: Quanquan Gu (UCLA) - New Variance Reduction Algorithms for Nonconvex Finite-Sum Optimization
Thu, Nov 08, 2018 @ 03:30 PM - 04:50 PM
Thomas Lord Department of Computer Science
Conferences, Lectures, & Seminars
Speaker: Quanquan Gu, UCLA
Talk Title: New Variance Reduction Algorithms for Nonconvex Finite-Sum Optimization
Series: Machine Learning Seminar Series
Abstract: Nonconvex finite-sum optimization problems are ubiquitous in machine learning such as training deep neural networks. To solve this class of problems, various variance reduction based stochastic optimization algorithms have been proposed, which are guaranteed to converge to stationary points and enjoy improved gradient complexity than vanilla stochastic gradient descent. An natural question is whether there is still space for improvement to further speed up the finding of first-order stationary points and even local minimas.
In the first part of this talk, I will introduce our work for finding first-order stationary points in nonconvex finite-sum optimization that further pushes the frontiers of this line of research. In particular, I will introduce a new stochastic nested variance reduced gradient algorithm (SNVRG) that achieves the fastest convergence rate to first-order stationary points in the literature by reducing the variance in stochastic algorithms through multiple referencing points and gradients. It outperforms the folklore variance reduction methods such as stochastic variance reduced gradient (SVRG) and stochastically controlled stochastic gradient (SCSG).
In the second part of the talk, I will talk about methods for finding second-order stationary points (i.e., local minima) in nonconvex finite-sum optimization. Specifically, I will introduce a stochastic variance reduced cubic regularization algorithm that achieves the state-of-the-art second-order oracle complexity for finding local minima in nonconvex optimization.
This lecture satisfies requirements for CSCI 591: Research Colloquium.
Host: Yan Liu, USC Machine Learning Center
Audiences: Everyone Is Invited
Contact: Computer Science Department