
MASCLE Machine Learning Seminar: Quanquan Gu (UCLA)  New Variance Reduction Algorithms for Nonconvex FiniteSum Optimization
Thu, Nov 08, 2018 @ 03:30 PM  04:50 PM
Computer Science
Conferences, Lectures, & Seminars
Speaker: Quanquan Gu, UCLA
Talk Title: New Variance Reduction Algorithms for Nonconvex FiniteSum Optimization
Series: Machine Learning Seminar Series
Abstract: Nonconvex finitesum optimization problems are ubiquitous in machine learning such as training deep neural networks. To solve this class of problems, various variance reduction based stochastic optimization algorithms have been proposed, which are guaranteed to converge to stationary points and enjoy improved gradient complexity than vanilla stochastic gradient descent. An natural question is whether there is still space for improvement to further speed up the finding of firstorder stationary points and even local minimas.
In the first part of this talk, I will introduce our work for finding firstorder stationary points in nonconvex finitesum optimization that further pushes the frontiers of this line of research. In particular, I will introduce a new stochastic nested variance reduced gradient algorithm (SNVRG) that achieves the fastest convergence rate to firstorder stationary points in the literature by reducing the variance in stochastic algorithms through multiple referencing points and gradients. It outperforms the folklore variance reduction methods such as stochastic variance reduced gradient (SVRG) and stochastically controlled stochastic gradient (SCSG).
In the second part of the talk, I will talk about methods for finding secondorder stationary points (i.e., local minima) in nonconvex finitesum optimization. Specifically, I will introduce a stochastic variance reduced cubic regularization algorithm that achieves the stateoftheart secondorder oracle complexity for finding local minima in nonconvex optimization.
This lecture satisfies requirements for CSCI 591: Research Colloquium.
Host: Yan Liu, USC Machine Learning Center
Location: Henry Salvatori Computer Science Center (SAL)  101
Audiences: Everyone Is Invited
Contact: Computer Science Department