Events Calendar

MASCLE Machine Learning Seminar: Quanquan Gu (UCLA) - New Variance Reduction Algorithms for Nonconvex Finite-Sum Optimization

Speaker: Quanquan Gu, UCLA

Talk Title: New Variance Reduction Algorithms for Nonconvex Finite-Sum Optimization

Series: Machine Learning Seminar Series

Abstract: Nonconvex finite-sum optimization problems are ubiquitous in machine learning such as training deep neural networks. To solve this class of problems, various variance reduction based stochastic optimization algorithms have been proposed, which are guaranteed to converge to stationary points and enjoy improved gradient complexity than vanilla stochastic gradient descent. An natural question is whether there is still space for improvement to further speed up the finding of first-order stationary points and even local minimas.

In the first part of this talk, I will introduce our work for finding first-order stationary points in nonconvex finite-sum optimization that further pushes the frontiers of this line of research. In particular, I will introduce a new stochastic nested variance reduced gradient algorithm (SNVRG) that achieves the fastest convergence rate to first-order stationary points in the literature by reducing the variance in stochastic algorithms through multiple referencing points and gradients. It outperforms the folklore variance reduction methods such as stochastic variance reduced gradient (SVRG) and stochastically controlled stochastic gradient (SCSG).

In the second part of the talk, I will talk about methods for finding second-order stationary points (i.e., local minima) in nonconvex finite-sum optimization. Specifically, I will introduce a stochastic variance reduced cubic regularization algorithm that achieves the state-of-the-art second-order oracle complexity for finding local minima in nonconvex optimization.


This lecture satisfies requirements for CSCI 591: Research Colloquium.


Host: Yan Liu, USC Machine Learning Center

Location: Henry Salvatori Computer Science Center (SAL) - 101

Audiences: Everyone Is Invited

Contact: Computer Science Department