Logo: University of Southern California

Events Calendar

  • MASCLE Machine Learning Seminar: Quanquan Gu (UCLA) - New Variance Reduction Algorithms for Nonconvex Finite-Sum Optimization

    Thu, Nov 08, 2018 @ 03:30 PM - 04:50 PM

    Thomas Lord Department of Computer Science

    Conferences, Lectures, & Seminars

    Speaker: Quanquan Gu, UCLA

    Talk Title: New Variance Reduction Algorithms for Nonconvex Finite-Sum Optimization

    Series: Machine Learning Seminar Series

    Abstract: Nonconvex finite-sum optimization problems are ubiquitous in machine learning such as training deep neural networks. To solve this class of problems, various variance reduction based stochastic optimization algorithms have been proposed, which are guaranteed to converge to stationary points and enjoy improved gradient complexity than vanilla stochastic gradient descent. An natural question is whether there is still space for improvement to further speed up the finding of first-order stationary points and even local minimas.

    In the first part of this talk, I will introduce our work for finding first-order stationary points in nonconvex finite-sum optimization that further pushes the frontiers of this line of research. In particular, I will introduce a new stochastic nested variance reduced gradient algorithm (SNVRG) that achieves the fastest convergence rate to first-order stationary points in the literature by reducing the variance in stochastic algorithms through multiple referencing points and gradients. It outperforms the folklore variance reduction methods such as stochastic variance reduced gradient (SVRG) and stochastically controlled stochastic gradient (SCSG).

    In the second part of the talk, I will talk about methods for finding second-order stationary points (i.e., local minima) in nonconvex finite-sum optimization. Specifically, I will introduce a stochastic variance reduced cubic regularization algorithm that achieves the state-of-the-art second-order oracle complexity for finding local minima in nonconvex optimization.

    This lecture satisfies requirements for CSCI 591: Research Colloquium.

    Host: Yan Liu, USC Machine Learning Center

    Location: Henry Salvatori Computer Science Center (SAL) - 101

    Audiences: Everyone Is Invited

    Contact: Computer Science Department

    Add to Google CalendarDownload ICS File for OutlookDownload iCal File

Return to Calendar