Events Calendar

PhD Thesis Proposal - Changzhi Xie

 

Title of Presentation: On the Dynamics of Learning Linear Functinos with Neural Networks
 
Date and Time: 4.14 2:30-3:30PM
 
Location: EEB 203
 
Committee Members: Mahdi Soltanolkotabi(committee chair), Haipeng Luo, Robin Jia, Vatsal Sharan, Adel Javanmard.
 
Abstract: We study the gradient descent training dynamics of fitting a one-hidden-layer network with multi-dimensional outputs to linear target functions. That is, we focus on a realizable model where the inputs are drawn i.i.d. from a Gaussian distribution and the labels are generated according to a planted linear model with multiple outputs. This framework serves as a good model for a variety of interesting problems including end-to-end training in inverse problems and various auto-encoder models in machine learning. Despite the seemingly simple formulation, understanding training dynamics is a challenging unresolved problem. This is in part due to the fact that the training landscape contains multiple local optima and it is completely unclear why gradient descent from random initialization is able to escape such bad optima. In this work, we develop the first comprehensive analysis of the gradient descent dynamics for learning linear target functions with ReLU networks. We show that gradient descent with moderately small random initialization converges to a global minimizer at a linear rate.  To rigorously show that GD avoids local optima, we develop intricate techniques to decompose the loss and control the GD trajectory, which may have broader implications for the analysis of non-convex optimization problems involving local optima. We corroborate our theoretical results with extensive experiments with various configurations.

Location: Hughes Aircraft Electrical Engineering Center (EEB) - 203

Audiences: Everyone Is Invited

Contact: Changzhi Xie

This event is open to all eligible individuals. USC Viterbi operates all of its activities consistent with the University's Notice of Non-Discrimination. Eligibility is not determined based on race, sex, ethnicity, sexual orientation, or any other prohibited factor.

Add to Google Calendar