Events Calendar

PhD Thesis Proposal - Qinyi Luo

PhD Thesis Proposal - Qinyi Luo
Title: High-Performance Heterogeneity-Aware Distributed Machine Learning Model Training
 
Committee members: Xuehai Qian (co-chair), Viktor Prasanna (co-chair), Ramesh Govindan, Chao Wang, Salman Avestimehr    
 
Abstract: The increasing size of machine learning models and the ever-growing amount of data result in days or even weeks of time required to train a machine learning model. To accelerate training, distributed training with parallel stochastic gradient descent is widely adopted as the go-to training method. This thesis proposal targets four challenges in distributed training: (1) performance degradation caused by large amount of data transfer among parallel workers, (2) heterogeneous computation and communication capacities in the training devices, i.e., the straggler problem, (3) huge memory consumption during training caused by huge model sizes, and (4) automatic selection of parallelization strategies. The proposal first introduces our work in decentralized training, including system support and algorithmic innovation that strengthen tolerance against stragglers in data-parallel training. Then, an adaptive during-training model compression technique is proposed to reduce the memory consumption of training huge recommender models. In the end, in the aspect of automatic parallelization of training workloads, a novel unified representation of parallelization strategies is proposed, as well as a search algorithm that selects superior parallel settings in the vast search space, and preliminary findings are discussed.     
 
Date and time: Feb 21 11am-12:30pm
Location: EEB 110  
 
Zoom link: https://usc.zoom.us/j/97299158202?pwd=bVlnRVFhTjJlZjVCY1hVNy9yWWE1UT09          

Location: Hughes Aircraft Electrical Engineering Center (EEB) - 110

Audiences: Everyone Is Invited

Contact: CS Events