NL Seminar-Inductive Biases for Data- and Parameter-Efficient Transfer Learning

Speaker: Mozhdeh Gheini, USC/ISI, USC/ISI

Talk Title: Inductive Biases for Data- and Parameter-Efficient Transfer Learning

Abstract: THIS TALK WILL "NOT" BE RECORDED, PLEASE WATCH LIVE OR ATTEND IN PERSON. https://usc.zoom.us/j/95381979100?pwd=yKkC6snFuqRddSnRCEwnVWvtP9ZdCX.1 Meeting ID: 953 8197 9100Passcode: 911377 Data- and resource-intensive pre-training and fine-tuning applied upon Transformer-based models is the dominant paradigm at the forefront of rapid advancements in natural language processing, human language technologies, and most notably, large language models. Such reliance on massive amounts of data, computation, and energy, while effective and impressive from a performance-only perspective, can hinder open, nonexclusive, and sustainable development of these technologies. In this talk, we present how certain inductive biases can be devised to adjust current natural language methods under resource-constrained scenarios and provide insights into why the proposed inductive biases are successful in such cases. Specifically, we discuss four research directions on data and parameter efficiency of fine-tuning and transfer learning in natural language processing: (1) a universal regimen that creates a single pre-trained checkpoint suitable for machine translation transfer to practically any language pair and eliminates the need for ad hoc pre-training; (2) an architecture-guided parameter-efficient fine-tuning method that performs competitively with full fine-tuning while exclusively updating cross-attention parameters; (3) an analysis of Mega, a recently introduced augmentation of the Transformer architecture to incorporate explicit recency bias, through the lens of transfer learning; and (4) a meta-learning algorithm to prime pre-trained models for specific fine-tuning strategies. Combined with ablations that show how they are effective and analyses that demonstrate their generalizability, these directions are meant to serve as tools for resource-efficient transfer learning for natural language processing.

Biography: Mozhdeh "Mo" Gheini is a PhD candidate at the University of Southern California advised by Jonathan May. Her PhD focus has been on investigating different inductive biases to build data- and parameter-efficient methods for transfer learning for natural language processing tasks like machine translation and beyond. She has also spent three summers interning with Apple, where she will be joining again in February as Machine Learning Research Engineer.

Host: Jonathan May and Katy Felkner

More Info: https://www.isi.edu/events/5379/inductive-biases-for-data-and-parameter-efficient-transfer-learning/

Webcast: https://usc.zoom.us/j/95381979100?pwd=yKkC6snFuqRddSnRCEwnVWvtP9ZdCX.1

Location: Information Science Institute (ISI) - Virtual and ISI-Conf Rm#689

WebCast Link: https://usc.zoom.us/j/95381979100?pwd=yKkC6snFuqRddSnRCEwnVWvtP9ZdCX.1

Audiences: Everyone Is Invited

Contact: Pete Zamar

Event Link: https://www.isi.edu/events/5379/inductive-biases-for-data-and-parameter-efficient-transfer-learning/