NL Seminar-Modular and Composable Transfer Learning
Thu, Nov 03, 2022 @ 11:00 AM - 12:00 PM
Information Sciences Institute
Conferences, Lectures, & Seminars
Speaker: Jonas Pfeiffer, Google
Talk Title: Modular and Composable Transfer Learning
Series: NL Seminar
Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you are highly encouraged to use your USC account to sign into Zoom.
If you are an outside visitor, please inform us at nlg DASH seminar DASH host AT isi DOT edu beforehand so we will be aware of your attendance and let you in.
In person attendance will be permitted for USC ISI faculty, staff, students only. Open to the public virtually via the zoom link and online.
With pre-trained transformer-based models continuously increasing in size, there is a dire need for parameter-efficient and modular transfer learning strategies. In this talk, we will touch base on adapter-based fine-tuning, where instead of fine-tuning all weights of a model, small neural network components are introduced at every layer. While the pre-trained parameters are frozen, only the newly introduced adapter weights are fine-tuned, achieving an encapsulation of the down-stream task information in designated parts of the model. We will demonstrate that adapters are modular components which can be composed for improvements on a target task and how they can be used for out of distribution generalization on the example of zero shot cross-lingual transfer. Finally, we will discuss how adding modularity during pre training can mitigate catastrophic interference and consequently lift the curse of multilinguality.
Biography: Jonas Pfeiffer is a Research Scientist at Google Research. He is interested in modular representation learning in multi task, multilingual, and multi-modal contexts, and in low resource scenarios. He worked on his PhD at the Technical University of Darmstadt, was a visiting researcher at the New York University and a Research Scientist Intern at Meta Research. Jonas has received the IBM PhD Research Fellowship award for 2021/2022. He has given numerous invited talks at academia, industry and ML summer schools, and has co-organized multiple workshops on multilinguality and multimodality
Host: Jon May and Meryem M'hamdi
More Info: https://nlg.isi.edu/nl-seminar/
Location: Information Science Institute (ISI) - Virtual and ISI-Conf Rm#689
WebCast Link: https://www.youtube.com/watch?v=hrGOb4okvI0
Audiences: Everyone Is Invited
Contact: Pete Zamar
Event Link: https://nlg.isi.edu/nl-seminar/