Events Calendar

NL Seminar-Modular and Composable Transfer Learning

Speaker: Jonas Pfeiffer, Google

Talk Title: Modular and Composable Transfer Learning

Series: NL Seminar

Abstract: REMINDER
Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you are highly encouraged to use your USC account to sign into Zoom.

If you are an outside visitor, please inform us at nlg DASH seminar DASH host AT isi DOT edu beforehand so we will be aware of your attendance and let you in.

In person attendance will be permitted for USC ISI faculty, staff, students only. Open to the public virtually via the zoom link and online.

With pre-trained transformer-based models continuously increasing in size, there is a dire need for parameter-efficient and modular transfer learning strategies. In this talk, we will touch base on adapter-based fine-tuning, where instead of fine-tuning all weights of a model, small neural network components are introduced at every layer. While the pre-trained parameters are frozen, only the newly introduced adapter weights are fine-tuned, achieving an encapsulation of the down-stream task information in designated parts of the model. We will demonstrate that adapters are modular components which can be composed for improvements on a target task and how they can be used for out of distribution generalization on the example of zero shot cross-lingual transfer. Finally, we will discuss how adding modularity during pre training can mitigate catastrophic interference and consequently lift the curse of multilinguality.


Biography: Jonas Pfeiffer is a Research Scientist at Google Research. He is interested in modular representation learning in multi task, multilingual, and multi-modal contexts, and in low resource scenarios. He worked on his PhD at the Technical University of Darmstadt, was a visiting researcher at the New York University and a Research Scientist Intern at Meta Research. Jonas has received the IBM PhD Research Fellowship award for 2021/2022. He has given numerous invited talks at academia, industry and ML summer schools, and has co-organized multiple workshops on multilinguality and multimodality

Host: Jon May and Meryem M'hamdi

More Info: https://nlg.isi.edu/nl-seminar/

Webcast: https://www.youtube.com/watch?v=hrGOb4okvI0

Location: Information Science Institute (ISI) - Virtual and ISI-Conf Rm#689

WebCast Link: https://www.youtube.com/watch?v=hrGOb4okvI0

Audiences: Everyone Is Invited

Contact: Pete Zamar

Event Link: https://nlg.isi.edu/nl-seminar/