NL Seminar-Designing and Evaluating Language Models for Human Interaction

Speaker: Mina Lee, Stanford University

Talk Title: Designing and Evaluating Language Models for Human Interaction

Abstract: REMINDER

Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you are highly encouraged to use your USC account to sign into Zoom.

If you are an outside visitor, please inform us at nlg DASH seminar DASH host AT isi DOT edu beforehand so we will be aware of your attendance and let you in.

Despite the recent advancements in language models LMs, most LMs are not optimized for, nor are they evaluated on, real-world usage with human interaction. In this talk, I will present my research on designing and evaluating LMs for human LM interaction. Concretely, I will first describe how we can support human editing needs by enabling any LM to perform text infilling at any position in a document i.e., fill in the blanks. I will then introduce CoAuthor, a platform for capturing human LM interaction in collaborative writing as rich, replayable, keystroke level interaction traces. With the platform, I demonstrate how collecting a large interaction dataset and analyzing the traces provide unique insights into LM capabilities regarding language, ideation, and collaboration. Lastly, I will propose a new framework, HALIE Human AI Language based Interaction Evaluation, that defines the components of interactive systems and evaluation metrics for human LM interaction beyond writing. I will conclude by discussing open challenges and future directions in this field.

Biography: Mina Lee is a final year Ph.D. candidate at Stanford University, advised by Professor Percy Liang. Her research goal is to design and evaluate language models to enhance our productivity and creativity and understand how these models change the way we write. She has built various writing assistants, including an autocomplete system, a contextual thesaurus system, and a creative story writing system, as well as evaluated language models based on their ability to interact with humans and augment human capabilities.

She was named one of MIT Technology Reviews Korean Innovators under 35 in 2022, and her work has been published in top tier venues in natural language processing e.g., ACL and NAACL, machine learning e.g., NeurIPS, and human computer interaction e.g., CHI. Her recent work on human AI collaborative writing received an Honorable Mention Award at CHI 2022 and was featured in various media outlets including The Economist.

Host: Jon May and Justin Cho

More Info: https://nlg.isi.edu/nl-seminar/

Webcast: https://www.youtube.com/watch?v=rfl3_fa8eHQ

Location: Information Science Institute (ISI) - Virtual and ISI-Conf Rm#689

WebCast Link: https://www.youtube.com/watch?v=rfl3_fa8eHQ

Audiences: Everyone Is Invited

Contact: Pete Zamar

Event Link: https://nlg.isi.edu/nl-seminar/

This event is open to all eligible individuals. USC Viterbi operates all of its activities consistent with the University's Notice of Non-Discrimination. Eligibility is not determined based on race, sex, ethnicity, sexual orientation, or any other prohibited factor.

Add to Google Calendar