Thu, Jul 01, 2021 @ 09:00 AM - 10:00 AM
Information Sciences Institute
Conferences, Lectures, & Seminars
Speaker: Kalpesh Krishna , (U Mass, Amherst)
Talk Title: Advances in Text Generation and the Perils of its Automatic Evaluation
Series: NL Seminar
Abstract: Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you're highly encouraged to use your USC account to sign into Zoom. If you're an outside visitor, please inform us at nlg-seminar-admin2 at isi dot edu beforehand so we'll be aware of your attendance and let you in.
Recent advances in large scale language modeling have significantly improved the capability of natural language generation NLG systems, opening up several new applications. Unfortunately, evaluating NLG systems remains challenging, making it hard to measure meaningful progress. In this talk I will present our recent efforts in building and evaluating NLG systems for 1. unsupervised sentence-level style transfer 2. paragraph-length abstractive question answering with the ELI5 dataset. We build NLG systems using large language models with paraphrase generation and retrieval respectively, that significantly outperform prior state of the art using standard automatic metrics. Unfortunately, we discover several issues with the current evaluation setups, including trivial baselines like input copying which can game these standard metrics, even outperforming real systems. Along the way I will discuss our efforts towards rectifying these issues, and conclude with a brief mention of other projects working towards more robust NLG evaluation.
Biography: Kalpesh is a third year PhD student at UMass Amherst, advised by Prof. Mohit Iyyer. He is primarily interested in natural language generation and the security of NLP systems. Before coming to UMass, he completed a bachelors' degree at IIT Bombay, advised by Prof. Preethi Jyothi. He has also spent time interning at Google, TTI-Chicago and Mozilla. His research is supported by a Google PhD Fellowship, which was awarded in 2021.
Host: Jon May and Mozhdeh Gheini
More Info: https://nlg.isi.edu/nl-seminar/
WebCast Link: https://youtu.be/bv95xMBZO_U
Audiences: Everyone Is Invited
Contact: Petet Zamar