Events Calendar

NL Seminar

Speaker: Kalpesh Krishna , (U Mass, Amherst)

Talk Title: Advances in Text Generation and the Perils of its Automatic Evaluation

Series: NL Seminar

Abstract: Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you're highly encouraged to use your USC account to sign into Zoom. If you're an outside visitor, please inform us at nlg-seminar-admin2 at isi dot edu beforehand so we'll be aware of your attendance and let you in.

Recent advances in large scale language modeling have significantly improved the capability of natural language generation NLG systems, opening up several new applications. Unfortunately, evaluating NLG systems remains challenging, making it hard to measure meaningful progress. In this talk I will present our recent efforts in building and evaluating NLG systems for 1. unsupervised sentence-level style transfer 2. paragraph-length abstractive question answering with the ELI5 dataset. We build NLG systems using large language models with paraphrase generation and retrieval respectively, that significantly outperform prior state of the art using standard automatic metrics. Unfortunately, we discover several issues with the current evaluation setups, including trivial baselines like input copying which can game these standard metrics, even outperforming real systems. Along the way I will discuss our efforts towards rectifying these issues, and conclude with a brief mention of other projects working towards more robust NLG evaluation.


Biography: Kalpesh is a third year PhD student at UMass Amherst, advised by Prof. Mohit Iyyer. He is primarily interested in natural language generation and the security of NLP systems. Before coming to UMass, he completed a bachelors' degree at IIT Bombay, advised by Prof. Preethi Jyothi. He has also spent time interning at Google, TTI-Chicago and Mozilla. His research is supported by a Google PhD Fellowship, which was awarded in 2021.


Host: Jon May and Mozhdeh Gheini

More Info: https://nlg.isi.edu/nl-seminar/

Webcast: https://youtu.be/bv95xMBZO_U

Location: Information Science Institute (ISI) - Virtual Only

WebCast Link: https://youtu.be/bv95xMBZO_U

Audiences: Everyone Is Invited

Contact: Petet Zamar

Event Link: https://nlg.isi.edu/nl-seminar/