Thu, Oct 05, 2023 @ 11:00 AM - 12:00 PM
Information Sciences Institute
Conferences, Lectures, & Seminars
Speaker: Shunyu Yao, Princeton University
Talk Title: On formulating and evaluating language agents
Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you are highly encouraged to use your USC account to sign into Zoom.
If you are an outside visitor, please inform us at nlg DASH seminar DASH host AT isi DOT edu beforehand so we will be aware of your attendance and let you in.
Language agents are AI systems that use large language models LLMs to interact with the world. While various methods have been developed, it is often hard to systematically understand or evaluate them. In this talk, we present Cognitive Architectures for Language Agents CoALA, a theoretical framework grounded in the classical research of cognitive architectures to make sense of existing agents and shed light into future directions. We also present three benchmarks WebShop, InterCode, Collie to develop and evaluate language agents using web, code, and grammar respectively. Notably, all three are scalable and practical, with simple and faithful evaluation metrics that do not rely on human preference labeling or LLM scoring.
Biography: Shunyu Yao is a final year Phd student with Karthik Narasimhan at Princeton NLP Group. His research focuses on language agents, and is supported by the Harold W. Dodds Fellowship from Princeton.
Host: Jon May and Justin Cho
More Info: https://nlg.isi.edu/nl-seminar/
WebCast Link: https://youtu.be/p6wSLDZat1w
Audiences: Everyone Is Invited
Contact: Pete Zamar
Event Link: https://nlg.isi.edu/nl-seminar/