Events Calendar

NL Seminar -On formulating and evaluating language agents

Speaker: Shunyu Yao, Princeton University

Talk Title: On formulating and evaluating language agents

Abstract: REMINDER:

Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you are highly encouraged to use your USC account to sign into Zoom.

If you are an outside visitor, please inform us at nlg DASH seminar DASH host AT isi DOT edu beforehand so we will be aware of your attendance and let you in.

Language agents are AI systems that use large language models LLMs to interact with the world. While various methods have been developed, it is often hard to systematically understand or evaluate them. In this talk, we present Cognitive Architectures for Language Agents CoALA, a theoretical framework grounded in the classical research of cognitive architectures to make sense of existing agents and shed light into future directions. We also present three benchmarks WebShop, InterCode, Collie to develop and evaluate language agents using web, code, and grammar respectively. Notably, all three are scalable and practical, with simple and faithful evaluation metrics that do not rely on human preference labeling or LLM scoring.



Biography: Shunyu Yao is a final year Phd student with Karthik Narasimhan at Princeton NLP Group. His research focuses on language agents, and is supported by the Harold W. Dodds Fellowship from Princeton.

Host: Jon May and Justin Cho

More Info: https://nlg.isi.edu/nl-seminar/

Webcast: https://youtu.be/p6wSLDZat1w

Location: Information Science Institute (ISI) - Virtual and ISI-Conf Rm#689

WebCast Link: https://youtu.be/p6wSLDZat1w

Audiences: Everyone Is Invited

Contact: Pete Zamar

Event Link: https://nlg.isi.edu/nl-seminar/