Events Calendar

AAI-CCI-MHI Seminar on CPS

Speaker: Roy Fox, Assistant Professor of Computer Science Department, University of California Irvine

Talk Title: World Models -” Driven by Vision and Language; Driving Transfer

Series: EE598 Seminar Series

Abstract: An agent learning to control its environment would often do well to first model it. Compared to control policies, world models use a richer training signal and tend to generalize and transfer better. However, for a model to induce good behavior, it must be highly accurate in all reachable states, which may require too much data. Because efforts to leverage web-scale data for control are yet to succeed as they famously have for vision and language, we ask: can information encoded in vision and language foundation models help guide world modeling? In the first part of this talk, we will see two such methods: one that uses a segmentation foundation model to block visual distractions and keep state representations task-relevant; and one that queries a language model to hypothesize about abstract world models that guide exploration and planning. In the second part of the talk, we will revisit the transfer power of world models in two settings: simulation-to-reality and delayed perception. We will see how a model of a simulator can be adapted to reality with a tiny amount of data; and how a world model can transfer across varying delays of the agent's observations. 

Biography: Roy Fox is an Assistant Professor of Computer Science at the University of California, Irvine. His research interests include theory and applications of control learning: reinforcement learning (RL), control theory, information theory, and robotics. His current research focuses on structured and model-based RL, language for RL and RL for language, and optimization in deep control learning of virtual and physical agents. 

Host: Stephen Tu

Location: Hughes Aircraft Electrical Engineering Center (EEB) - 132

Audiences: Everyone Is Invited

Contact: Ariana Perez