AI Seminar-Things Multimodal LLMs Cannot See: Toward Discovering and Mitigating Perceptual Biases in Neural Networks through Visual Interventions

Speaker: Mahyar Khayatkhoei , USC/ISI

Talk Title: Things Multimodal LLMs Cannot See: Toward Discovering and Mitigating Perceptual Biases in Neural Networks through Visual Interventions

Abstract: In this talk, I will discuss our recent research on the use of pixel-space interventions for discovering and mitigating biases in visual neural networks, including in multimodal large language models (MLLMs). I will start by showcasing our discovered perceptual limitations and biases of MLLMs (including commercial ones such as GPT-4V and LLaVA). I will then discuss our simple yet effective intervention-based approach for mitigating such limitations, which can do so without requiring any training. Finally, I will more broadly discuss the problem of removing attribute-specific bias from neural networks, present our latest information theoretic bounds on this problem, and explain our adversarial input-intervention approach for removing strong attribute bias.
This event will be recorded but only shared with AI Division Leadership.

Biography: I am a Computer Scientist at the AI Division of the USC Information Sciences Institute. I received my Ph.D. and M.Sc. in computer science from Rutgers University working with Dr. Ahmed Elgammal, and my B.Sc. in electrical engineering from the University of Tehran. My research explores the theory and application of deep generative models, and has identified and resolved major bottlenecks in neural networks’ ability to learn from heterogeneous data (NeurIPS 2018), to learn high frequency features (AAAI 2022), and in their reliable evaluation (ICML 2023). My latest focus is on adopting large-scale generative neural networks to real-world mission-critical tasks. I am particularly interested in developing reliable and efficient data-driven computational models of real-world phenomena that would enhance our current physics-based models. My personal website is at https://mahyarkoy.github.io 

Host: Host: Adam Russell, POC Justina Gilleland and Alma Nava

More Info: https://www.isi.edu/events/4966/things-multimodal-llms-cannot-see-toward-discovering-and-mitigating-perceptual-biases-in-neural-networks-through-visual-interventions/

Webcast: https://usc.zoom.us/j/93179461297?pwd=d2RpNWlEblhxcHRFMU9RbnRxbWJBUT09

Location: Information Science Institute (ISI) - Conf Rm#1135

WebCast Link: https://usc.zoom.us/j/93179461297?pwd=d2RpNWlEblhxcHRFMU9RbnRxbWJBUT09

Audiences: Everyone Is Invited

Contact: Pete Zamar

Event Link: https://www.isi.edu/events/4966/things-multimodal-llms-cannot-see-toward-discovering-and-mitigating-perceptual-biases-in-neural-networks-through-visual-interventions/