CS Colloquium: Jason Wu - Computational Understanding of User Interfaces

Speaker: Jason Wu, CMU

Talk Title: Computational Understanding of User Interfaces

Series: Computer Science Colloquium

Abstract: A grand challenge in human-computer interaction (HCI) is constructing user interfaces (UIs) that make computers useful for all users across all contexts. Today, most UIs are manually designed for a rigid set of assumptions and are unable to dynamically accommodate the diversity of user abilities, usage contexts, or computing technologies. The goal of my research is to build a machine that can understand and operate any UI then dynamically convert it into a new personalized, context-dependent representation. In this talk, I focus on three areas that define this approach for enhancing human-computer interaction. First, I describe approaches for understanding user ability and context embodied by a recommendation system that recommends device settings (e.g., accessibility features) based on sensed usage behaviors and user interaction logs. Next, I introduce several machine learning models that reliably understand the semantics (content and functionality) of any graphical UI from its visual appearance, unlocking new possibilities for many existing systems such as assistive technology, software testing, and UI automation. Finally, I present systems that incorporate both user and UI understanding to synthesize improved interfaces using a novel fine-tuned large language model (LLM) for UI generation. Improved machine understanding of UIs has the potential to redefine how we use computers in the future and drive advances in many fields such as HCI, machine learning and software engineering.  
 
This lecture satisfies requirements for CSCI 591: Research Colloquium

Biography: Jason Wu is a PhD candidate in the HCI Institute at Carnegie Mellon University advised by Jeffrey Bigham. In his research, Jason builds data-driven and computational systems that understand, manipulate, and synthesize user interfaces to maximize the usability and accessibility of computers . His research has been published in top venues for human-computer interaction, user interface technology, accessibility, and machine learning, where he has received several best paper awards (CHI 2021, W4A 2021) and honorable mention awards (CHI 2020, CHI 2023). His work has also been recognized outside of academic conferences by a Fast Company Innovation by Design Student Finalist Award, press coverage in major outlets such as TechCrunch and AppleInsider, and by the FCC Chair Awards for Advancements in Accessibility. Jason is a recipient of the NSF Graduate Research Fellowship and selected as a Heidelberg Laureate Forum Young Researcher. 

Host: Souti Chattopadhyay

Location: Olin Hall of Engineering (OHE) - 136

Audiences: Everyone Is Invited

Contact: CS Faculty Affairs