Events Calendar

PhD Thesis Proposal - Zhonghao Shi

Title: Enabling Personalized Multimodal Interaction for Socially Assistive Robots and Agents
 
Date and Time: 6/18 2:30pm-4pm
 
Location: Ronald Tutor Hall (RTH) Room 406
 
Committee Members:
- Prof. Maja Matarić (Committee Chair)
- Prof. Gale Lucas
- Prof. Erdem Biyik
- Prof. David Traum
- Prof. Jonathan Tarbox (external) 
 
Abstract:
 
Socially assistive robots and agents have shown great promise in providing cost-effective support across domains such as education and healthcare. However, existing socially assistive AI systems still lack the ability to engage users in real-time interactions while effectively supporting users’ cognitive learning goals and socio-emotional needs. This dissertation proposal outlines both completed and remaining work toward enabling personalized, multimodal, real-time interaction capabilities between the robots/agents and human users, particularly for users with diverse needs. The work consists of two main research directions: personalized human perception and individualized interaction design. In the area of personalized human perception, the proposal presents completed work on applying domain adaptation methods to enable personalized multimodal affect recognition for children on the autism spectrum [1], and personalized child speech recognition [2]. It also includes work on benchmarking vision-language models to better understand the performance-latency tradeoffs of frontier models for real-time human perception [3]. In the area of individualized interaction design, the proposal discusses the development of an open-source socially assistive robot learning module for accessible and personalized AI and robotics education [4], as well as work on evaluating and personalizing text-to-speech voices for delivering mental health support [5]. Finally, this proposal will present an ongoing efforts  developing a personalized multimodal AI tutor to support speech therapy and learning [6].
 
[1] Shi, Z., Groechel, T. R., Jain, S., Chima, K., Rudovic, O., & Matarić, M. J. (2022). Toward personalized affect-aware socially assistive robot tutors for long-term interventions with children with autism. ACM Transactions on Human-Robot Interaction (THRI), 11(4), 1-28.
 
[2] Shi, Z., Shi, X., Feng, T., Xu, A., Narayanan, S., & Matarić, M. (2025). Examining test-time adaptation for personalized child speech recognition. Paper accepted for presentation at Interspeech 2025.
 
[3] Shi, Z., Zhao, E., Dennler, N., Wang, J., Xu, X., Shrestha, K., Seita, D., & Matarić, M. (2025). HRIBench: Benchmarking vision-language models for real-time human perception in human-robot interaction. Paper accepted for presentation at the International Symposium on Experimental Robotics (ISER) 2025.
 
[4] Shi, Z.*, O'Connell, A.*, Li, Z.*, Liu, S., Ayissi, J., Hoffman, G., ... & Matarić, M. J. (2024, March). Build your own robot friend: An open-source learning module for accessible and engaging AI education. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 38, No. 21, pp. 23137-23145).
 
[5] Shi, Z.*, Chen, H.*, Velentza, A. M., Liu, S., Dennler, N., O'Connell, A., & Mataric, M. (2023, March). Evaluating and personalizing user-perceived quality of text-to-speech voices for delivering mindfulness meditation with different physical embodiments. In Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction (pp. 516-524).
 
[6] Shi, Z.*, Chung, D.*, Du, Y., Zhang, J., Raina, S. & Matarić, M. J. (2025) Is AI Ready to Support Speech Therapy for Children? A Systematic Review of AI-Enabled Mobile Apps for Pediatric Speech Therapy. Accepted to Interaction Design and Children (IDC) Conference.
 

Location: Ronald Tutor Hall of Engineering (RTH) - 406

Audiences: Everyone Is Invited

Contact: Zhonghao Shi

This event is open to all eligible individuals. USC Viterbi operates all of its activities consistent with the University's Notice of Non-Discrimination. Eligibility is not determined based on race, sex, ethnicity, sexual orientation, or any other prohibited factor.

Add to Google Calendar