ECE Seminar

Speaker: Jorge F. Silva, PhD, Universidad de Chile

Talk Title: Information Theoretic Measures for Representation Learning

Abstract: Information-theoretic measures have been widely adopted for machine learning (ML) feature design. Inspired by this, we look at the relationship between information loss in the Shannon sense and the operation loss in the minimum probability of error (MPE) sense when considering a family of lossy representations (or encoders). In this talk, we introduce a series of results that show how adequate the adoption of mutual information (MI) is for predicting the operational quality of a representation in classification. Our findings support the observation that selecting/designing representations that capture informational sufficiency (IS) is appropriate for learning. However, we also show that this selection is rather conservative if the intended goal is achieving MPE in classification. We conclude by discussing the capacity of the information bottleneck (IB) method to achieve lossless prediction and the expressive power of digital encoders in ML.

Biography: Information-theoretic measures have been widely adopted for machine learning (ML) feature design. Inspired by this, we look at the relationship between information loss in the Shannon sense and the operation loss in the minimum probability of error (MPE) sense when considering a family of lossy representations (or encoders). In this talk, we introduce a series of results that show how adequate the adoption of mutual information (MI) is for predicting the operational quality of a representation in classification. Our findings support the observation that selecting/designing representations that capture informational sufficiency (IS) is appropriate for learning. However, we also show that this selection is rather conservative if the intended goal is achieving MPE in classification. We conclude by discussing the capacity of the information bottleneck (IB) method to achieve lossless prediction and the expressive power of digital encoders in ML.

Host: Dr. Eduardo Pavez

More Information: Jorge Silva Seminar 2.23.24.pdf

Location: Hughes Aircraft Electrical Engineering Center (EEB) - 132

Audiences: Everyone Is Invited

Contact: Gloria Halfacre