Logo: University of Southern California

Events Calendar


  • INFORMATION DROPOUT: LEARNING OPTIMAL REPRESENTATIONS THROUGH NOISY COMPUTATION

    Tue, Mar 07, 2017 @ 11:00 AM - 12:00 PM

    Information Sciences Institute

    Conferences, Lectures, & Seminars


    Speaker: Alessandro Achille, UCLA

    Talk Title: INFORMATION DROPOUT: LEARNING OPTIMAL REPRESENTATIONS THROUGH NOISY COMPUTATION

    Series: Natural Language Seminar

    Abstract: The cross-entropy loss commonly used in deep learning is closely related to the information theoretic properties defining an optimal representation of the data, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. Our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity.
    When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of disentangled representations of the input simply by enforcing a factorized prior, a fact that has been also observed empirically in recent work.
    Our experiments validate the theoretical intuitions behind our method, and we find that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise structure to the architecture of the network, as well as to the test sample.




    Biography: Alessandro Achille is a PhD student in Computer Science at UCLA, working with Prof. Stefano Soatto. He focuses on variational inference, representation learning, and their applications to deep learning and computer vision. Before coming to UCLA, he obtained a Master's degree in Pure Math at the Scuola Normale Superiore in Pisa, where he studied model theory and algebraic topology with Prof. Alessandro Berarducci.


    Host: Greg Ver Steeg

    More Info: https://arxiv.org/abs/1611.01353

    Location: Information Science Institute (ISI) - 6th Flr -CR#689 (ISI/Marina Del Rey)

    Audiences: Everyone Is Invited

    Contact: Peter Zamar

    OutlookiCal

Return to Calendar