Events Calendar

INFORMATION DROPOUT: LEARNING OPTIMAL REPRESENTATIONS THROUGH NOISY COMPUTATION

Speaker: Alessandro Achille, UCLA

Talk Title: INFORMATION DROPOUT: LEARNING OPTIMAL REPRESENTATIONS THROUGH NOISY COMPUTATION

Series: Natural Language Seminar

Abstract: The cross-entropy loss commonly used in deep learning is closely related to the information theoretic properties defining an optimal representation of the data, but does not enforce some of the key properties. We show that this can be solved by adding a regularization term, which is in turn related to injecting multiplicative noise in the activations of a Deep Neural Network, a special case of which is the common practice of dropout. Our regularized loss function can be efficiently minimized using Information Dropout, a generalization of dropout rooted in information theoretic principles that automatically adapts to the data and can better exploit architectures of limited capacity.
When the task is the reconstruction of the input, we show that our loss function yields a Variational Autoencoder as a special case, thus providing a link between representation learning, information theory and variational inference. Finally, we prove that we can promote the creation of disentangled representations of the input simply by enforcing a factorized prior, a fact that has been also observed empirically in recent work.
Our experiments validate the theoretical intuitions behind our method, and we find that Information Dropout achieves a comparable or better generalization performance than binary dropout, especially on smaller models, since it can automatically adapt the noise structure to the architecture of the network, as well as to the test sample.




Biography: Alessandro Achille is a PhD student in Computer Science at UCLA, working with Prof. Stefano Soatto. He focuses on variational inference, representation learning, and their applications to deep learning and computer vision. Before coming to UCLA, he obtained a Master's degree in Pure Math at the Scuola Normale Superiore in Pisa, where he studied model theory and algebraic topology with Prof. Alessandro Berarducci.


Host: Greg Ver Steeg

More Info: https://arxiv.org/abs/1611.01353

Location: Information Science Institute (ISI) - 6th Flr -CR#689 (ISI/Marina Del Rey)

Audiences: Everyone Is Invited

Contact: Peter Zamar

Event Link: https://arxiv.org/abs/1611.01353