Events Calendar

NL Seminar -Prioritized training on points that are learnable, worth learning, and not yet learned

Speaker: Sören Mindermann & Jan Brauner, University of Oxford

Talk Title: Prioritized training on points that are learnable, worth learning, and not yet learned

Series: NL Seminar

Abstract: REMINDER
Meeting hosts only admit guests that they know to the Zoom meeting. Hence, you are highly encouraged to use your USC account to sign into Zoom.

If you are an outside visitor, please inform us at nlg DASH seminar DASH host AT isi DOT edu beforehand so we will be aware of your attendance and let you in.

In person attendance will be permitted for USC ISI faculty, staff, students only. Open to the public virtually via the zoom link and online.

Training on web scale data can take months. But much computation and time is wasted on redundant and noisy points that are already learnt or not learnable. To accelerate training, we introduce Reducible Holdout Loss Selection RHO LOSS , a simple but principled technique which selects approximately those points for training that most reduce the models generalization loss.

As a result, RHO LOSS mitigates the weaknesses of existing data selection methods techniques from the optimization literature typically select hard eg high loss points, but such points are often noisy not learnable or less task relevant. Conversely, curriculum learning prioritizes easy points, but such points need not be trained on once learned. In contrast, RHO LOSS selects points that are learnable, worth learning, and not yet learnt. RHO LOSS trains in far fewer steps than prior art, improves accuracy, and speeds up training on a wide range of datasets, hyperparameters, and architectures MLPs, CNNs, and BERT. On the large web scraped image dataset Clothing 1M, RHO LOSS trains in 18 times fewer steps and reaches 2 percent higher final accuracy than uniform data shuffling.


Biography: Bio Soren Mindermann
Soren is a final year PhD student in machine learning at the University of Oxford, supervised by Yarin Gal. My interests in machine learning include how it scales, causal inference and statistical modeling, as well as robustly aligning machine learning models to adopt human wishes and value.

Bio Jan Brauner
Jan is a PhD candidate in the Centre for Doctoral Training on Intelligent and Autonomous Machines and Systems AIMS CDT, supervised by Yarin Gal. His current research interests include AI safety and applications of AI in medicine biomedical research.


Host: Jon May and Meryem M'hamdi

More Info: https://nlg.isi.edu/nl-seminar/

Webcast: https://www.youtube.com/watch?v=uRKrSBRAG0k

Location: Information Science Institute (ISI) - Virtual and ISI-Conf Rm#689

WebCast Link: https://www.youtube.com/watch?v=uRKrSBRAG0k

Audiences: Everyone Is Invited

Contact: Pete Zamar

Event Link: https://nlg.isi.edu/nl-seminar/