Dept. of Mathematical and Statistical Sciences Fall 2023 Seminar Series Presents

Date and time: Monday December 4th 2023 , from 12:30pm to 1:30pm

Location: Student Commons Building room 4125

Speaker: Lionel Eyrault-Dubois, Inria Bordeaux Sud-Ouest

Title: Optimal rematerialization algorithms for Memory-efficient learning

Abstract: The training phase of Deep Neural Networks is often a very memory-intensive procedure, where large amounts of intermediate data have to be kept in memory during one iteration. One possible approach to reduce memory usage is rematerialization, aka gradient checkpointing, where some intermediate data are recomputed when needed rather than kept in memory. This provides a tradeoff between memory usage and recomputation time. In this talk I will present several approaches for the optimization problem, where one wants to minimize the recomputation time given a fixed memory budget. The corresponding algorithms have been implemented in easy-to-use libraries for the PyTorch framework, which can significantly reduce memory usage with reasonable overhead.