About this Event
1201 Larimer Street
Dept. of Mathematical and Statistical Sciences Fall 2023 Seminar Series Presents
Date and time: Monday December 4th 2023 , from 12:30pm to 1:30pm
Location: Student Commons Building room 4125
Speaker: Lionel Eyrault-Dubois, Inria Bordeaux Sud-Ouest
Title: Optimal rematerialization algorithms for Memory-efficient learning
Abstract: The training phase of Deep Neural Networks is often a very memory-intensive procedure, where large amounts of intermediate data have to be kept in memory during one iteration. One possible approach to reduce memory usage is rematerialization, aka gradient checkpointing, where some intermediate data are recomputed when needed rather than kept in memory. This provides a tradeoff between memory usage and recomputation time. In this talk I will present several approaches for the optimization problem, where one wants to minimize the recomputation time given a fixed memory budget. The corresponding algorithms have been implemented in easy-to-use libraries for the PyTorch framework, which can significantly reduce memory usage with reasonable overhead.