Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release/2.5] ModuleTracker: Add explicit garbage collection #1661

Open
wants to merge 1 commit into
base: release/2.5
Choose a base branch
from
Open

[release/2.5] ModuleTracker: Add explicit garbage collection #1661

wants to merge 1 commit into from
+2 −1

Commits on Oct 30, 2024

  1. [release/2.5] ModuleTracker: Add explicit garbage collection 

    When running an FSDP model with FlopCounterMode, we are experiencing a memory
leak. It is coming from ModuleTracker class. Even though
ModuleTracker class is keeping weakrefrences of the operators, the
tensors/operators are not being freed after the backward pass. To force
free these tensors/operators after backwardpass, I explicitly added
garbage collection in the post forward hook.

(cherry picked from commit 63dc40d)
    @pragupta
    pragupta committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    328b01c View commit details
    Browse the repository at this point in the history