Skip to content
This repository has been archived by the owner on Mar 15, 2024. It is now read-only.

GPU memory usage grows continually over time #26

Open
YeonsungJung opened this issue Apr 14, 2022 · 4 comments
Open

GPU memory usage grows continually over time #26

YeonsungJung opened this issue Apr 14, 2022 · 4 comments

Comments

@YeonsungJung
Copy link

YeonsungJung commented Apr 14, 2022 •
edited
Loading

Hi

GPU memory usage grows continually over time.
FFHQ with 8 RTX3090s. Is it natural?
@MultiPath
Copy link
Contributor

Do you mean in the training time? I think it is not natural.

@YeonsungJung
Copy link
Author

YeonsungJung commented Apr 15, 2022 •
edited
Loading

it first happens after 50 ticks, then at some intervals.

@MultiPath
Copy link
Contributor

Will it grow infinitely and eventually OOM? I never met this before. Usually it took more memory when it start validating FID

@zhoupeizhu
Copy link

it first happens after 50 ticks, then at some intervals.

I encountered the same problem, have you solved it?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@MultiPath @zhoupeizhu @YeonsungJung