Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Beginner] CUDA out of memory #24

Open
realisticdreamer114514 opened this issue Dec 30, 2021 · 3 comments
Open

[Beginner] CUDA out of memory #24

realisticdreamer114514 opened this issue Dec 30, 2021 · 3 comments

Comments

@realisticdreamer114514
Copy link

I'm a beginner who's dissatisfied with the official SimSwap_512 beta model and trying to finetune the official people model for better performance at -image_size 512 --display_winsize 512. When I'm starting my finetuning there's this:

RuntimeError: CUDA out of memory. Tried to allocate 64.00MiB (GPU 0; 4.00 GiB total capacity; 2.28 GiB already allocated; 54.45 MiB free; 2.42 GiB reserved in total by Pytorch)

This happens even at -image_size 224 --display_winsize 224 and I found out it is a relatively common problem. I'm not sure if it's my hardware limitations (GTX 1650 with Max-Q Design) or some underlying issue with the code/my configuration, since I have no other concurrent graphical processes when I ran the command.

Since this is my first major experience with training models and machine learning in general I'll be asking some basic questions as they come up, and I hope anyone following this project will be patient with me. Thanks!

@netrunner-exe
Copy link

Add --batchSize option to command. It happens because your GTX 1650 is haven't enough memory to pass a large dataset through it. Default value of batchSize is 8. Try lower value, example : CUDA_VISIBLE_DEVICES=0 python train.py --batchSize 4 --name CelebA_512_finetune --which_epoch latest --dataroot ./dataset/CelebA --image_size 512 --display_winsize 512 --continue_train

@Lyntai
Copy link

Lyntai commented Dec 31, 2021

I don't have a NVIDIA card so I'm trying on colab and having this error too when finetuning, but it will be generally tedious work for the colab limitations of time too.
I wonder if it is possible if someone just trains the dataset once and then releases the end files, are there any copyright issues?

@realisticdreamer114514
Copy link
Author

Kept reducing the batch size and the same error keeps happening, with different numbers for RAM allocation & free RAM. Is it something to do with my configuration, or is my GPU underpowered?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants