Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

The training process is stopping after loading weights and doesn't go on #13

Open
sms95 opened this issue Sep 10, 2018 · 1 comment
Open

Comments

@sms95
Copy link

sms95 commented Sep 10, 2018

The terminal shows the number of iteration and doesn't go on .
This program doesn't stand the cuda memory.

@sms95 sms95 changed the title The training process is topping after loading weights and doesn't go on The training process is stopping after loading weights and doesn't go on Sep 10, 2018
@wzt9332
Copy link

wzt9332 commented Sep 19, 2018

@sms95 Did you train the model? There was a gradient explosion in my training. Training data is cowc.

Region Avg IOU: nan, Class: nan, Obj: -nan, No Obj: -nan, Avg Recall: 0.000000, count: 18
Region Avg IOU: nan, Class: nan, Obj: -nan, No Obj: -nan, Avg Recall: 0.000000, count: 12
Region Avg IOU: nan, Class: nan, Obj: -nan, No Obj: -nan, Avg Recall: 0.000000, count: 18

Do you know the solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants