Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CTC loss is 0 on GPU Tesla V100 #162

Open
bjlgcxc opened this issue Apr 3, 2020 · 7 comments
Open

CTC loss is 0 on GPU Tesla V100 #162

bjlgcxc opened this issue Apr 3, 2020 · 7 comments

Comments

@bjlgcxc
Copy link

bjlgcxc commented Apr 3, 2020

I get the problem of continuously get zero loss (loss = 0) on GPU Tesla V100,if anyone got the same problem or have some solutions?

@raotnameh
Copy link

raotnameh commented Apr 25, 2020

facing the same problem on longer audios of around 40-60 seconds.
@SeanNaren

@bjlgcxc
Copy link
Author

bjlgcxc commented May 7, 2020

I have solved the problem, just put the input of ctc loss on cpu

for example , CTC(acts, labels, act_lens, label_lens),

do acts.cpu() before run CTC loss

@Stonesjtu
Copy link

The ctc loss will return 0 on errors. One is the ctc loss is not compiled with cuda support but got cuda tensor input. Another situation is input sequence is too long.

@IAASSIBLCU
Copy link

The ctc loss will return 0 on errors. One is the ctc loss is not compiled with cuda support but got cuda tensor input. Another situation is input sequence is too long.

I also face the same problem. so I want to know how to check the ctc loss is compiled with cuda or not. and, if it really isn't compiled, what can I do to make them compiled. thanks a lot!!!

@raotnameh
Copy link

@IAASSIBLCU I would recommend you to try using PyTorch CTC.

@SeanNaren
Copy link
Owner

I too would also suggest using PyTorch CTC :) this library is fairly old, and I've also moved this project to use it successfully: https://github.com/SeanNaren/deepspeech.pytorch

@IAASSIBLCU
Copy link

@raotnameh @SeanNaren thanks a lot!! I will try PyTorch CTC in my project. Hope to get the dream result.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants