Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Results vary across different GPUs #5

Open
ZiyaoLi opened this issue Dec 8, 2021 · 3 comments
Open

Results vary across different GPUs #5

ZiyaoLi opened this issue Dec 8, 2021 · 3 comments
Labels
Remark User attention required

Comments

@ZiyaoLi
Copy link
Member

ZiyaoLi commented Dec 8, 2021

Dear users,

We found that in a small amount of test cases (e. g. T1038 in CASP-14), using different GPUs may lead to different output results, sometimes with large variances on the PDB structures and their evaluation metrics. The issue is potentially caused by the kernel-fusion strategies of jax.jit, but we are not sure currently.

We are sorry for the inconvenience, and are working on locating the problem.

As a temporary solution, we suggest using NVIDIA V100 / A100 / 3090Ti for model inference on local machines.

@ZiyaoLi ZiyaoLi changed the title Results varies across different GPUs Results vary across different GPUs Dec 8, 2021
@ZiyaoLi ZiyaoLi added the bug Something isn't working label Dec 8, 2021
@ZiyaoLi ZiyaoLi added Remark User attention required and removed bug Something isn't working labels Dec 17, 2021
@Zhang038
Copy link

Zhang038 commented Mar 4, 2022

Hi, could you please tell me what is the Cuda version you are using, would the cuda version also make a difference? Thanks a lot.

@ZiyaoLi
Copy link
Member Author

ZiyaoLi commented Mar 7, 2022

The code was tested on CUDA 11.0 (the default setting), and we reckoned in previous experiments that slight differences happened across different CUDA versions.

@Zhang038
Copy link

Zhang038 commented Mar 7, 2022

Thank you so much!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Remark User attention required
Projects
None yet
Development

No branches or pull requests

2 participants