Errors encountered when trying to run multimer mode

[mingyue80]

Hi there, I tried to run multimer mode (monomer mode worked well), but encountered the following errors:
context 0x5618aefca000: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2022-02-07 22:20:55.956803: E external/org_tensorflow/tensorflow/stream_executor/stream.cc:310] failed to allocate stream during initialization
2022-02-07 22:20:55.956815: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2022-02-07 22:20:55.956811: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:618] unable to add host callback: CUDA_ERROR_INVALID_HANDLE: invalid resource handle
2022-02-07 22:20:55.956826: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_event.cc:29] Error polling for event status: failed to query event: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
2022-02-07 22:20:55.956834: E external/org_tensorflow/tensorflow/stream_executor/cuda/cuda_driver.cc:699] could not allocate CUDA stream for context 0x5618aefca000: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

I checked tensorflow, CUDA, etc. and it all seems to have been installed correctly. One thing I did notice is that only GPU 0 was used - even though I specified 0,1,2,3 in the commands, and also all 4 GPUs were visible. Are these issues related, and how do I resolve them?

[Old-Shatterhand]

Hi @mingyue80,

sorry for the late response. In general, this seems to be an issue with TensorFlow and CUDA.

Can you provide more information on how you called alphafold and give the full StackTrace? So that we can try to reproduce this error.

Best, Roman