Training time and GPU usage

[icrto]

Hi,
First of all, thank you for making this code available!
I also have access to the MIMIC-CXR database and was wondering if you experienced problems during training concerning GPU usage.
I am training on an RTX 2080Ti and tqdm shows an average of 1.3 it/sec, predicting it will take approximately 4 hours to complete one epoch...which is completely unfeasible. I noticed the GPU usage is very low (~20%) and it is idle (0% usage) for long periods of time. Even if I increase the batch size the behaviour is the same.
Have you also experienced this? If so, any ideas on how to improve the training speed?

Thanks in advance!

[edreisMD]

Hi, in my experience lots of time is spent loading the image from the HD and preprocessing it on the CPU, so I use an SSD which speeds up the images loading time in each batch, making the GPU usage more efficient. I don't know if there are other techniques to improve training speed. If anyone knows, please let me know, so I'll also benefit!

[icrto]

Thank you so much for your insight! Unfortunately, I don't have enough space on my SSD for the whole dataset...I guess I will try to at least resize the images offline and see if it helps.

[zoule41]

Hello, I encountered a problem while executing the code of the warehouse author. I would like to ask you: What is the data structure of "csv_file" in the config file? I'm not sure how to set up this file. Thank you~