GPU利用率问题

[vegetablebird5]

Hello, author, first of all thank you for your open source code. Here is a question I want to ask you, that is, when using GPU for training, the GPU memory is occupied very high , but the GPU utilization is very low. What is the reason about this issue?

[andreYoo]

Dear friend I can see a similar situation when the feature memory is initialised or re-initialised. it seems like cos GPUs are only working for extracting features, and not for gradient computation. So, when u see this situation, and if u can see the messages 'Graph Re-initisliation' or 'Look-up table Overhaul - [reinitialising]', it is normal. best regards.

…

---------------------------------------------------------------------------------------------------------- Ph.D Jongmin Yu Research Associate Institute of IT Convergence Korea Advanced Institute of Science and Technology (KAIST) KAIST 291 Daehak-ro, Yuseong-gu. Daejeon 34141, Republic of Korea E-mail: ***@***.*** ***@***.*** ***@***.***> mail.com)

----------------------------------------------------------------------------------------------------------

On Sun, 26 Sept 2021 at 09:45, vegetablebird5 ***@***.***> wrote: [image: image] <https://user-images.githubusercontent.com/85222184/134790343-86e1e9f2-23cb-4155-8364-a359b37a60a3.png> 作者您好，首先感谢您的代码开源，这里有个问题想请教您，就是在使用GPU进行训练时，显存占的很高，但是GPU利用率很低，请问是什么原因呢? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFO2B7WZCLPV4G3CJYUMG3UDZ3MZANCNFSM5EYGYDSA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[vegetablebird5]

Firstly,Thanks for your reply. Is it the mean that when you training the model yourself ,you also have the same issue on GPU utilization,right? Secondly, that is to say ,this issue is a normal phenomenon？Last but not the least，is there any method to solve the problem.

[andreYoo]

Dear friend. This is my answer. 1. I also saw the same situation when I did some experiments. 2. Idk it, is normal or not. but, during my experiments, it wasn't a problem in optimising a model. it took a quite a long time for the graph initialisation when u use a large-scale dataset. For example, I took approx 15 mins only for the graph initialisation when I used MSMT dataset containing 126K images. you just need to wait until graph initialisation or re-initialisation is over.

…

---------------------------------------------------------------------------------------------------------- Ph.D Jongmin Yu Research Associate Institute of IT Convergence Korea Advanced Institute of Science and Technology (KAIST) KAIST 291 Daehak-ro, Yuseong-gu. Daejeon 34141, Republic of Korea E-mail: ***@***.*** ***@***.*** ***@***.***> mail.com)

----------------------------------------------------------------------------------------------------------

On Sun, 26 Sept 2021 at 16:34, vegetablebird5 ***@***.***> wrote: Firstly,Thanks for your reply. Is it the mean that when you training the model yourself ,you also have the same issue on GPU utilization,right? Secondly, that is to say ,this issue is a normal phenomenon？Last but not the least，is there any method to solve the problem. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFO2B46AS722MJHWOFI47DUD3LKBANCNFSM5EYGYDSA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[vegetablebird5]

okay,I got it. thanks a lot again. I understand what do you mean，when using graph initialization on large-scale datasets,it takes a long time. To sum up,thanks for your reply.