-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
GPU利用率问题 #1
Comments
Dear friend
I can see a similar situation when the feature memory is initialised or
re-initialised.
it seems like cos GPUs are only working for extracting features, and not
for gradient computation.
So, when u see this situation, and if u can see the messages 'Graph
Re-initisliation' or 'Look-up table Overhaul - [reinitialising]',
it is normal.
best regards.
…----------------------------------------------------------------------------------------------------------
Ph.D Jongmin Yu
Research Associate
Institute of IT Convergence
Korea Advanced Institute of Science and Technology (KAIST)
KAIST 291 Daehak-ro, Yuseong-gu. Daejeon 34141, Republic of Korea
E-mail: ***@***.*** ***@***.*** ***@***.***>
mail.com)
----------------------------------------------------------------------------------------------------------
On Sun, 26 Sept 2021 at 09:45, vegetablebird5 ***@***.***> wrote:
[image: image]
<https://user-images.githubusercontent.com/85222184/134790343-86e1e9f2-23cb-4155-8364-a359b37a60a3.png>
作者您好,首先感谢您的代码开源,这里有个问题想请教您,就是在使用GPU进行训练时,显存占的很高,但是GPU利用率很低,请问是什么原因呢?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#1>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADFO2B7WZCLPV4G3CJYUMG3UDZ3MZANCNFSM5EYGYDSA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
Firstly,Thanks for your reply. Is it the mean that when you training the model yourself ,you also have the same issue on GPU utilization,right? Secondly, that is to say ,this issue is a normal phenomenon?Last but not the least,is there any method to solve the problem. |
Dear friend.
This is my answer.
1. I also saw the same situation when I did some experiments.
2. Idk it, is normal or not. but, during my experiments, it wasn't a
problem in optimising a model.
it took a quite a long time for the graph initialisation when u
use a large-scale dataset. For example, I took approx 15 mins only for the
graph initialisation when I used MSMT dataset containing 126K images.
you just need to wait until graph initialisation or re-initialisation is
over.
…----------------------------------------------------------------------------------------------------------
Ph.D Jongmin Yu
Research Associate
Institute of IT Convergence
Korea Advanced Institute of Science and Technology (KAIST)
KAIST 291 Daehak-ro, Yuseong-gu. Daejeon 34141, Republic of Korea
E-mail: ***@***.*** ***@***.*** ***@***.***>
mail.com)
----------------------------------------------------------------------------------------------------------
On Sun, 26 Sept 2021 at 16:34, vegetablebird5 ***@***.***> wrote:
Firstly,Thanks for your reply. Is it the mean that when you training the
model yourself ,you also have the same issue on GPU utilization,right?
Secondly, that is to say ,this issue is a normal phenomenon?Last but not
the least,is there any method to solve the problem.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADFO2B46AS722MJHWOFI47DUD3LKBANCNFSM5EYGYDSA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
|
okay,I got it. thanks a lot again. I understand what do you mean,when using graph initialization on large-scale datasets,it takes a long time. To sum up,thanks for your reply. |
Hello, author, first of all thank you for your open source code. Here is a question I want to ask you, that is, when using GPU for training, the GPU memory is occupied very high , but the GPU utilization is very low. What is the reason about this issue?
The text was updated successfully, but these errors were encountered: