2 GPU is 10 times slower than 1 GPU, using colossal, normal?

[Arsmart123]

Hi! I am comparing the difference between split a single layer into two GPUs, and my result is like:
2GPUs single layer with parameter: 1,600,000*4000(about...) => 111s
1 GPU single layer with same parameter => 9s
Such delay is normal? Thank you!

[FrankLeeeee]

Hi, do you have a code snippet to reproduce this issue?

[Arsmart123]

Thank you very much!!! I will try that tomorrow!

[Arsmart123]

https://github.com/Arsmart123/colossal-time-question
I post a link here, if you could check it and help me, it will be highly appreciated!!!!

[FrankLeeeee]

Great, let me try to reproduce this problem.

[FrankLeeeee]

Hi @Arsmart123 , I have run your code. The time taken is 2.67s for one process and 2.55s for two processes.

[FrankLeeeee]

I can't reproduce your performance. Can you tell me more details about your environment?

[FrankLeeeee]

I also notice that this model is rather small. When a model can fit in one GPU, there is no need to introduce tensor parallel.

[Arsmart123]

I can't reproduce your performance. Can you tell me more details about your environment?
Hi! I really can not install the latest colossalai in NUS HPC, so I am using a older verison, below is the link:
https://github.com/Arsmart123/colossal-time-question/blob/main/ColossalAI-main-%E6%97%A7%E7%89%88%E5%A4%87%E4%BB%BD.zip

Thank you very much!!!!

[Arsmart123]

I can't reproduce your performance. Can you tell me more details about your environment?
Also, I am using torch 1.10 cu11.3