Issue with evaluation

[Davidyao99]

I followed the steps in your readme and encountered the following error. Any idea what is causing this?

[honglin-chen]

Hi @Davidyao99, thank you for pointing out the issue. I'm able to run the evaluation script without encountering the error above. To help reproduce the error, could you please share the evaluation command and Pytorch version you are using? It will also be helpful if you can kindly print the shape of query tensor and the max / min of sample_inds tensor before line 154 in eisen.py, where the error was encountered.

[Davidyao99]

This is the result from my nvidia-smi. I am using pytorch 1.11, python 3.9.12. I am also running this on WSL2.

This is the output of the shapes and the max/min. I realised that when I try to find the max value of indices, it throws the same error. This makes me suspect that there is something wrong with this line:
indices = sample_inds[-1].view(B, T * K).unsqueeze(-1).expand(-1, -1, D)

Additionally, just wanted to check but for line 155 here where we are gathering the query vectors, should we be using sample_inds[0,2] instead of sample_inds[0,2]?

Thank you so much ;)) hope to resolve this soon

[honglin-chen]

Thank you for providing more information. The shape of the query tensor and max/min of the sample_inds tensor look correct to me. However, the shape of sample_inds seems wrong. It should have a shape of [3, 1, 16384, 1024]. To debug, we can look at the generate_affinity_sample_indices function to understand why the shapes are different.

I didn't quite understand your last question. I think it should use sample_inds[0,1] to gather the query tensor and sample_inds[0,2] to gather the key tensor.

[Davidyao99]

So sry, the shape of the sample_inds is [2, 1, 16384, 1024] on mine. The shape that I printed is the indices after performing the unsqueeze and expand. I think the issue might be in this line: indices = sample_inds[-1].view(B, T * K).unsqueeze(-1).expand(-1, -1, D). I am getting the same memory issue when calling torch.max() on the indices variable. If I remove the expand function, I am able to call torch.max() without error on indices.

With regards to my last question, perhaps I misunderstood the code. I will look at it again.