Computing the Random Image Correlation

[ManuelEck]

Hi guys,

this is a fantastic approach!

I understand how the self corrolation and the kNN corrolation is computed but i dont quite get how the random image corrolation is computed.

STEGO/src/modules.py

Lines 382 to 391 in eb4d6b5

	for i in range(self.cfg.neg_samples):
	perm_neg = super_perm(orig_feats.shape[0], orig_feats.device)
	feats_neg = sample(orig_feats[perm_neg], coords2)
	code_neg = sample(orig_code[perm_neg], coords2)
	neg_inter_loss, neg_inter_cd = self.helper(
	feats, feats_neg, code, code_neg, self.cfg.neg_inter_shift)
	neg_losses.append(neg_inter_loss)
	neg_cds.append(neg_inter_cd)
	neg_inter_loss = torch.cat(neg_losses, axis=0)
	neg_inter_cd = torch.cat(neg_cds, axis=0)

with perm_neg in line 383 one random image tensor of the batch gets chosen. (or its embeddings)

Then samples are created from the original and the random chosen. But only from the batch?
coords2 are coordinates that will get used for grid_sample that transforms the chosen image tensor?

afterwards the grid sample from the frozen backbone and the head will get corrolated

so I'm not sure if i don't quite grasp the code yet or this code just gets a random image from current batch?

that meaning, a big batch size is quite important to get enough negative corrolation?

anyone can help me with this one, or even @mhamilton723 ?

Thanks in advance