where is the pixelwise sound

[TaoZheng9]

Hi, I saw the func: forward_pixelwise in the code synthesizer, this is the one version of forward function that produce pixel-wise mask. However, throughout the code, and I found only the foward func is invoked but it is not the one of pixel-wise sound. Is there any demo that can produce pixel-wise sound?

[thirteen-bears]

The function forward_pixelwise(self, feats_img, feat_sound) helps compute the audio spectrum for all points on the feature map. Specifically, it calculates the correlation between the visual features (from an image or video frame) and the audio features (from a spectrogram), producing a result that associates each feature map point with its corresponding audio spectrum.

If upsampling is applied after this function, the output will correspond to the audio spectrum of each individual pixel in the original image. This means that each point in the image will have its own corresponding audio spectrum, which can be used for tasks such as sound source localization.