How to capture Conv-TasNet's output of encoder

[cemox35]

Hello,

I want to see the encoder output of each frame and calculate the SI-SNR score. How can I do that?

Thank you for all your answers!

[cemox35]

Any comment? 🥺 I cannot solve how to save encoders output each frame. Could you help me? @JorisCos

[JorisCos]

Hi, I am not sure, I understand. The encoder output is a 2D representation of the signal (n_filters, n_frrames). The SI-SNR is calculated on the output of the decoder which has the same dimension as the input of the network.
Do you want to compare SI-SNR without the masker ? If so, you should also consider the effect of the decoder.
To compare frame by frame an easy solution would be to slice the input and output of the network with the size of the frame (kernel size of the encoder) with a hop_size equal to stride.

[cemox35]

I just want to take one frame and calculate the SI-SNR score. However, Asteroid encoder input (batchsize, channel, time), output is (batchsize, n_filters, n_frames) with using F.conv1D. Therefore, I cannot take any function outputs.

I can take all frames then split the n_frames. Then, How can I tune the decoder part :D Mismatch input shape...

[mpariente]

I just want to take one frame and calculate the SI-SNR score

Also not clear for me. Can you clarify please?

[cemox35]

Actually, I want to take only one frame's encoder output. Sorry for telling you wrong. Also, I want to that specific frame's loss, not the whole clip. If I have 10 clips and 100 frames for each clip, I will find 10*100 SI-SNR loss value.

[mpariente]

I reformulate to make sure I understand.
You want to compute the SI-SNR for each segment corresponding to the input segmentation, is that right?

First thing to note, the frames overlap in most cases, so when you compute SI-SNR for an output segment the size of a frame, it has contributions from three frames, is this clear?

Another thing, computing SI-SNR for each segment might be weird because an optimal scaling factor will be computed for each segment, biaising the whole measure.

This said, you can assume alignement between input and output (in case training data are aligned), and compute the SI-SNR for each output frame, and link it to the corresponding input frame.