How does the output from get_metrics differ from the loss from PITLossWrapper?

[denmonz]

When computing loss, it's my understanding that PITLossWrapper returns an SI-SDR score as a loss when initialized with the following parameters:

loss = PITLossWrapper(PairwiseNegSDR("sisdr"), pit_from='perm_avg')

but the codebase also has a function get_metrics() in asteroid\metrics.py that accesses a site-package pb_bss_eval which also returns an SI-SDR score (among other metrics). One major difference being that PITLossWrapper works well with the tensors stored on GPU, whereas get_metrics () errors in pb_bss_eval\evaluation\module_si_sdr.py line 37, stating that the tensors must first be detatched from GPU and stored on CPU in order for the computations to be done. Another difference being that PITLossWrapper works with batches, but for get_metrics(), you need to iterate over each instance in the batch and manually average the values.

So my questions are:

1. What's the difference between the loss retrieved from PITLossWrapper and the SI-SDR score retrieved from get_metrics() and
2. If there is a difference that would impact training, would it be worth re-writing the evaluation\ folder to work with tensors stored on GPU?

[mpariente]

1. SI-SDR are pretty much the same. We do remove the mean in the version for loss computation.
2. The evaluation also includes PESQ, STOI etc.. that work on CPU and we use reference implementation for all the metrics, to be have trustable results, so I would not rewrite metrics on GPU.

[denmonz]

Thank you for your reply! You've been so helpful and responsive.

Another quick question for you:
This presentation @ timestep 3:07 mentions that PIT loss function is preferred for differentiating sources of the same type. In my data I have different, but very similar source types. In your opinion, is the way I initialize the PITLossWrapper the best way of handling the loss for my data? For context, the domain I'm coding against is neither for speech nor music.

[denmonz]

For my use-case, I want to use SI-SDR as a loss function. After some investigating, it seems as though using the PITLossWrapper with the PairwiseNegSDR loss and pit_from setting equal to pw_mtx will provide the best results for my training model. @mpariente , correct me if I'm wrong, but the other pit_from settings, namely pw_pt and perm_avg aren't adequate for my setup with one mixture (X) containing multiple (2 or more) sources (y), and since PairwiseNegSDR loss already computes the pairwise loss, I would want to use the pit_from setting.

[mpariente]

Indeed.