Multilabel textcat (transformers): Weights for classes in loss function (Unbalanced data)

[hp52]

Hi,

I'm training a multilabel text classificator with transformers and Spacy 3.0. My data is quite unbalanced, so I'd like to work with loss weights for the different classes. Is it possible to define custom weights that will be used in the loss function?

Thanks for your help.

[svlandeg]

I think right now, the only way to accomplish that, is to create a new subclass of the MultiLabel_TextCategorizer object and reimplement the get_loss function (note that this function is defined by the class TextCategorizer)

[vsocrates]

@einarbmag Were you able to get this to work? I've seen some other threads saying that the get_loss function is often tightly coupled with the rest of the implementation but doesn't seem that way at first glance. My plan was to simply compute the weighted MSE and multiple d_scores by the respective weights, as you describe.

I'd really appreciate seeing some sample could if you got it to work and would be willing to share! Thanks in advance!

[vsocrates]

I implemented by subclassing TextCategorizer as suggested above and overriding the relevant lines in get_loss method as shown below and it seems to work fine, just in case this helps anyone else!

    if weights is not None:
        squared = d_scores**2
        mean_square_error = numpy.sum(squared * weights) / numpy.sum(weights)
        d_scores *= weights
    else:
        mean_square_error = (d_scores**2).mean()
    # print(d_scores)    
    return float(mean_square_error), d_scores

[biomed-model]

@vsocrates I'm trying to train binary classifiers (regarding the described experimental model, e.g. animal model, cell culture, ...) using spacy.TextCatBOW.v3 for quite unbalanced classes (15% True to 85% False) in a biomedical dataset (PubMed-Abstracts). Can you provide some more information on your solution (subclassing TextCategorizer and overriding get_loss)? I'd really appreciate some guidance on this.

[vsocrates]

Here's the full file with line linked: https://github.com/vsocrates/moore-followup/blob/cfd52278bba001931a73e100eb531b4cd1dcb945/Weighted_TextCategorizer.py#L348

Hope that helps!

[biomed-model]

Yes, that helps a lot. Thank you very much!