Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a metric for empty values #50

Open
PonteIneptique opened this issue Feb 18, 2020 · 1 comment
Open

Add a metric for empty values #50

PonteIneptique opened this issue Feb 18, 2020 · 1 comment

Comments

@PonteIneptique
Copy link
Contributor

Hi Enrique !
I am thinking that run_check could use another metric for some tasks, which is Score (Not empty) (so accuracy, prediction, recall on those) .
This is something we have using eval full with classes scores, but I do think that could be useful to track over training.

Basically, in task declaration have an optional not null "empty_value": "_" new key (I am thinking about the dictionary with name in it. And compute accuracy, prediction and recall on filtered results as well. While it's important to know the accuracy of empty values as well (Tense should not be applied to Nouns), morphological features are incredibly skewed (as an exemple, there is definitely less Tense occurrences than Case for example).

Inter-categorical information is also important as well but we do have that in eval (ie. morphological features that are not empty are also skewed data: there is less subjunctive perfect than there is present, less 2nd person than there is 3rd ones...)

@emanjavacas
Copy link
Owner

I realize I never answered to this! I think this would be a cool addition indeed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants