Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow for custom loss function to be passed as measure argument #13

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

mikoontz
Copy link

@mikoontz mikoontz commented Dec 8, 2022

Allows for a custom loss function to be supplied to the measure argument. Must take truth, response, and prob as arguments (as they are output from the predict_learner() function followed by the few modifications at the beginning of the compute_loss() function (which themselves might be further modified by a call to the modify_trp() function). Returns a single vector (which may be a one-element vector) representing the loss (i.e., 1 - model skill measure).

Addresses some of the need described in #10.

I added an example to the help file that shows an implementation of Matthew's correlation coefficient. I included a note in that example about MCC being a batch-level skill measure and so that statistical approach currently implemented in this package isn't yet set up to handle that. The work around is to have the user set up a series of test_data data frames ahead of time, which may mimic a resampling strategy (like repeated cross-validation, with some number of iterations and folds defined), then to use that series of batch-level CPI values to conduct an external significance test.

All build tests and checks pass. DESCRIPTION and cpi.Rd files changed using {roxygen2}.

…ent. Must take truth, response, and prob as arguments (as they are output from the `predict_learner()` function followed by the few modifications at the beginning of the `compute_loss()` function (which themselves might be further modified by a call to the `modify_trp()` function). Returns a single vector (which may be a one-element vector) representing the loss (i.e., 1 - model skill measure).
@mnwright
Copy link
Member

Thanks! I might want to simplify the example a little bit but I think this is very useful.

I'll als check with mlr3 if we could have some way to use all their implemented measures, i.e. without averaging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants