Contributor Covenant Code of Conduct
@@ -151,116 +74,97 @@Contributor Covenant Code of Conduct
This document was adapted from the Tidyverse Code of Conduct.
--Our Pledge
+This document was adapted from the Tidyverse Code of Conduct.
+Our Pledge
We as members, contributors, and leaders pledge to make participation in our community a harassment-free experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex characteristics, gender identity and expression, level of experience, education, socio-economic status, nationality, personal appearance, race, religion, or sexual identity and orientation.
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and healthy community.
-Our Standards
+Our Standards
Examples of behavior that contributes to a positive environment for our community include:
--
-
- Demonstrating empathy and kindness toward other people +
- Demonstrating empathy and kindness toward other people
- Being respectful of differing opinions, viewpoints, and experiences
- Giving and gracefully accepting constructive feedback
- Accepting responsibility and apologizing to those affected by our mistakes, and learning from the experience
- Focusing on what is best not just for us as individuals, but for the overall community -
- The use of sexualized language or imagery, and sexual attention or advances of any kind +
- The use of sexualized language or imagery, and sexual attention or advances of any kind
- Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment
- Publishing others’ private information, such as a physical or email address, without their explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional setting -
Examples of unacceptable behavior include:
--
-
Examples of unacceptable behavior include:
+Enforcement Responsibilities
Community leaders are responsible for clarifying and enforcing our standards of acceptable behavior and will take appropriate and fair corrective action in response to any behavior that they deem inappropriate, threatening, offensive, or harmful.
Community leaders have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, and will communicate reasons for moderation decisions when appropriate.
-Scope
+Scope
This Code of Conduct applies within all community spaces, and also applies when an individual is officially representing the community in public spaces. Examples of representing our community include using an official e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event.
-Enforcement
+Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the community leaders responsible for enforcement at [INSERT CONTACT METHOD]. All complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of any incident.
-Enforcement Guidelines
+Enforcement Guidelines
Community leaders will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:
--1. Correction
+1. Correction
Community Impact: Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community.
Consequence: A private, written warning from community leaders, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. A public apology may be requested.
-2. Warning
+2. Warning
Community Impact: A violation through a single incident or series of actions.
Consequence: A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces as well as external channels like social media. Violating these terms may lead to a temporary or permanent ban.
-3. Temporary Ban
+3. Temporary Ban
Community Impact: A serious violation of community standards, including sustained inappropriate behavior.
Consequence: A temporary ban from any sort of interaction or public communication with the community for a specified period of time. No public or private interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, is allowed during this period. Violating these terms may lead to a permanent ban.
-4. Permanent Ban
+4. Permanent Ban
Community Impact: Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals.
Consequence: A permanent ban from any sort of public interaction within the community.
-Attribution
-This Code of Conduct is adapted from the Contributor Covenant, version 2.0, available at https://www.contributor-covenant.org/version/2/0/ code_of_conduct.html.
-Community Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.
-For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https:// www.contributor-covenant.org/translations.
+Attribution
+This Code of Conduct is adapted from the Contributor Covenant, version 2.0, available at https://www.contributor-covenant.org/version/2/0/ code_of_conduct.html.
+Community Impact Guidelines were inspired by Mozilla’s code of conduct enforcement ladder.
+For answers to common questions about this code of conduct, see the FAQ at https://www.contributor-covenant.org/faq. Translations are available at https:// www.contributor-covenant.org/translations.
Contributing to mikropml
@@ -151,40 +74,31 @@Contributing to mikropml
This document was adapted from the Tidyverse Contributing guide.
--Fixing typos
-You can fix typos, spelling mistakes, or grammatical errors in the documentation directly using the GitHub web interface, as long as the changes are made in the source file. This generally means you’ll need to edit roxygen2 comments in an .R, not a .Rd file. You can find the .R file that generates the .Rd by reading the comment in the first line.
This document was adapted from the Tidyverse Contributing guide.
+Fixing typos
+You can fix typos, spelling mistakes, or grammatical errors in the documentation directly using the GitHub web interface, as long as the changes are made in the source file. This generally means you’ll need to edit roxygen2 comments in an .R, not a .Rd file. You can find the .R file that generates the .Rd by reading the comment in the first line.
-Bigger changes
-If you want to make a bigger change, it’s a good idea to first file an issue and make sure someone from the team agrees that it’s needed. If you’ve found a bug, please file an issue that illustrates the bug with a minimal reprex (this will also help you write a unit test, if needed).
--Pull request process
--
-
Fork the package and clone onto your computer. If you haven’t done this before, we recommend using
usethis::create_from_github("SchlossLab/mikropml", fork = TRUE).
-Install all development dependences with
devtools::install_dev_deps(), and then make sure the package passes R CMD check by runningdevtools::check(). If R CMD check doesn’t pass cleanly, it’s a good idea to ask for help before continuing.
+Fork the package and clone onto your computer. If you haven’t done this before, we recommend using
usethis::create_from_github("SchlossLab/mikropml", fork = TRUE).
+Install all development dependences with
devtools::install_dev_deps(), and then make sure the package passes R CMD check by runningdevtools::check(). If R CMD check doesn’t pass cleanly, it’s a good idea to ask for help before continuing.Create a Git branch for your pull request (PR). We recommend using
usethis::pr_init("brief-description-of-change").
-Make your changes, commit to git, and then create a PR by running
usethis::pr_push(), and following the prompts in your browser. The title of your PR should briefly describe the change. The body of your PR should containFixes #issue-number.
-For user-facing changes, add a bullet to the top of
NEWS.md(i.e. just below the first header). Follow the style described in https://style.tidyverse.org/news.html.
-Make your changes, commit to git, and then create a PR by running
usethis::pr_push(), and following the prompts in your browser. The title of your PR should briefly describe the change. The body of your PR should containFixes #issue-number.
+For user-facing changes, add a bullet to the top of
NEWS.md(i.e. just below the first header). Follow the style described in https://style.tidyverse.org/news.html.
+New code should follow the tidyverse style guide. You can use the styler package to apply these styles, but please don’t restyle code that has nothing to do with your PR.
+We use roxygen2, with Markdown syntax, for documentation.
+We use testthat for unit tests. Contributions with test cases included are easier to accept.
+New code should follow the tidyverse style guide. You can use the styler package to apply these styles, but please don’t restyle code that has nothing to do with your PR.
-We use roxygen2, with Markdown syntax, for documentation.
-We use testthat for unit tests. Contributions with test cases included are easier to accept.
-
Bigger changes
+If you want to make a bigger change, it’s a good idea to first file an issue and make sure someone from the team agrees that it’s needed. If you’ve found a bug, please file an issue that illustrates the bug with a minimal reprex (this will also help you write a unit test, if needed).
+Pull request process
+Code style
+-Code style
--
-
-Code of Conduct
+Code of Conduct
Please note that the mikropml project is released with a Contributor Code of Conduct. By contributing to this project you agree to abide by its terms.
License
@@ -156,31 +79,27 @@License
MIT License
@@ -160,31 +83,27 @@MIT License
Getting help with mikropml
@@ -152,25 +75,20 @@Getting help with mikropml
Thanks for using mikropml! Before filing an issue, there are a few places to explore and pieces to put together to make the process as smooth as possible.
--Make a reprex
-Start by making a minimal reproducible example using the reprex package. If you haven’t heard of or used reprex before, you’re in for a treat! Seriously, reprex will make all of your R-question-asking endeavors easier (which is a pretty insane ROI for the five to ten minutes it’ll take you to learn what it’s all about). For additional reprex pointers, check out the Get help! section of the tidyverse site.
+Make a reprex
+Start by making a minimal reproducible example using the reprex package. If you haven’t heard of or used reprex before, you’re in for a treat! Seriously, reprex will make all of your R-question-asking endeavors easier (which is a pretty insane ROI for the five to ten minutes it’ll take you to learn what it’s all about). For additional reprex pointers, check out the Get help! section of the tidyverse site.
-Where to ask?
-Armed with your reprex, the next step is to figure out where to ask.
--
-
If it’s a question: start with community.rstudio.com, and/or StackOverflow. There are more people there to answer questions.
-If it’s a bug: you’re in the right place, file an issue.
+If it’s a question: start with community.rstudio.com, and/or StackOverflow. There are more people there to answer questions.
+If it’s a bug: you’re in the right place, file an issue.
If you’re not sure: let the community help you figure it out! If your problem is a bug or a feature request, you can easily return here and report it.
-
Where to ask?
+Armed with your reprex, the next step is to figure out where to ask.
+Before opening a new issue, be sure to search issues and pull requests to make sure the bug hasn’t been reported and/or already fixed in the development version. By default, the search will be pre-populated with is:issue is:open. You can edit the qualifiers (e.g. is:pr, is:closed) as needed. For example, you’d simply remove is:open to search all issues in the repo, open or closed.
Before opening a new issue, be sure to search issues and pull requests to make sure the bug hasn’t been reported and/or already fixed in the development version. By default, the search will be pre-populated with is:issue is:open. You can edit the qualifiers (e.g. is:pr, is:closed) as needed. For example, you’d simply remove is:open to search all issues in the repo, open or closed.
-What happens next?
+What happens next?
To be as efficient as possible, development of tidyverse packages tends to be very bursty, so you shouldn’t worry if you don’t get an immediate response. Typically we don’t look at a repo until a sufficient quantity of issues accumulates, then there’s a burst of intense activity as we focus our efforts. That makes development more efficient because it avoids expensive context switching between problems, at the cost of taking longer to get back to you. This process makes a good reprex particularly important because it might be multiple months between your initial report and when we start working on it. If we can’t reproduce the bug, we can’t fix it!
Articles
@@ -153,46 +76,40 @@Articles
Paper
- -Vignettes
--
-
- Introduction to mikropml -
- -
- Preprocessing data -
- -
- Hyperparameter tuning -
- -
- Parallel processing -
- -
-
-
- - - - - -
- Reference @@ -61,12 +57,14 @@
-
-
+
@@ -106,10 +104,10 @@
@@ -129,23 +127,25 @@
Introduction to mikropml
-Zena Lapp
+Zena Lapp
- Source:vignettes/introduction.Rmd+ Source:vignettes/introduction.Rmdintroduction.RmdZena Lapp
- Model performance metrics
- (Optional) feature importance metrics
---It’s running so slow!
++-It’s running so slow! +
Since I assume a lot of you won’t read this entire vignette, I’m going to say this at the beginning. If the
run_ml()function is running super slow, you should consider parallelizing. Seevignette("parallel")for examples.--Understanding the inputs
---The input data
++Understanding the inputs +
++-The input data +
The input data to
run_ml()is a dataframe where each row is a sample or observation. One column (assumed to be the first) is the outcome of interest, and all of the other columns are the features. We packageotu_mini_binas a small example dataset withmikropml.#install.packages("devtools") #devtools::install_github("SchlossLab/mikropml") -library(mikropml) -head(otu_mini_bin) +library(mikropml) +#> Warning in register(): Can't find generic `scale_type` in package ggplot2 to +#> register S3 method. +head(otu_mini_bin) #> dx Otu00001 Otu00002 Otu00003 Otu00004 Otu00005 Otu00006 Otu00007 #> 1 normal 350 268 213 1 208 230 70 #> 2 normal 568 1320 13 293 671 103 48 @@ -161,13 +161,13 @@#> 5 1 537 533 #> 6 251 155 122
Here,
-dxis the outcome column (normal or cancer), and there are 10 features (Otu00001throughOtu00010). Because there are only 2 outcomes, we will be performing binary classification in the majority of the examples below. At the bottom, we will also briefly provide examples of multi-class and continuous outcomes. As you’ll see, you run them in the same way as for binary classification!The feature columns are the amount of each Operational Taxonomic Unit (OTU) in microbiome samples from patients with cancer and without cancer. The goal is to predict
+dx, which stands for diagnosis. This diagnosis can be cancer or not based on an individual’s microbiome. No need to understand exactly what that means, but if you’re interested you can read more about it from the original paper (Topçuoğlu et al. 2020).The feature columns are the amount of each Operational Taxonomic Unit (OTU) in microbiome samples from patients with cancer and without cancer. The goal is to predict
dx, which stands for diagnosis. This diagnosis can be cancer or not based on an individual’s microbiome. No need to understand exactly what that means, but if you’re interested you can read more about it from the original paper (Topçuoğlu et al. 2020).For real machine learning applications you’ll need to use more features, but for the purposes of this vignette we’ll stick with this example dataset so everything runs faster.
---The methods we support
-All of the methods we use are supported by a great ML wrapper package
+caret, which we use to train our machine learning models.+The methods we support +
+All of the methods we use are supported by a great ML wrapper package
caret, which we use to train our machine learning models.The methods we have tested (and their backend packages) are:
- Logistic/multiclass/linear regression (
"glmnet")
@@ -176,19 +176,18 @@ - Support vector machine with a radial basis kernel (
"svmRadial") - xgboost (
"xgbTree")
For documentation on these methods, as well as many others, you can look at the available models (or see here for a list by tag). While we have not vetted the other models used by
+caret, our function is general enough that others might work. While we can’t promise that we can help with other models, feel free to open an issue on GitHub if you have questions about other models and we might be able to help.For documentation on these methods, as well as many others, you can look at the available models (or see here for a list by tag). While we have not vetted the other models used by
caret, our function is general enough that others might work. While we can’t promise that we can help with other models, feel free to open an issue on GitHub if you have questions about other models and we might be able to help.We will first focus on
glmnet, which is our default implementation of L2-regularized logistic regression. Then we will cover a few other examples towards the end.--Before running ML
++-Before running ML +
Before you execute
run_ml(), you should consider preprocessing your data, either on your own or with thepreprocess_data()function. You can learn more about this in the preprocessing vignette:vignette("preprocess").---The simplest way to
+run_ml()-+The simplest way to
run_ml()+As mentioned above, the minimal input is your dataset (
dataset) and the machine learning model you want to use (method).You may also want to provide:
-
@@ -209,12 +208,12 @@
Now, let’s dig into the output a bit. The results is a list of 4 things:
--
names(results) +names(results) #> [1] "trained_model" "test_data" "performance" #> [4] "feature_importance"
+trained_modelis the trained model fromcaret. There is a bunch of info in this that we won’t get into, because you can learn more from thecaret::train()documentation.trained_modelis the trained model fromcaret. There is a bunch of info in this that we won’t get into, because you can learn more from thecaret::train()documentation.-
names(results$trained_model) +names(results$trained_model) #> [1] "method" "modelInfo" "modelType" "results" "pred" #> [6] "bestTune" "call" "dots" "metric" "control" #> [11] "finalModel" "preProcess" "trainingData" "resample" "resampledCM" @@ -222,7 +221,7 @@#> [21] "terms" "coefnames" "xlevels"
test_datais the partition of the dataset that was used for testing. In machine learning, it’s always important to have a held-out test dataset that is not used in the training stage. In this pipeline we do that usingrun_ml()where we split your data into training and testing sets. The training data are used to build the model (e.g. tune hyperparameters, learn the data) and the test data are used to evaluate how well the model performs.-
head(results$test_data) +head(results$test_data) #> dx Otu00009 Otu00005 Otu00010 Otu00001 Otu00008 Otu00004 Otu00003 #> 9 normal 119 142 248 256 363 112 871 #> 14 normal 60 209 70 86 96 1 123 @@ -240,13 +239,13 @@performanceis a dataframe of (mainly) performance metrics (1 column for cross-validation performance metric, several for test performance metrics, and 2 columns at the end with ML method and seed):+#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> +#> 1 0.622 0.684 0.647 0.606 0.590 0.179 0.6 0.6 0.579 +#> # … with 8 more variables: Pos_Pred_Value <dbl>, Neg_Pred_Value <dbl>, +#> # Precision <dbl>, Recall <dbl>, Detection_Rate <dbl>, +#> # Balanced_Accuracy <dbl>, method <chr>, seed <dbl>results$performance -#> # A tibble: 1 × 17 +#> # A tibble: 1 × 17 #> cv_metric_AUC logLoss AUC prAUC Accuracy Kappa F1 Sensitivity Specificity -#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> -#> 1 0.622 0.684 0.647 0.606 0.590 0.179 0.6 0.6 0.579 -#> # … with 8 more variables: Pos_Pred_Value <dbl>, Neg_Pred_Value <dbl>, -#> # Precision <dbl>, Recall <dbl>, Detection_Rate <dbl>, -#> # Balanced_Accuracy <dbl>, method <chr>, seed <dbl>When using logistic regression for binary classification, area under the receiver-operator characteristic curve (AUC) is a useful metric to evaluate model performance. Because of that, it’s the default that we use for
mikropml. However, it is crucial to evaluate your model performance using multiple metrics. Below you can find more information about other performance metrics and how to use them in our package.cv_metric_AUCis the AUC for the cross-validation folds for the training data. This gives us a sense of how well the model performs on the training data.Most of the other columns are performance metrics for the test data — the data that wasn’t used to build the model. Here, you can see that the AUC for the test data is not much above 0.5, suggesting that this model does not predict much better than chance, and that the model is overfit because the cross-validation AUC (
@@ -256,14 +255,13 @@cv_metric_AUC, measured during training) is much higher than the testing AUC. This isn’t too surprising since we’re using so few features with this example dataset, so don’t be discouraged. The default option also provides a number of other performance metrics that you might be interested in, including area under the precision-recall curve (prAUC).results$feature_importance #> [1] "Skipped feature importance"--Customizing parameters
-There are a few arguments that allow you to change how you execute
-run_ml(). We’ve chosen reasonable defaults for you, but we encourage you to change these if you think something else would be better for your data.--Changing
kfold,cv_times, andtraining_frac++Customizing parameters
+There are a few arguments that allow you to change how you execute
+run_ml(). We’ve chosen reasonable defaults for you, but we encourage you to change these if you think something else would be better for your data.+Changing
kfold,cv_times, andtraining_frac+-
kfold: The number of folds to run for cross-validation (default: 5).
@@ -291,14 +289,14 @@
#> This warning usually means that the model didn't converge in some cross-validation folds because it is predicting something close to a constant. As a result, certain performance metrics can't be calculated. This suggests that some of the hyperparameters chosen are doing very poorly. #> Training complete.
You might have noticed that this one ran faster — that’s because we reduced
-kfoldandcv_times. This is okay for testing things out and may even be necessary for smaller datasets. But in general it may be better to have larger numbers for these parameters; we think the defaults are a good starting point (Topçuoğlu et al. 2020).---Custom training indices
++Custom training indices +
When
training_fracis a fraction between 0 and 1, a random sample of observations in the dataset are chosen for the training set to satisfy thetraining_frac. However, in some cases you might wish to control exactly which observations are in the training set. You can instead assigntraining_fraca vector of indices that correspond to which rows of the dataset should go in the training set (all remaining sequences will go in the testing set).-
n_obs <- otu_mini_bin %>% nrow() +n_obs <- otu_mini_bin %>% nrow() training_size <- 0.8 * n_obs -training_rows <- sample(n_obs, training_size) +training_rows <- sample(n_obs, training_size) results_custom_train <- run_ml(otu_mini_bin, 'glmnet', kfold = 2, @@ -313,21 +311,21 @@#> Training complete.
--Changing the performance metric
+++Changing the performance metric +
There are two arguments that allow you to change what performance metric to use for model evaluation, and what performance metrics to calculate using the test data.
-perf_metric_functionis the function used to calculate the performance metrics.The default for classification is
+caret::multiClassSummary()and the default for regression iscaret::defaultSummary(). We’d suggest not changing this unless you really know what you’re doing.The default for classification is
caret::multiClassSummary()and the default for regression iscaret::defaultSummary(). We’d suggest not changing this unless you really know what you’re doing.perf_metric_nameis the column name from the output ofperf_metric_function. We chose reasonable defaults (AUC for binary, logLoss for multiclass, and RMSE for continuous), but the default functions calculate a bunch of different performance metrics, so you can choose a different one if you’d like.The default performance metrics available for classification are:
-
+#> [1] "logLoss" "AUC" "prAUC" -#> [4] "Accuracy" "Kappa" "Mean_F1" -#> [7] "Mean_Sensitivity" "Mean_Specificity" "Mean_Pos_Pred_Value" -#> [10] "Mean_Neg_Pred_Value" "Mean_Precision" "Mean_Recall" -#> [13] "Mean_Detection_Rate" "Mean_Balanced_Accuracy"#> [1] "logLoss" "AUC" "prAUC" +#> [4] "Accuracy" "Kappa" "Mean_F1" +#> [7] "Mean_Sensitivity" "Mean_Specificity" "Mean_Pos_Pred_Value" +#> [10] "Mean_Neg_Pred_Value" "Mean_Precision" "Mean_Recall" +#> [13] "Mean_Detection_Rate" "Mean_Balanced_Accuracy"The default performance metrics available for regression are:
-
+#> [1] "RMSE" "Rsquared" "MAE"#> [1] "RMSE" "Rsquared" "MAE"Here’s an example using prAUC instead of AUC:
-results_pr <- run_ml(otu_mini_bin, @@ -346,23 +344,23 @@You’ll see that the cross-validation metric is prAUC, instead of the default AUC:
-results_pr$performance -#> # A tibble: 1 × 17 +#> # A tibble: 1 × 17 #> cv_metric_prAUC logLoss AUC prAUC Accuracy Kappa F1 Sensitivity -#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> -#> 1 0.577 0.691 0.663 0.605 0.538 0.0539 0.690 1 -#> # … with 9 more variables: Specificity <dbl>, Pos_Pred_Value <dbl>, -#> # Neg_Pred_Value <dbl>, Precision <dbl>, Recall <dbl>, Detection_Rate <dbl>, -#> # Balanced_Accuracy <dbl>, method <chr>, seed <dbl>-+-Using groups
+#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> +#> 1 0.577 0.691 0.663 0.605 0.538 0.0539 0.690 1 +#> # … with 9 more variables: Specificity <dbl>, Pos_Pred_Value <dbl>, +#> # Neg_Pred_Value <dbl>, Precision <dbl>, Recall <dbl>, Detection_Rate <dbl>, +#> # Balanced_Accuracy <dbl>, method <chr>, seed <dbl>+Using groups +
The optional
groupsis a vector of groups to keep together when splitting the data into train and test sets and for cross-validation. Sometimes it’s important to split up the data based on a grouping instead of just randomly. This allows you to control for similarities within groups that you don’t want to skew your predictions (i.e. batch effects). For example, with biological data you may have samples collected from multiple hospitals, and you might like to keep observations from the same hospital in the same partition.Here’s an example where we split the data into train/test sets based on groups:
# make random groups -set.seed(2019) -grps <- sample(LETTERS[1:8], nrow(otu_mini_bin), replace=TRUE) +set.seed(2019) +grps <- sample(LETTERS[1:8], nrow(otu_mini_bin), replace=TRUE) results_grp <- run_ml(otu_mini_bin, 'glmnet', cv_times = 2, @@ -377,9 +375,9 @@#> Training the model... #> Training complete.
The one difference here is
-run_ml()will report how much of the data is in the training set if you run the above code chunk. This can be a little finicky depending on how many samples and groups you have. This is because it won’t be exactly what you specify withtraining_frac, since you have to include all of one group in either the training set or the test set.---Controling how groups are assigned to partitions
++Controling how groups are assigned to partitions +
When you use the
groupsparameter as above, by defaultrun_ml()will assume that you want all of the observations from each group to be placed in the same partition of the train/test split. This makes sense when you want to use groups to control for batch effects. However, in some cases you might prefer to control exactly which groups end up in which partition, and you might even be okay with some observations from the same group being assigned to different partitions.For example, say you want groups A and B to be used for training, C and D for testing, and you don’t have a preference for what happens to the other groups. You can give the
group_partitionsparameter a named list to specify which groups should go in the training set and which should go in the testing set.@@ -388,8 +386,8 @@
cv_times = 2, training_frac = 0.8, groups = grps, - group_partitions = list(train = c('A', 'B'), - test = c('C', 'D') + group_partitions = list(train = c('A', 'B'), + test = c('C', 'D') ), seed = 2019) #> Using 'dx' as the outcome column. @@ -408,8 +406,8 @@
kfold = 2, training_frac = 0.5, groups = grps, - group_partitions = list(train = c("A", "B", "C", "D", "E", "F"), - test = c("A", "B", "C", "D", "E", "F", "G", "H") + group_partitions = list(train = c("A", "B", "C", "D", "E", "F"), + test = c("A", "B", "C", "D", "E", "F", "G", "H") ), seed = 2019) #> Using 'dx' as the outcome column. @@ -423,9 +421,9 @@
--Finding feature importance
++Finding feature importance +
To find which features are contributing to predictive power, you can use
find_feature_importance = TRUE. How we use permutation importance to determine feature importance is described in (Topçuoğlu et al. 2020). Briefly, it permutes each of the features individually (or correlated ones together) and evaluates how much the performance metric decreases. The more performance decreases when the feature is randomly shuffled, the more important that feature is. The default isFALSEbecause it takes a while to run and is only useful if you want to know what features are important in predicting your outcome.Let’s look at some feature importance results:
@@ -466,7 +464,7 @@
seed: The seed (if set).As you can see here, the differences are negligible (close to zero), which makes sense since our model isn’t great. If you’re interested in feature importance, it’s especially useful to run multiple different train/test splits, as shown in our example snakemake workflow.
+As you can see here, the differences are negligible (close to zero), which makes sense since our model isn’t great. If you’re interested in feature importance, it’s especially useful to run multiple different train/test splits, as shown in our example snakemake workflow.
You can also choose to permute correlated features together using
corr_thresh(default: 1). Any features that are above the correlation threshold are permuted together; i.e. perfectly correlated features are permuted together when using the default value.-results_imp_corr <- run_ml(otu_mini_bin, @@ -502,18 +500,18 @@If you previously executed
run_ml()without feature importance but now wish to find feature importance after the fact, see the example code in theget_feature_importance()documentation.get_feature_importance()can show a live progress bar, seevignette("parallel")for examples.--Tuning hyperparameters (using the
+hyperparameterargument)+-Tuning hyperparameters (using the
hyperparameterargument) +This is important, so we have a whole vignette about them. The bottom line is we provide default hyperparameters that you can start with, but it’s important to tune your hyperparameters. For more information about what the default hyperparameters are, and how to tune hyperparameters, see
vignette("tuning").--Other models
++Other models +
Here are examples of how to train and evaluate other models. The output for all of them is very similar, so we won’t go into those details.
---Random forest
++-Random forest +
results_rf <- run_ml(otu_mini_bin, 'rf', @@ -527,35 +525,35 @@ntree = 10, seed = 2019)
--Decision tree
++-Decision tree +
results_dt <- run_ml(otu_mini_bin, 'rpart2', cv_times = 5, seed = 2019)---SVM
++SVM +
-results_svm <- run_ml(otu_mini_bin, 'svmRadial', cv_times = 5, seed = 2019)If you get a message “maximum number of iterations reached”, see this issue in caret.
+If you get a message “maximum number of iterations reached”, see this issue in caret.
--Other data
---Multiclass data
++-Other data +
+++Multiclass data +
We provide
otu_mini_multiwith a multiclass outcome (three or more outcomes):-
otu_mini_multi %>% dplyr::pull('dx') %>% unique() +otu_mini_multi %>% dplyr::pull('dx') %>% unique() #> [1] "adenoma" "carcinoma" "normal"Here’s an example of running multiclass data:
-@@ -566,18 +564,18 @@
The performance metrics are slightly different, but the format of everything else is the same:
-results_multi$performance -#> # A tibble: 1 × 17 +#> # A tibble: 1 × 17 #> cv_metric_logLoss logLoss AUC prAUC Accuracy Kappa Mean_F1 Mean_Sensitivity -#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <chr> <dbl> -#> 1 1.07 1.11 0.506 0.353 0.382 0.0449 <NA> 0.360 -#> # … with 9 more variables: Mean_Specificity <dbl>, Mean_Pos_Pred_Value <chr>, -#> # Mean_Neg_Pred_Value <dbl>, Mean_Precision <chr>, Mean_Recall <dbl>, -#> # Mean_Detection_Rate <dbl>, Mean_Balanced_Accuracy <dbl>, method <chr>, -#> # seed <dbl>-+-Continuous data
+#> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <chr> <dbl> +#> 1 1.07 1.11 0.506 0.353 0.382 0.0449 NA 0.360 +#> # … with 9 more variables: Mean_Specificity <dbl>, Mean_Pos_Pred_Value <chr>, +#> # Mean_Neg_Pred_Value <dbl>, Mean_Precision <chr>, Mean_Recall <dbl>, +#> # Mean_Detection_Rate <dbl>, Mean_Balanced_Accuracy <dbl>, method <chr>, +#> # seed <dbl>+Continuous data +
And here’s an example for running continuous data, where the outcome column is numerical:
results_cont <- run_ml(otu_mini_bin[, 2:11], @@ -587,21 +585,21 @@Again, the performance metrics are slightly different, but the format of the rest is the same:
+#> <dbl> <dbl> <dbl> <dbl> <chr> <dbl> +#> 1 622. 731. 0.0893 472. glmnet 2019results_cont$performance -#> # A tibble: 1 × 6 +#> # A tibble: 1 × 6 #> cv_metric_RMSE RMSE Rsquared MAE method seed -#> <dbl> <dbl> <dbl> <dbl> <chr> <dbl> -#> 1 622. 731. 0.0893 472. glmnet 2019--References
++@@ -618,11 +616,13 @@References +
-Tang, Shengpu, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W. Sjoding, and Jenna Wiens. 2020. “Democratizing EHR Analyses with FIDDLE: A Flexible Data-Driven Preprocessing Pipeline for Structured Clinical Data.” J Am Med Inform Assoc, October. https://doi.org/10.1093/jamia/ocaa139.
+Tang, Shengpu, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W. Sjoding, and Jenna Wiens. 2020. “Democratizing EHR Analyses with FIDDLE: A Flexible Data-Driven Preprocessing Pipeline for Structured Clinical Data.” J Am Med Inform Assoc, October. https://doi.org/10.1093/jamia/ocaa139.
-Topçuoğlu, Begüm D., Nicholas A. Lesniak, Mack T. Ruffin, Jenna Wiens, and Patrick D. Schloss. 2020. “A Framework for Effective Application of Machine Learning to Microbiome-Based Classification Problems.” mBio 11 (3). https://doi.org/10.1128/mBio.00434-20.
+Topçuoğlu, Begüm D., Nicholas A. Lesniak, Mack T. Ruffin, Jenna Wiens, and Patrick D. Schloss. 2020. “A Framework for Effective Application of Machine Learning to Microbiome-Based Classification Problems.” mBio 11 (3). https://doi.org/10.1128/mBio.00434-20.
@@ -631,5 +631,7 @@
+ + diff --git a/docs/articles/paper.html b/docs/articles/paper.html index 9b4baf38..c3041b11 100644 --- a/docs/articles/paper.html +++ b/docs/articles/paper.html @@ -26,6 +26,8 @@ + +
@@ -38,18 +40,12 @@-
-
- - - - - -
- Reference @@ -61,12 +57,14 @@
-
-
+
@@ -106,96 +104,96 @@
-
mikropml: User-Friendly R Package for Supervised Machine Learning Pipelines
-Begüm D. Topçuoğlu, Zena Lapp, Kelly L. Sovacool, Evan Snitkin, Jenna Wiens, Patrick D. Schloss
+Begüm D. Topçuoğlu, Zena Lapp, Kelly L. Sovacool, Evan Snitkin, Jenna Wiens, Patrick D. Schloss
-2020
+2020
- Source:vignettes/paper.Rmd+ Source:vignettes/paper.Rmdpaper.Rmd--Summary
++-Summary +
-
Machine learning (ML) for classification and prediction based on a set of features is used to make decisions in healthcare, economics, criminal justice and more. However, implementing an ML pipeline including preprocessing, model selection, and evaluation can be time-consuming, confusing, and difficult. Here, we present
+mikropml(prononced “meek-ROPE em el”), an easy-to-use R package that implements ML pipelines using regression, support vector machines, decision trees, random forest, or gradient-boosted trees. The package is available on GitHub, CRAN, and conda.Machine learning (ML) for classification and prediction based on a set of features is used to make decisions in healthcare, economics, criminal justice and more. However, implementing an ML pipeline including preprocessing, model selection, and evaluation can be time-consuming, confusing, and difficult. Here, we present
mikropml(prononced “meek-ROPE em el”), an easy-to-use R package that implements ML pipelines using regression, support vector machines, decision trees, random forest, or gradient-boosted trees. The package is available on GitHub, CRAN, and conda.--Statement of need
++-Statement of need +
Most applications of machine learning (ML) require reproducible steps for data pre-processing, cross-validation, testing, model evaluation, and often interpretation of why the model makes particular predictions. Performing these steps is important, as failure to implement them can result in incorrect and misleading results (Teschendorff 2019; Wiens et al. 2019).
Supervised ML is widely used to recognize patterns in large datasets and to make predictions about outcomes of interest. Several packages including
-caret(Kuhn 2008) andtidymodels(Kuhn, Wickham, and RStudio 2020) in R,scikitlearn(Pedregosa et al. 2011) in Python, and the H2OautoMLplatform (H2O.ai 2020) allow scientists to train ML models with a variety of algorithms. While these packages provide the tools necessary for each ML step, they do not implement a complete ML pipeline according to good practices in the literature. This makes it difficult for practitioners new to ML to easily begin to perform ML analyses.To enable a broader range of researchers to apply ML to their problem domains, we created
+mikropml, an easy-to-use R package (R Core Team 2020) that implements the ML pipeline created by Topçuoğlu et al. (Topçuoğlu et al. 2020) in a single function that returns a trained model, model performance metrics and feature importance.mikropmlleverages thecaretpackage to support several ML algorithms: linear regression, logistic regression, support vector machines with a radial basis kernel, decision trees, random forest, and gradient boosted trees. It incorporates good practices in ML training, testing, and model evaluation (Topçuoğlu et al. 2020; Teschendorff 2019). Furthermore, it provides data preprocessing steps based on the FIDDLE (FlexIble Data-Driven pipeLinE) framework outlined in Tang et al. (Tang et al. 2020) and post-training permutation importance steps to estimate the importance of each feature in the models trained (Breiman 2001; Fisher, Rudin, and Dominici 2018).To enable a broader range of researchers to apply ML to their problem domains, we created
mikropml, an easy-to-use R package (R Core Team 2020) that implements the ML pipeline created by Topçuoğlu et al. (Topçuoğlu et al. 2020) in a single function that returns a trained model, model performance metrics and feature importance.mikropmlleverages thecaretpackage to support several ML algorithms: linear regression, logistic regression, support vector machines with a radial basis kernel, decision trees, random forest, and gradient boosted trees. It incorporates good practices in ML training, testing, and model evaluation (Topçuoğlu et al. 2020; Teschendorff 2019). Furthermore, it provides data preprocessing steps based on the FIDDLE (FlexIble Data-Driven pipeLinE) framework outlined in Tang et al. (Tang et al. 2020) and post-training permutation importance steps to estimate the importance of each feature in the models trained (Breiman 2001; Fisher, Rudin, and Dominici 2018).mikropmlcan be used as a starting point in the application of ML to datasets from many different fields. It has already been applied to microbiome data to categorize patients with colorectal cancer (Topçuoğlu et al. 2020), to identify differences in genomic and clinical features associated with bacterial infections (Lapp et al. 2020), and to predict gender-based biases in academic publishing (Hagan et al. 2020).--mikropml package
-The
+mikropmlpackage includes functionality to preprocess the data, train ML models, evaluate model performance, and quantify feature importance (Figure 1). We also provide vignettes and an example Snakemake workflow (Köster and Rahmann 2012) to showcase how to run an ideal ML pipeline with multiple different train/test data splits. The results can be visualized using helper functions that useggplot2(Wickham 2016).+mikropml package +
+The
mikropmlpackage includes functionality to preprocess the data, train ML models, evaluate model performance, and quantify feature importance (Figure 1). We also provide vignettes and an example Snakemake workflow (Köster and Rahmann 2012) to showcase how to run an ideal ML pipeline with multiple different train/test data splits. The results can be visualized using helper functions that useggplot2(Wickham 2016).While mikropml allows users to get started quickly and facilitates reproducibility, it is not a replacement for understanding the ML workflow which is still necessary when interpreting results (Pollard et al. 2019). To facilitate understanding and enable one to tailor the code to their application, we have heavily commented the code and have provided supporting documentation which can be read online.
---Preprocessing data
++-Preprocessing data +
We provide the function
preprocess_data()to preprocess features using several different functions from thecaretpackage.preprocess_data()takes continuous and categorical data, re-factors categorical data into binary features, and provides options to normalize continuous data, remove features with near-zero variance, and keep only one instance of perfectly correlated features. We set the default options based on those implemented in FIDDLE (Tang et al. 2020). More details on how to usepreprocess_data()can be found in the accompanying vignette.--Running ML
++-Running ML +
The main function in mikropml,
run_ml(), minimally takes in the model choice and a data frame with an outcome column and feature columns. For model choice,mikropmlcurrently supports logistic and linear regression (glmnet: Friedman, Hastie, and Tibshirani 2010), support vector machines with a radial basis kernel (kernlab: Karatzoglou et al. 2004), decision trees (rpart: Therneau et al. 2019), random forest (randomForest: Liaw and Wiener 2002), and gradient-boosted trees (xgboost: Chen et al. 2020).run_ml()randomly splits the data into train and test sets while maintaining the distribution of the outcomes found in the full dataset. It also provides the option to split the data into train and test sets based on categorical variables (e.g. batch, geographic location, etc.).mikropmluses thecaretpackage (Kuhn 2008) to train and evaluate the models, and optionally quantifies feature importance. The output includes the best model built based on tuning hyperparameters in an internal and repeated cross-validation step, model evaluation metrics, and optional feature importances. Feature importances are calculated using a permutation test, which breaks the relationship between the feature and the true outcome in the test data, and measures the change in model performance. This provides an intuitive metric of how individual features influence model performance and is comparable across model types, which is particularly useful for model interpretation (Topçuoğlu et al. 2020). Our introductory vignette contains a comprehensive tutorial on how to userun_ml().
mikropml pipeline
--Ideal workflow for running mikropml with many different train/test splits
-To investigate the variation in model performance depending on the train and test set used (Topçuoğlu et al. 2020; Lapp et al. 2020), we provide examples of how to
+run_ml()many times with different train/test splits and how to get summary information about model performance on a local computer or on a high-performance computing cluster using a Snakemake workflow.+-Ideal workflow for running mikropml with many different train/test splits +
+To investigate the variation in model performance depending on the train and test set used (Topçuoğlu et al. 2020; Lapp et al. 2020), we provide examples of how to
run_ml()many times with different train/test splits and how to get summary information about model performance on a local computer or on a high-performance computing cluster using a Snakemake workflow.--Tuning & visualization
++-Tuning & visualization +
One particularly important aspect of ML is hyperparameter tuning. We provide a reasonable range of default hyperparameters for each model type. However practitioners should explore whether that range is appropriate for their data, or if they should customize the hyperparameter range. Therefore, we provide a function
plot_hp_performance()to plot the cross-validation performance metric of a single model or models built using different train/test splits. This helps evaluate if the hyperparameter range is being searched exhaustively and allows the user to pick the ideal set. We also provide summary plots of test performance metrics for the many train/test splits with different models usingplot_model_performance(). Examples are described in the accompanying vignette on hyperparameter tuning.---Dependencies
++Dependencies +
mikropml is written in R (R Core Team 2020) and depends on several packages:
dplyr(Wickham et al. 2020),rlang(Henry, Wickham, and RStudio 2020) andcaret(Kuhn 2008). The ML algorithms supported bymikropmlrequire:glmnet(Friedman, Hastie, and Tibshirani 2010),e1071(Meyer et al. 2020), andMLmetrics(Yan 2016) for logistic regression,rpart2(Therneau et al. 2019) for decision trees,randomForest(Liaw and Wiener 2002) for random forest,xgboost(Chen et al. 2020) for xgboost, andkernlab(Karatzoglou et al. 2004) for support vector machines. We also allow for parallelization of cross-validation and other steps using theforeach,doFuture,future.apply, andfuturepackages (Bengtsson and Team 2020). Finally, we useggplot2for plotting (Wickham 2016).--Acknowledgments
++-Acknowledgments +
We thank members of the Schloss Lab who participated in code clubs related to the initial development of the pipeline, made documentation improvements, and provided general feedback. We also thank Nick Lesniak for designing the mikropml logo.
We thank the US Research Software Sustainability Institute (NSF #1743188) for providing training to KLS at the Winter School in Research Software Engineering.
--Funding
++-Funding +
Salary support for PDS came from NIH grant 1R01CA215574. KLS received support from the NIH Training Program in Bioinformatics (T32 GM070449). ZL received support from the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 1256260. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.