Precision/Recall

[xinwei-sher]

Hi, there:

Really nice tool to help find the best model!

In Readme.md file, precision and recall are completely different from what you can compute from the confusion matrix. I guess it's due to different threshold is used. My question is why not use a same threshold all of them?

AutoMl select the best model by logloss. Is it possible to use other metric, such as f1 or accuracy?

Metric details

	score	threshold
logloss	0.129662	nan
auc	0.902071	nan
f1	0.646154	0.244579
accuracy	0.965732	0.471619
precision	0.904762	0.516277
recall	1	1.99414e-06
mcc	0.652699	0.471619

Confusion matrix (at threshold=0.471619)

	Predicted as 0	Predicted as 1
Labeled as 0	600	3
Labeled as 1	19	20

[pplonski]

Hi @xinwei-sher, the reported metrics are maximums with provided threshold, maybe it will be worth to add one more table with metrics that are computed at the same threshold as confusion matrix? What do you think?

You can optimize other metrics by setting eval_metric, example:

automl = AutoML(eval_metric="f1")
automl.fit(X, y)

Supported evaluation metrics (eval_metric argument in AutoML())

• for binary classification: logloss, auc, f1, average_precision, accuracy- default is logloss
• for mutliclass classification: logloss, f1, accuracy - default is logloss
• for regression: rmse, mse, mae, r2, mape, spearman, pearson - default is rmse

If you don't find eval_metric that you need, please add a new issue. I will add it.

[xinwei-sher]

Thanks, @pplonski.

It definitely would help to have the extra table withe metrics all computed at the same threshold as the confusion table.

[pplonski]

I've added issue to add such functionality #422