[python-package] LGBMClassifier produces empty trees

[znacer]

Description

While exploring the trees produced by LGBMClassifier, it appears that some trees can have only one leaves.
This behavior seems strange as we would expect at least one split for every trees.

Reproducible example

import numpy as np
from sklearn.datasets import load_digits
from lightgbm import LGBMClassifier
np.random.seed(0)

# data | regression | binary | multi class
data_mult = load_digits(as_frame=True)
# Model
model_mult = LGBMClassifier(**{'verbosity': -1, }).fit(data_mult.data, data_mult.target)

for tree in model_mult._Booster.dump_model()["tree_info"]:
    if tree["num_leaves"] == 1:
        print(f"Tree {tree['tree_index']} has only one {tree['num_leaves']} leaf")

Environment info

LightGBM version or commit hash: 4.0.0

Command(s) you used to install LightGBM

pip install lightgbm

Additional Comments

This bug has been found while working on an issue on the SHAP library.

[jameslamb]

Thanks for using LightGBM and for the report.

Short Answer

we would expect at least one split for every trees

The existence of trees with 0 splits is not a bug in LightGBM.

LightGBM comes with many settings to limit overfitting, and it's completely possible that during training, it may intentionally choose to produce a tree with 0 splits.

In fact, there's even a separate thread going on here specifically about adding support for such 0-split trees in shap:

• Stump has no leaf_count inside dump_model() output #5962
• WIP: Unify dump_model output (fixes #5962) #5964

cc @thatlittleboy

---

Long Answer

For debugging activities like this, it's useful to include the logs produced by your program.

I ran the following modified version of your code (with verbosity=1):

import numpy as np
from sklearn.datasets import load_digits
from lightgbm import LGBMClassifier
np.random.seed(0)

# data | regression | binary | multi class
data_mult = load_digits(as_frame=True)

# Model
model_mult = LGBMClassifier(verbosity=1).fit(data_mult.data, data_mult.target)

for tree in model_mult._Booster.dump_model()["tree_info"]:
    if tree["num_leaves"] == 1:
        print(f"Tree {tree['tree_index']} has only one {tree['num_leaves']} leaf")

Doing that, I can see that the logs are full of this same warning:

[LightGBM] [Warning] Auto-choosing row-wise multi-threading, the overhead of testing was 0.000249 seconds.
You can set `force_row_wise=true` to remove the overhead.
And if memory is not enough, you can set `force_col_wise=true`.
[LightGBM] [Info] Total Bins 846
[LightGBM] [Info] Number of data points in the train set: 1797, number of used features: 54
[LightGBM] [Info] Start training from score -2.312090
[LightGBM] [Info] Start training from score -2.289867
[LightGBM] [Info] Start training from score -2.317724
[LightGBM] [Info] Start training from score -2.284388
[LightGBM] [Info] Start training from score -2.295377
[LightGBM] [Info] Start training from score -2.289867
[LightGBM] [Info] Start training from score -2.295377
[LightGBM] [Info] Start training from score -2.306488
[LightGBM] [Info] Start training from score -2.334819
[LightGBM] [Info] Start training from score -2.300917
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf

It appears to me that in this example, LightGBM added new trees with splits for the first few iterations, and then was not able to find any additional splits satisfying its constraints for splitting.

Examples of those constraints:

• min_gain_to_split = how much gain (in terms of the objective function) must a split offer to be added to a tree in the model?
• min_data_in_leaf = how many samples must, at minimum, fall into all leaves?

You can find documentation on these and others here:

• https://lightgbm.readthedocs.io/en/stable/Parameters.html
• https://lightgbm.readthedocs.io/en/stable/Parameters-Tuning.html

If you search for that warning message here and on Stack Overflow, you'll find many explanations of this:

• Trained tree unexpectedly contains only root  #4826
• [LightGBM] [Warning] No further splits with positive gain, best gain: -inf #4649 (comment)
• https://stackoverflow.com/questions/47770123/lightgbm-how-to-deal-with-no-further-splits-with-positive-gain-best-gain-inf
• https://stackoverflow.com/a/65066781/3986677

[znacer]

Thanks for your quick return and explanation.
I did not saw that this issue was already explained. It is a duplicated. closing the issue.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.