Adding support for SHAP explainability to tree model types

[kikisq7]

SHAP Explainability provides an explanation for the output of a machine learning model. It provides insights as to how each feature contributes to the model's predictions, aiding in gaining an understanding of what happens in the "black box."

I’m working on functionality to add support for including SHAP explainability when converting a tree model to ONNX and I’d like to contribute this work back to this community. TreeSHAP, an algorithm used for analyzing tree models, provides an "explainability" vector that indicates how much each feature contributed to the model's output. TreeSHAP is currently available as a tree model execution environment for Python, implemented in C++. I will be referencing the existing TreeSHAP algorithm and adding ONNX operations during the conversion to support SHAP explainability for tree models. During model execution, SHAP adjusts the feature's value in the "explainability" vector, based on the tree's decision at each node.

When this option is enabled the output tensor will be adjusted to include the vector of SHAP values (with length that is the number of features) following the converted ONNX model at execution time. There will also be an option to enable/disable this feature at conversion time and this feature will be disabled by default.

I really appreciate these tools and I'm excited to contribute to this community!

[xadupre]

So, you would like to create an onnx model with two trees, one for the model, one for SHAP values? Is it specific to one library (xgboost, lightgbm or scikit-learn)?

[danwhale]

Hi, I'm also interested in supporting of SHAP tree for gradient boosting models like xgboost/lightgbm. It will be perfect if we have an ability to provide model inference results and SHAP values for each prediction inside a single ONNX graph.

[ntrost-targ]

This feature would really be a thing. To have models in production it is often required to provide some kind of explainability for all the decisions. If i want to serve a model via onnx i would have to provide the SHAP Calculation elsewhere and have the model in some second place. If we can add TreeSHAP support to ONNX this would make things much easier.

TreeSHAP is not a second Tree that is being evaluated, but it is a computation that evaluates the original Tree differently. It should be possible to write it for general tree models independent of the library