Wrong error trace when LightGBM model file is not found

[jliebana]

Hello,

I'm reporting the following error because it was not trivial to see why this was happening.

I had the following structure:

model_repository/
└── nonhist
    ├── 1
    │   └── nonhist.txt
    └── config.pbtxt

The content of the config was:

backend: "fil"
max_batch_size: 32768
input [                                 
 {  
    name: "input__0"
    data_type: TYPE_FP32
    dims: [ 119 ]                    
 } 
]
output [
 {
    name: "output__0"
    data_type: TYPE_FP32
    dims: [ 2 ]
  }
]
instance_group [{ kind: KIND_CPU }]
parameters [
  {
    key: "model_type"
    value: { string_value: "lightgbm" }
  },
  {
    key: "predict_proba"
    value: { string_value: "true" }
  },
  {
    key: "output_class"
    value: { string_value: "true" }
  },
  {
    key: "threshold"
    value: { string_value: "0.5" }
  },
  {
    key: "storage_type"
    value: { string_value: "AUTO" }
  }
]

And when I tried to run Triton server I got this error:

I0117 10:10:20.044072 1 model_lifecycle.cc:459] loading: nonhist:1
I0117 10:10:20.081066 1 initialize.hpp:43] TRITONBACKEND_Initialize: fil
I0117 10:10:20.081089 1 backend.hpp:47] Triton TRITONBACKEND API version: 1.10
I0117 10:10:20.081095 1 backend.hpp:52] 'fil' TRITONBACKEND API version: 1.10
I0117 10:10:20.081197 1 model_initialize.hpp:37] TRITONBACKEND_ModelInitialize: nonhist (version 1)
I0117 10:10:20.082173 1 instance_initialize.hpp:46] TRITONBACKEND_ModelInstanceInitialize: nonhist_0 (CPU device 0)
I0117 10:10:20.082341 1 model_finalize.hpp:36] TRITONBACKEND_ModelFinalize: delete model state
E0117 10:10:20.082404 1 model_lifecycle.cc:597] failed to load 'nonhist' version 1: Unknown: [10:10:20] /rapids_triton/build/_deps/treelite-src/src/frontend/lightgbm.cc:279: Check failed: it != global_dict.end(): Ill-formed LightGBM model file: need max_feature_idx
I0117 10:10:20.082482 1 server.cc:563] 

By the error trace, it would seem that is reading the text file of the tree and it's not able to find the max_feature_idx property, however the file did indeed contain that value:

$ head model_repository/nonhist/1/nonhist.txt 

tree
version=v3
num_class=1
num_tree_per_iteration=1
label_index=0
max_feature_idx=118

After trying several things and different model files, I did a simple test of changing the filename to model.txt and with that change, everything started to work. So the issue seems to be that Triton was not finding the model file, but it was throwing an error about the content of the file, which made it harder to debug.

Proposed solutions:

• Improve error message specifying that the file was not found
• Improve documentation to indicate that the default model filename must be "model.txt"

[wphicks]

Thanks! Yes, the error message can definitely be improved here. We'll make it happen.