Add Enterprise Search API endpoints for 1 Click ELSER ML Model Deployment

[markjhoy]

Summary

Adds Enterprise Search internal API endpoints for deploying and monitoring the deployment status of an ELSER ML model (and possibly other models in the future) via the 1 click deployment process. This is to not allow a direct call from the Kibana front end to the underlying Elasticsearch ML endpoints.

Closes https://github.com/elastic/enterprise-search-team/issues/4295 and https://github.com/elastic/enterprise-search-team/issues/4397

Checklist

• Unit or functional tests were updated or added to match the most common scenarios
• This was checked for cross-browser compatibility

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[demjened]

@markjhoy Great first draft, thanks!

There are a couple of issues we need to sort out before ELSER download process can work properly:

• In get_ml_model_deployment_status.ts the trainedModelsProvider.getTrainedModels() call returns a 404 error if the model does not exist, so we need to wrap it in a try/catch block rather than checking modelDetailsResponse. We should check for error.statusCode !== 404 within the catch block; this is what that API returns upon a missing model.
• While the model is downloading (but not deployed), the modelDetailsResponse.trained_model_configs property does not exist. The correct check is !modelDetailsResponse.trained_model_configs || !modelDetailsResponse.trained_model_configs[0].fully_defined.
• We'll need to make sure the types in the ML JS client have been updated before this PR can be merged: MlTrainedModelConfig.fully_defined, MlInclude.definition_status.
• Are you sure the new deployment state is called is_fully_downloaded, not fully_downloaded? The latter is more in line with the other states, e.g. fully_allocated.

[markjhoy]

• In get_ml_model_deployment_status.ts the trainedModelsProvider.getTrainedModels() call returns a 404 error if the model does not exist, so we need to wrap it in a try/catch block rather than checking modelDetailsResponse. We should check for error.statusCode !== 404 within the catch block; this is what that API returns upon a missing model.

Ah - gotcha - I wasn't 100% certain.... updated to check this.

• While the model is downloading (but not deployed), the modelDetailsResponse.trained_model_configs property does not exist. The correct check is !modelDetailsResponse.trained_model_configs || !modelDetailsResponse.trained_model_configs[0].fully_defined.

Good catch. Updated.

• We'll need to make sure the types in the ML JS client have been updated before this PR can be merged: MlTrainedModelConfig.fully_defined, MlInclude.definition_status.

Yep - hopefully that'll come soon...

• Are you sure the new deployment state is called is_fully_downloaded, not fully_downloaded? The latter is more in line with the other states, e.g. fully_allocated.

I think that was some of the initial wording that was there, but subsequently changed... I've cleaned up the code better to reflect the current state of things. 👍

[TattdCodeMonkey]

Overall LGTM, the TODOs and missing types make a little uneasy given where we are in release cycle.

[TattdCodeMonkey]

@demjened I think this is ok since we don't fetch model status if ml is disabled, but we should ensure thats the case.

[markjhoy]

the TODOs and missing types make a little uneasy given where we are in release cycle

100% agree... hopefully the missing types will be there before BC2 and we can alleviate these....

[jgowdyelastic]

the TODOs and missing types make a little uneasy given where we are in release cycle

100% agree... hopefully the missing types will be there before BC2 and we can alleviate these....

Just to confirm, there were two sets of missing types affecting this PR. The ones in the elasticsearch-js client, and the ones for ML's TrainedModelsProvider.
The TrainedModelsProvider types are now up to date and have been used in this PR.

The elasticsearch-js client types I do not believe will make this version as historically a client update comes with a lot of types churn throughout kibana.
These missing types are minimal, one missing param in the request and one property in the response of the get trained models api.

[markjhoy]

Just to confirm, there were two sets of missing types affecting this PR. The ones in the elasticsearch-js client, and the ones for ML's TrainedModelsProvider. The TrainedModelsProvider types are now up to date and have been used in this PR.

The elasticsearch-js client types I do not believe will make this version as historically a client update comes with a lot of types churn throughout kibana. These missing types are minimal, one missing param in the request and one property in the response of the get trained models api.

Thanks @jgowdyelastic -- we also have a follow on task to check and update this in the future when things are available: https://github.com/elastic/enterprise-search-team/issues/4432

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 155213 locally

[sphilipse]

Removing version/backport labels because it's too late to add backports now.

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 155213 locally

[kibanamachine]

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add the label auto-backport or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 155213 locally