[Metrics store] Add support for custom templates

[inqueue]

This change adds the ability to customize indices in the metrics store. The goal is to support ILM policies and eventually shift to using data streams.

• Support the addition of component templates having the @custom suffix.
• Add documentation.

[gareth-ellis]

The rally-annotations is a single alias of a single backing index. (ie rally-annotations -> reindexed-v7-rally-annotations ) I think we can remove the comment about the subset, as the documents itself just refer to chart names, rather than metrics themselves

[gbanasiak]

TypeError: put_index_template() got an unexpected keyword argument 'ignore_missing_component_templates'

We would need to modify assumption on metric store ES version to support this approach. We're still maintaining compatibility with 6.8 for target cluster which I think is something we could drop, but 7.17.x must stay for now. Unfortunately ignore_missing_component_templates was added in 8.7. Do we want to bifurcate and have different requirements for metric store and target cluster?

[inqueue]

TypeError: put_index_template() got an unexpected keyword argument 'ignore_missing_component_templates'

We would need to modify assumption on metric store ES version to support this approach. We're still maintaining compatibility with 6.8 for target cluster which I think is something we could drop, but 7.17.x must stay for now. Unfortunately ignore_missing_component_templates was added in 8.7. Do we want to bifurcate and have different requirements for metric store and target cluster?

Thanks, Grzegorz. I hadn't realized ignore_missing_component_templates was so new. The parameter is not even supported in elasticsearch-py until 8.14 via elastic/elasticsearch-py#2525 (Rally uses 8.6.1 currently).

Rally would need to load an empty *@custom component template to support this, if one did was not already in the metrics cluster:

{
  "template": {}
}

The ultimate goal was to slowly move toward using data streams, or at least provide the option for it, to allow for better management of metrics retention, and pushing metrics to serverless. We are not there yet on any of these fronts, and therefore, closing this PR.