[RFC] Version Support Policy

[travisbenedict]

Is your feature request related to a problem? Please describe.

Indefinitely supporting every version of OpenSearch can be a maintenance burden. Certain updates may be required to the source code as well as the workloads in order to support new versions of OpenSearch. Maintaining support for older versions of OpenSearch adds additional complexity to the codebase and increases the likelihood of introducing bugs.

The OpenSearch Benchmark Workloads repository uses different branches to separate workloads for different versions. As future versions of OpenSearch are released these version numbers will begin to present confusion for users.

OpenSearch Benchmark currently supports executing tests against OpenSearch 1, 2 and 3 as well as Elasticsearch 6 and 7, though Elasticsearch 6 is not supported as a data store.

Describe the solution you'd like

A concrete policy defining what versions of OpenSearch and Elasticsearch OpenSearch Benchmark will support executing tests against.

Some possible policies include:

1. Support executing tests against at least the two most recent released major versions of OpenSearch.
2. Support executing tests against at least the two most recent released major versions of OpenSearch and continue to support Elasticsearch 7. Support for Elasticsearch 7 could be reevaluated as OpenSearch and Elasticsearch continue to diverge
3. Support executing tests against the five most recent versions of OpenSearch. This would mean we would continue to support OpenSearch 1, 2, 3 and Elasticsearch 6, 7 then drop Elasticsearch 6 support when support for OpenSearch 4 is added

Feel free to suggest any other policies that would fit your use case

[dblock]

It feels like benchmarks should follow the OpenSearch EOL policy, which currently isn't quite articulated beyond this FAQ. cc: @elfisher

[bbarani]

OpenSearch EOL policy is documented here.