Improvement in release perf test

[heemin32]

Description

Simply the process to run performance test for release

Previous behavior

0. Update cluster setting with "knn.algo_param.index_thread_qty" : 4
1. Download hdf5 data
2. Modify test template with correct value of endpoint, port, path to json files, and path to hdf5 files manually.
3. Run knn-per-tool.py
4. Repeat step2 and step3 for 12 test cases

After change

1. Download hdf5 data in k-NN/benchmakrs/perf-tool/dataset/
2. Run run_all_tests.sh --endpoint <your_endpoint>

Issues Resolved

N/A

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[navneet1v]

can we add some sleep time between running of all the tests to make CPU come down and cluster to become stable.

[heemin32]

can we add some sleep time between running of all the tests to make CPU come down and cluster to become stable.

Do you think 1 min will be enough for that purpose?

[navneet1v]

yeah 1 min is fine.

[codecov]

Codecov Report

Merging #1243 (1fa7a8f) into main (34d697f) will not change coverage.
Report is 1 commits behind head on main.
The diff coverage is n/a.

@@            Coverage Diff            @@
##               main    #1243   +/-   ##
=========================================
  Coverage     85.07%   85.07%           
  Complexity     1210     1210           
=========================================
  Files           160      160           
  Lines          4932     4932           
  Branches        449      449           
=========================================
  Hits           4196     4196           
  Misses          537      537           
  Partials        199      199           

[navneet1v]

@heemin32 do you know why this [k-NN Rolling-Upgrade BWC Tests (17, ubuntu-latest, 2.10.0-SNAPSHOT, 3.0.0-SNAPSHOT)](https://github.com/opensearch-project/k-NN/actions/runs/6489712314/job/17624422073?pr=1243#logs) is coming even when we have removed to 2.10.0 from 2.10.0-SNAPSHOT? here: 34d697f

[heemin32]

@heemin32 do you know why this [k-NN Rolling-Upgrade BWC Tests (17, ubuntu-latest, 2.10.0-SNAPSHOT, 3.0.0-SNAPSHOT)](https://github.com/opensearch-project/k-NN/actions/runs/6489712314/job/17624422073?pr=1243#logs) is coming even when we have removed to 2.10.0 from 2.10.0-SNAPSHOT? here: 34d697f

#1245

[navneet1v]

@heemin32 do you know why this [k-NN Rolling-Upgrade BWC Tests (17, ubuntu-latest, 2.10.0-SNAPSHOT, 3.0.0-SNAPSHOT)](https://github.com/opensearch-project/k-NN/actions/runs/6489712314/job/17624422073?pr=1243#logs) is coming even when we have removed to 2.10.0 from 2.10.0-SNAPSHOT? here: 34d697f

#1245

thanks for raising the PR

[martin-gaievski]

curious if with every next run time changes, as this all is on same cluster, I think some query stuff may be cached after multiple runs?
I think we may extend it in future by allowing different endpoints/clusters for different tests.

[navneet1v]

curious if with every next run time changes, as this all is on same cluster, I think some query stuff may be cached after multiple runs?
I think we may extend it in future by allowing different endpoints/clusters for different tests.

@martin-gaievski queries won’t be cached because we delete the index at the start of every run.
the only thing that may cause issue is RSS. But generally we run with 4xlarges so that shouldn’t not be a problem too

[heemin32]

curious if with every next run time changes, as this all is on same cluster, I think some query stuff may be cached after multiple runs? I think we may extend it in future by allowing different endpoints/clusters for different tests.

This is what we have been doing for release benchmark test as long as I know. We are not using different cluster for each test run.