2.x Distribution bundle bwc tests

[vibrantvarun]

Description

There is a framework for BWC tests on distribution level as well which will set up the test cluster with the bundle distribution including all plugins engineering effectiveness team builds.

It will be beneficial during the release cycle since we will be testing the exact distribution that we would ship to the customers.
This PR consist configuration to run bwc tests on distribution bundle. The PR is on Branch 2.x

The bwc distribution will fail on neural search for opensearch versions <2.11 because of the bug from the ml-commons.
opensearch-project/ml-commons#1838

rg.opensearch.neuralsearch.bwc.SemanticSearchIT > testSemanticSearch_E2EFlow FAILED
    org.opensearch.client.ResponseException: method [POST], host [http://127.0.0.1:39449], URI [/neuralsearch-bwc-testsemanticsearch_e2eflow/_search?size=1], status line [HTTP/1.1 500 Internal Server Error]
    {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"unexpected byte [0x48]"}],"type":"illegal_state_exception","reason":"unexpected byte [0x48]"},"status":500}
        at __randomizedtesting.SeedInfo.seed([9C3D08353B6D4C7A:2A2FA7AD9C7ED3BD]:0)
        at app//org.opensearch.client.RestClient.convertResponse(RestClient.java:376)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:346)
        at app//org.opensearch.client.RestClient.performRequest(RestClient.java:321)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:417)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:378)
        at app//org.opensearch.neuralsearch.BaseNeuralSearchIT.search(BaseNeuralSearchIT.java:359)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.validateTestIndexOnUpgrade(SemanticSearchIT.java:79)
        at app//org.opensearch.neuralsearch.bwc.SemanticSearchIT.testSemanticSearch_E2EFlow(SemanticSearchIT.java:46)

Issues Resolved

564

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Commits are signed as per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (5d9a8d9) 85.24% compared to head (47e8a18) 85.24%.

Additional details and impacted files

@@            Coverage Diff            @@
##                2.x     #574   +/-   ##
=========================================
  Coverage     85.24%   85.24%           
  Complexity      543      543           
=========================================
  Files            40       40           
  Lines          1586     1586           
  Branches        249      249           
=========================================
  Hits           1352     1352           
  Misses          117      117           
  Partials        117      117           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jmazanec15]

Instead of checking in sample credentials, can we just pull them like we do for our security enabled tests? I prefer to avoid checking in credentials - its hard to maintain. https://github.com/opensearch-project/neural-search/blob/main/build.gradle#L95-L103

[zelinh]

This actually looks good to me. Have you tried to run locally with the bundle BWC tests? @vibrantvarun

[vibrantvarun]

@zelinh yes I did

[martin-gaievski]

@zelinh yes I did

@vibrantvarun can you please post results of local run here?

[vibrantvarun]

@zelinh @martin-gaievski

./gradlew bwcTestSuite -Dtests.security.manager=false -PcustomDistributionDownloadType=bundle

> Configure project :
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/esnode.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/esnode-key.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/root-ca.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/kirk.pem
Download https://raw.githubusercontent.com/opensearch-project/security/main/bwc-test/src/test/resources/security/kirk-key.pem
=======================================
OpenSearch Build Hamster says Hello!
  Gradle Version        : 8.4
  OS Info               : Linux 5.10.205-172.807.amzn2int.x86_64 (amd64)
  JDK Version           : 21 (Amazon Corretto JDK)
  JAVA_HOME             : /local/home/varunudr/.sdkman/candidates/java/21.0.1-amzn
  Random Testing Seed   : CF6DFE7F6339FBDD
  In FIPS 140 mode      : false
=======================================

> Task :generatePomFileForNebulaPublication
Maven publication 'nebula' pom metadata warnings (silence with 'suppressPomMetadataWarningsFor(variant)'):
  - Variant testFixturesApiElements:
      - Declares capability org.opensearch:neural-search-test-fixtures:2.12.0.0-SNAPSHOT which cannot be mapped to Maven
  - Variant testFixturesRuntimeElements:
      - Declares capability org.opensearch:neural-search-test-fixtures:2.12.0.0-SNAPSHOT which cannot be mapped to Maven
These issues indicate information that is lost in the published 'pom' metadata file, which may be an issue if the published library is consumed by an old Gradle version or Apache Maven.
The 'module' metadata file, which is used by Gradle 6+ is not affected.

> Task :qa:restart-upgrade:testAgainstOldCluster
Feb 01, 2024 12:31:32 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:restart-upgrade:testAgainstNewCluster
Feb 01, 2024 12:32:44 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testAgainstOldCluster
Feb 01, 2024 12:33:41 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testAgainstOneThirdUpgradedCluster
Feb 01, 2024 12:34:37 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testAgainstTwoThirdsUpgradedCluster
Feb 01, 2024 12:35:06 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release

> Task :qa:rolling-upgrade:testRollingUpgrade
Feb 01, 2024 12:35:35 PM sun.util.locale.provider.LocaleProviderAdapter <clinit>
WARNING: COMPAT locale provider will be removed in a future release
<============-> 92% EXECUTING [9m 16s]

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

For more on this, please refer to https://docs.gradle.org/8.4/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.

BUILD SUCCESSFUL in 9m 59s
38 actionable tasks: 10 executed, 28 up-to-date

[vibrantvarun]

Closing this as I have raised the PR on main #579. Then backport it on 2.x to maintain consistency within branches.