…ildSrc (opensearch-project#8141)

* Bump com.networknt:json-schema-validator in /buildSrc

Bumps [com.networknt:json-schema-validator](https://github.com/networknt/json-schema-validator) from 1.0.83 to 1.0.84.
- [Release notes](https://github.com/networknt/json-schema-validator/releases)
- [Changelog](https://github.com/networknt/json-schema-validator/blob/master/CHANGELOG.md)
- [Commits](networknt/json-schema-validator@1.0.83...1.0.84)

---
updated-dependencies:
- dependency-name: com.networknt:json-schema-validator
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

opensearch-project#8035)

This commit refactors the Stream IO classes from the server to core
library to support cloud native and serverless extensions.

The following classes are refactored:

* StreamInput
* StreamOutput
* Writeable
* NamedWriteable
* NamedWriteableRegistry
* Index
* ShardId
* BytesReference
* ByteArray
* BigArray
* SecureString
* Text
* ParsingException
* RestStatus

The namespace is left unchanged but will be refactored in a follow up
commit to avoid split package in order to support jigsaw modularity.

Signed-off-by: Nicholas Walter Knize <[email protected]>

…y-azure-classic (opensearch-project#8140)

* Bump commons-io:commons-io in /plugins/discovery-azure-classic

Bumps commons-io:commons-io from 2.12.0 to 2.13.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Added changelog entry

Signed-off-by: Kartik Ganesh <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Kartik Ganesh <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>
Co-authored-by: Kartik Ganesh <[email protected]>

Multiple changes were incorrectly placed in the "Unreleased 3.0" section
on main and then backported. This brings the changelog on main in line
with 2.x for changes that have been backported.

Signed-off-by: Andrew Ross <[email protected]>

This fixes some minor differences with the 2.x changelog

Signed-off-by: Kartik Ganesh <[email protected]>

…earch-project#7919)

* Updates Node start to now pass localNode info so that it is available directly to a plugin instead of hacky implementation

Signed-off-by: Darshit Chanpura <[email protected]>

* Adds this PR to changelog

Signed-off-by: Darshit Chanpura <[email protected]>

* Makes onNodeStarted call plugin agnostic

Signed-off-by: Darshit Chanpura <[email protected]>

* Marking onNodeStarted() as deprecated

Signed-off-by: Darshit Chanpura <[email protected]>

* Adds a test to verify the behaviour of new signature of onNodeStarted

Signed-off-by: Darshit Chanpura <[email protected]>

* Updates test to also check for nodeId

Signed-off-by: Darshit Chanpura <[email protected]>

* Adds instructions to use the overloaded implementation of onNodeStarted and adds a test to check that current plugins are not affect with this implementation

Signed-off-by: Darshit Chanpura <[email protected]>

---------

Signed-off-by: Darshit Chanpura <[email protected]>
Signed-off-by: Darshit Chanpura <[email protected]>

…pensearch-project#8096)

Signed-off-by: Daniel Widdis <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>

…oject#6969)

* add ExtensionActionUtill class

Signed-off-by: Kuanysh <[email protected]>

* add ExtensionActionUtilTest class

Signed-off-by: Kuanysh <[email protected]>

* refactoring

Signed-off-by: Kuanysh <[email protected]>

* add method convertParamsToBytes

Signed-off-by: Kuanysh <[email protected]>

* code refactoring

Signed-off-by: Kuanysh <[email protected]>

* add ExtensionActionUtill class

Signed-off-by: Kuanysh <[email protected]>

* add ExtensionActionUtilTest class

Signed-off-by: Kuanysh <[email protected]>

* refactoring

Signed-off-by: Kuanysh <[email protected]>

* add method convertParamsToBytes

Signed-off-by: Kuanysh <[email protected]>

* code refactoring

Signed-off-by: Kuanysh <[email protected]>

* refactoring

Signed-off-by: Kuanysh <[email protected]>

* added new name to tests

Signed-off-by: Kuanysh <[email protected]>

* code refactoring

Signed-off-by: Kuanysh <[email protected]>

* test code refactoring

Signed-off-by: Kuanysh <[email protected]>

* test class renaming

Signed-off-by: Kuanysh <[email protected]>

---------

Signed-off-by: Kuanysh <[email protected]>
Signed-off-by: Kuanysh <[email protected]>

…oject#7456)

* add equels for ClusterSettings

Signed-off-by: Kuanysh <[email protected]>

* added junit

Signed-off-by: Kuanysh <[email protected]>

* code refactoring

Signed-off-by: Kuanysh <[email protected]>

* added changes to handleAddSettingsUpdateConsumer

Signed-off-by: Kuanysh <[email protected]>

* code refactoring

Signed-off-by: Kuanysh Aimurzinov <[email protected]>

* code refactoring

Signed-off-by: Kuanysh Aimurzinov <[email protected]>

* changed main method

Signed-off-by: Kuanysh <[email protected]>

---------

Signed-off-by: Kuanysh <[email protected]>
Signed-off-by: Kuanysh <[email protected]>
Signed-off-by: Kuanysh Aimurzinov <[email protected]>

Signed-off-by: Andrew Ross <[email protected]>

… it (opensearch-project#8029)

* Implemented REST API for initializing extension

Signed-off-by: Owais Kazi <[email protected]>

* Cleanup extensions.yml design

Signed-off-by: Owais Kazi <[email protected]>

* Added tests for RestInitializeExtensionAction

Signed-off-by: Owais Kazi <[email protected]>

* Pulled extensions REST request in extensions directory

Signed-off-by: Owais Kazi <[email protected]>

* Removed forbidden APIs from rest action and modified tests

Signed-off-by: Owais Kazi <[email protected]>

* Added entry in changelog

Signed-off-by: Owais Kazi <[email protected]>

* Added test for parse

Signed-off-by: Owais Kazi <[email protected]>

* Addressed PR comments

Signed-off-by: Owais Kazi <[email protected]>

* Addressed PR comments

Signed-off-by: Owais Kazi <[email protected]>

* Spotless Fixed

Signed-off-by: Owais Kazi <[email protected]>

* Handled exceptions

Signed-off-by: Owais Kazi <[email protected]>

* Handled test failure

Signed-off-by: Owais Kazi <[email protected]>

---------

Signed-off-by: Owais Kazi <[email protected]>

…rch-project#7998)

Add compatibility check gradle task ./gradlew checkCompatibility

Signed-off-by: Sayali Gaikawad <[email protected]>

…rch-project#8063)

* fix thread issue

Signed-off-by: Stephen Crawford <[email protected]>

* fix thread issue

Signed-off-by: Stephen Crawford <[email protected]>

* Fix thresholds

Signed-off-by: Stephen Crawford <[email protected]>

* Swap to object based

Signed-off-by: Stephen Crawford <[email protected]>

* Spotless

Signed-off-by: Stephen Crawford <[email protected]>

* Swap to preserve nulls

Signed-off-by: Stephen Crawford <[email protected]>

* Spotless

Signed-off-by: Stephen Crawford <[email protected]>

* Resolve npe

Signed-off-by: Stephen Crawford <[email protected]>

* remove final declerations

Signed-off-by: Stephen Crawford <[email protected]>

* spotless

Signed-off-by: Stephen Crawford <[email protected]>

* add annotations

Signed-off-by: Stephen Crawford <[email protected]>

* push to rerun tests

Signed-off-by: Stephen Crawford <[email protected]>

* Fix idea

Signed-off-by: Stephen Crawford <[email protected]>

* Fix idea

Signed-off-by: Stephen Crawford <[email protected]>

---------

Signed-off-by: Stephen Crawford <[email protected]>

… search source (opensearch-project#8206)

* Return null for aggregations factory even if empty aggs is present in search source

Signed-off-by: Jay Deng <[email protected]>

* Update rest-api-spec/src/main/resources/rest-api-spec/test/search.aggregation/60_empty.yml

Co-authored-by: Andriy Redko <[email protected]>
Signed-off-by: Jay Deng <[email protected]>

---------

Signed-off-by: Jay Deng <[email protected]>
Co-authored-by: Andriy Redko <[email protected]>

…ch-project#8175)

Signed-off-by: Ashish Singh <[email protected]>

Signed-off-by: Andriy Redko <[email protected]>

…rategy change when configuring remote storage with cluster setting (opensearch-project#8186)

* Removing overriding of replication type setting in updateRemoteStoreSettings()

Signed-off-by: Rishikesh1159 <[email protected]>

* fixing failing test testRemoteStoreNoUserOverrideConflictingReplicationTypeIndexSettings() by modifying exception message.

Signed-off-by: Rishikesh1159 <[email protected]>

* Fix failing tests.

Signed-off-by: Rishikesh1159 <[email protected]>

* remove unnecessary system index integ test.

Signed-off-by: Rishikesh1159 <[email protected]>

* revert previous change of removing system index integ test.

Signed-off-by: Rishikesh1159 <[email protected]>

* Address comments on PR.

Signed-off-by: Rishikesh1159 <[email protected]>

* Address comments.

Signed-off-by: Rishikesh1159 <[email protected]>

* Address comments on PR.

Signed-off-by: Rishikesh1159 <[email protected]>

---------

Signed-off-by: Rishikesh1159 <[email protected]>
Signed-off-by: Rishikesh Pasham <[email protected]>

This change was backported to 2.x in opensearch-project#7945.

Signed-off-by: Andrew Ross <[email protected]>

…ne (opensearch-project#7350)

Signed-off-by: Subhobrata Dey <[email protected]>

…8228)

* Fixes duplicate extension unique id

Signed-off-by: Owais Kazi <[email protected]>

* Handled different extension unique id

Signed-off-by: Owais Kazi <[email protected]>

---------

Signed-off-by: Owais Kazi <[email protected]>

…mote store is used with system indices (opensearch-project#8235)

* Removing throwing of exception when remote store is used with system indices.

Signed-off-by: Rishikesh1159 <[email protected]>

* Add check of feature flag enabled.

Signed-off-by: Rishikesh1159 <[email protected]>

* Add unit test to improve code coverage.

Signed-off-by: Rishikesh1159 <[email protected]>

* Add warning log.

Signed-off-by: Rishikesh1159 <[email protected]>

---------

Signed-off-by: Rishikesh1159 <[email protected]>

…ocation (opensearch-project#8244)

Signed-off-by: Marc Handalian <[email protected]>

…points and skip tests for 1.x bwc versions (opensearch-project#8203)

* [Segment Replication] Verify segment replication stats in bwc test

Signed-off-by: Suraj Singh <[email protected]>

* Log cleanup

Signed-off-by: Suraj Singh <[email protected]>

* Spotless check

Signed-off-by: Suraj Singh <[email protected]>

* Add version check to skip test for 1.x bwc branches

Signed-off-by: Suraj Singh <[email protected]>

* Add version check to skip test for 1.x bwc branches for mixed clusters

Signed-off-by: Suraj Singh <[email protected]>

* Add version string in build to identify bwc version

Signed-off-by: Suraj Singh <[email protected]>

* Use correct bwc version string

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments from opensearch-project#7626

Signed-off-by: Suraj Singh <[email protected]>

---------

Signed-off-by: Suraj Singh <[email protected]>

…t/fixtures/hdfs-fixture (opensearch-project#8257)

* Bump org.apache.hadoop:hadoop-minicluster in /test/fixtures/hdfs-fixture

Bumps org.apache.hadoop:hadoop-minicluster from 3.3.5 to 3.3.6.

---
updated-dependencies:
- dependency-name: org.apache.hadoop:hadoop-minicluster
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…ildSrc (opensearch-project#8255)

* Bump com.networknt:json-schema-validator in /buildSrc

Bumps [com.networknt:json-schema-validator](https://github.com/networknt/json-schema-validator) from 1.0.84 to 1.0.85.
- [Release notes](https://github.com/networknt/json-schema-validator/releases)
- [Changelog](https://github.com/networknt/json-schema-validator/blob/master/CHANGELOG.md)
- [Commits](networknt/json-schema-validator@1.0.84...1.0.85)

---
updated-dependencies:
- dependency-name: com.networknt:json-schema-validator
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…n /plugins/repository-azure (opensearch-project#8256)

* Bump io.projectreactor.netty:reactor-netty-http

Bumps [io.projectreactor.netty:reactor-netty-http](https://github.com/reactor/reactor-netty) from 1.1.7 to 1.1.8.
- [Release notes](https://github.com/reactor/reactor-netty/releases)
- [Commits](reactor/reactor-netty@v1.1.7...v1.1.8)

---
updated-dependencies:
- dependency-name: io.projectreactor.netty:reactor-netty-http
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>

* Updating SHAs

Signed-off-by: dependabot[bot] <[email protected]>

* Update changelog

Signed-off-by: dependabot[bot] <[email protected]>

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

Bumps [org.jruby.joni:joni](https://github.com/jruby/joni) from 2.1.48 to 2.2.1.
- [Commits](jruby/joni@joni-2.1.48...joni-2.2.1)

---
updated-dependencies:
- dependency-name: org.jruby.joni:joni
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Signed-off-by: Andriy Redko <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Signed-off-by: bansvaru <[email protected]>

* [Upgrade] Lucene 9.7.0 release

Upgrades to the official 9.7.0 release of lucene

Signed-off-by: Nicholas Walter Knize <[email protected]>

* update changelog

Signed-off-by: Nicholas Walter Knize <[email protected]>

---------

Signed-off-by: Nicholas Walter Knize <[email protected]>

Signed-off-by: Sayali Gaikawad <[email protected]>

Signed-off-by: Kunal Kotwani <[email protected]>

…opensearch-project#9053)

This changes the version check for MediaType in PercolateQueryBuilder
for BWC.

Signed-off-by: Nicholas Walter Knize <[email protected]>

…ch-project#8767)

* [Segment Replication] Refactor remote replication source

Signed-off-by: Suraj Singh <[email protected]>

* Unit test updates

Signed-off-by: Suraj Singh <[email protected]>

* Self review

Signed-off-by: Suraj Singh <[email protected]>

* Self review

Signed-off-by: Suraj Singh <[email protected]>

* Segregate shard level tests for node to node and remote store segment replication

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing unit tests

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing UT

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing UT

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

* Fix more unit tests

Signed-off-by: Suraj Singh <[email protected]>

* Improve RemoteStoreReplicationSourceTests, remove unnecessary mocks and use actual failures for failure/exception use cases

Signed-off-by: Suraj Singh <[email protected]>

* Spotless check fix

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

* Ignore files already in store while computing segment file diff with primary

Signed-off-by: Suraj Singh <[email protected]>

* Spotless fix

Signed-off-by: Suraj Singh <[email protected]>

* Fix failing UT

Signed-off-by: Suraj Singh <[email protected]>

* Spotless fix

Signed-off-by: Suraj Singh <[email protected]>

* Move read/writes from IndexInput/Output to RemoteSegmentMetadata

Signed-off-by: Suraj Singh <[email protected]>

* Address review commnt

Signed-off-by: Suraj Singh <[email protected]>

* Update recovery flow to perform commits during recovery

Signed-off-by: Suraj Singh <[email protected]>

* Remove un-necessary char

Signed-off-by: Suraj Singh <[email protected]>

* Address review comments

Signed-off-by: Suraj Singh <[email protected]>

* Update comment nit-pick

Signed-off-by: Suraj Singh <[email protected]>

* Remove deletion logic causing read issues due to deleted segments_N

Signed-off-by: Suraj Singh <[email protected]>

* Spotless fix

Signed-off-by: Suraj Singh <[email protected]>

* Fix unit tests

Signed-off-by: Suraj Singh <[email protected]>

---------

Signed-off-by: Suraj Singh <[email protected]>

…nsearch-project#8737)

* Disallowing compression level for lz4 and best_compression codec
* setting up codec settings interface for validation

Signed-off-by: Sarthak Aggarwal <[email protected]>

Signed-off-by: Sarthak Aggarwal <[email protected]>

Added the featureFlagSettings method to OpenSearchSingleNodeTestCase

Signed-off-by: Stefano Maglione <[email protected]>

…rch-project#9073)

This commit refactors the following network and transport libraries to
the opensearch common and core libraries respectively:

* o.o.common.network.Cidrs -> :libs:opensearch-common
* o.o.common.network.InetAddresses -> :libs:opensearch-common
* o.o.common.network.NetworkAddress -> :libs:opensearch-common
* o.o.common.transport.NetworkExceptionHelper -> :libs:opensearch-common
* o.o.common.transport.PortsRange -> :libs:opensearch-common

* o.o.common.transport.TransportAddress -> :libs:opensearch-core
* o.o.common.transport.BoundTransportAddress -> :libs:opensearch-core
* o.o.transport.TransportMessage -> :libs:opensearch-core
* o.o.transport.TransportResponse -> :libs:opensearch-core

The purpose is to reduce the change surface area of the core APIs to
minimize impact to downstream consumers while moving toward establishing
a formal API for cloud native or serverless implementations.

Signed-off-by: Nicholas Walter Knize <[email protected]>

…st index deletion. (opensearch-project#8472)

Signed-off-by: Harish Bhakuni <[email protected]>

…replay (opensearch-project#8578)


Signed-off-by: Gaurav Bafna <[email protected]>

…search-project#9057)

---------

Signed-off-by: Ashish Singh <[email protected]>

…ckpoint validation (opensearch-project#8889)

* Fix test testDropPrimaryDuringReplication and clean up ReplicationCheckpoint validation.

This test is now occasionally failing with replicas having 0 documents. This occurs in a couple of ways:
1. After dropping the old primary the new primary is not publishing a checkpoint to replicas unless it indexes docs from translog after flipping to primary mode.
If there is nothing to index, it will not publish a checkpoint, but the other replica could have never sync'd with the original primary and be left out of date.
- This PR fixes this by force publishing a checkpoint after the new primary flips to primary mode.
2. The replica receives a checkpoint post failover and cancels its sync with the former primary that is still active, recognizing a primary term bump.
However this cancellation is async and immediately starting a new replication event could fail as its still replicating.
- This PR fixes this by attempting to process the latest received checkpoint on failure, if the shard is not failed and still behind.

This PR also introduces a few changes to ensure the accuracy of the ReplicationCheckpoint tracked on primary & replicas.
- Ensure the checkpoint stored in SegmentReplicationTarget is the checkpoint passed from the primary and not locally computed.  This ensures checks for primary term are accurate and not using a locally compued operationPrimaryTerm.
- Introduces a refresh listener for both primary & replica to update the ReplicationCheckpoint and store it in replicationTracker post refresh rather than redundantly computing when accessed.
- Removes unnecessary onCheckpointPublished method used to start replication timers manually.  This will happen automatically on primaries once its local cp is updated.

Signed-off-by: Marc Handalian <[email protected]>

* Handle NoSuchFileException when attempting to delete decref'd files.

To avoid divergent logic with remote store, we always incref/decref the segmentinfos.files(true) which includes the segments_n file.
Decref to 0 will attempt to delete the file from the store and its possible this _n file does not yet exist. This change will ignore if we get a noSuchFile while attempting to delete.

Signed-off-by: Marc Handalian <[email protected]>

* Add more unit tests.

Signed-off-by: Marc Handalian <[email protected]>

* Clean up IndexShardTests.testCheckpointReffreshListenerWithNull

Signed-off-by: Marc Handalian <[email protected]>

* Remove unnecessary catch for NoSuchFileException.

Signed-off-by: Marc Handalian <[email protected]>

* Add another test for non segrep.

Signed-off-by: Marc Handalian <[email protected]>

* PR Feedback.

Signed-off-by: Marc Handalian <[email protected]>

* re-compute replication checkpoint on primary promotion.

Signed-off-by: Marc Handalian <[email protected]>

---------

Signed-off-by: Marc Handalian <[email protected]>

…e request during SearchContext creation (opensearch-project#9059)

* Change to determine if concurrent segment search should be used by the request during SearchContext creation. It caches the e evaluated output for all future invocation. It also provide executor to IndexSearcher based on this evaluation

Signed-off-by: Sorabh Hamirwasia <[email protected]>

* Add test case to enable/disable concurrent search cluster setting and verify context object state

Signed-off-by: Sorabh Hamirwasia <[email protected]>

---------

Signed-off-by: Sorabh Hamirwasia <[email protected]>

…ect#9098)

The [commit to comment out `forceFailure()`][1] was done about 8 years
ago so I don't think we'll be bringing it back. Also this makes the code
base a slightly friendlier place :)

[1]: opensearch-project@e5a699f

Signed-off-by: Andrew Ross <[email protected]>

Signed-off-by: Suraj Singh <[email protected]>

…ng search (opensearch-project#9047)

Signed-off-by: Neetika Singhal <[email protected]>

…opensearch-project#9120)

This commit refactors the CollectionUtils, and Iterators utility classes
to the core and common libraries, respectively. Several other base
Strings utility methods are also refactored to the core Strings utility
library. This is done to incrementally move the remaining Strings
methods in server to the core String utility.

Signed-off-by: Nicholas Walter Knize <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>

Signed-off-by: Poojita Raj <[email protected]>