Merge branch 'main' of https://github.com/opensearch-project/OpenSearch…

… into pittestsfix

CHANGELOG.md

@@ -23,6 +23,10 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Add missing Javadoc tag descriptions in unit tests ([#4629](https://github.com/opensearch-project/OpenSearch/pull/4629))
 ### Dependencies
 - Bumps `log4j-core` from 2.18.0 to 2.19.0
+- Bumps `reactor-netty-http` from 1.0.18 to 1.0.23
+- Bumps `jettison` from 1.5.0 to 1.5.1
+- Bumps `azure-storage-common` from 12.18.0 to 12.18.1
+- Bumps `forbiddenapis` from 3.3 to 3.4
 
 
 ### Dependencies
@@ -36,6 +40,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Bumps `com.diffplug.spotless` from 6.10.0 to 6.11.0 ([#4547](https://github.com/opensearch-project/OpenSearch/pull/4547))
 - Bumps `reactor-core` from 3.4.18 to 3.4.23 ([#4548](https://github.com/opensearch-project/OpenSearch/pull/4548))
 - Bumps `jempbox` from 1.8.16 to 1.8.17 ([#4550](https://github.com/opensearch-project/OpenSearch/pull/4550))
+- Bumps `hadoop-hdfs` from 3.3.3 to 3.3.4 ([#4644](https://github.com/opensearch-project/OpenSearch/pull/4644))
 
 ### Changed
 - Dependency updates (httpcore, mockito, slf4j, httpasyncclient, commons-codec) ([#4308](https://github.com/opensearch-project/OpenSearch/pull/4308))
@@ -50,11 +55,16 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - Unmute test RelocationIT.testRelocationWhileIndexingRandom ([#4580](https://github.com/opensearch-project/OpenSearch/pull/4580))
 - Add DecommissionService and helper to execute awareness attribute decommissioning ([#4084](https://github.com/opensearch-project/OpenSearch/pull/4084))
 - Further simplification of the ZIP publication implementation ([#4360](https://github.com/opensearch-project/OpenSearch/pull/4360))
+- Relax visibility of the HTTP_CHANNEL_KEY and HTTP_SERVER_CHANNEL_KEY to make it possible for the plugins to access associated Netty4HttpChannel / Netty4HttpServerChannel instance ([#4638](https://github.com/opensearch-project/OpenSearch/pull/4638))
+- Load the deprecated master role in a dedicated method instead of in setAdditionalRoles() ([#4582](https://github.com/opensearch-project/OpenSearch/pull/4582))
+- Add APIs (GET/PUT) to decommission awareness attribute ([#4261](https://github.com/opensearch-project/OpenSearch/pull/4261))
+- Update to Apache Lucene 9.4.0 ([#4661](https://github.com/opensearch-project/OpenSearch/pull/4661))
 
 ### Deprecated
 
 ### Removed
 - Remove deprecated code to add node name into log pattern of log4j property file ([#4568](https://github.com/opensearch-project/OpenSearch/pull/4568))
+- Unused object and import within TransportClusterAllocationExplainAction ([#4639](https://github.com/opensearch-project/OpenSearch/pull/4639))
 
 ### Fixed
 - `opensearch-service.bat start` and `opensearch-service.bat manager` failing to run ([#4289](https://github.com/opensearch-project/OpenSearch/pull/4289))
@@ -74,14 +84,23 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - [Segment Replication] Update flaky testOnNewCheckpointFromNewPrimaryCancelOngoingReplication unit test ([#4414](https://github.com/opensearch-project/OpenSearch/pull/4414))
 - Fixed the `_cat/shards/10_basic.yml` test cases fix.
 - [Segment Replication] Fix timeout issue by calculating time needed to process getSegmentFiles ([#4426](https://github.com/opensearch-project/OpenSearch/pull/4426))
-- [Bug]: gradle check failing with java heap OutOfMemoryError (([#4328](https://github.com/opensearch-project/OpenSearch/
+- [Bug]: gradle check failing with java heap OutOfMemoryError ([#4328](https://github.com/opensearch-project/OpenSearch/))
 - `opensearch.bat` fails to execute when install path includes spaces ([#4362](https://github.com/opensearch-project/OpenSearch/pull/4362))
 - Getting security exception due to access denied 'java.lang.RuntimePermission' 'accessDeclaredMembers' when trying to get snapshot with S3 IRSA ([#4469](https://github.com/opensearch-project/OpenSearch/pull/4469))
 - Fixed flaky test `ResourceAwareTasksTests.testTaskIdPersistsInThreadContext` ([#4484](https://github.com/opensearch-project/OpenSearch/pull/4484))
 - Fixed the ignore_malformed setting to also ignore objects ([#4494](https://github.com/opensearch-project/OpenSearch/pull/4494))
 - [Segment Replication] Ignore lock file when testing cleanupAndPreserveLatestCommitPoint ([#4544](https://github.com/opensearch-project/OpenSearch/pull/4544))
 - Updated jackson to 2.13.4 and snakeyml to 1.32 ([#4556](https://github.com/opensearch-project/OpenSearch/pull/4556))
 - Fixing PIT flaky tests ([4632](https://github.com/opensearch-project/OpenSearch/pull/4632))
+- Fixed day of year defaulting for round up parser ([#4627](https://github.com/opensearch-project/OpenSearch/pull/4627))
+- Fixed the SnapshotsInProgress error during index deletion ([#4570](https://github.com/opensearch-project/OpenSearch/pull/4570))
+- [Segment Replication] Adding check to make sure checkpoint is not processed when a shard's shard routing is primary ([#4630](https://github.com/opensearch-project/OpenSearch/pull/4630))
+- [Bug]: Fixed invalid location of JDK dependency for arm64 architecture([#4613](https://github.com/opensearch-project/OpenSearch/pull/4613))
+- [Bug]: Alias filter lost after rollover ([#4499](https://github.com/opensearch-project/OpenSearch/pull/4499))
+- Fixed the SnapshotsInProgress error during index deletion ([#4570](https://github.com/opensearch-project/OpenSearch/pull/4570))
+- [Segment Replication] Adding check to make sure checkpoint is not processed when a shard's shard routing is primary ([#4630](https://github.com/opensearch-project/OpenSearch/pull/4630))
+- [Bug]: Fixed invalid location of JDK dependency for arm64 architecture([#4613](https://github.com/opensearch-project/OpenSearch/pull/4613))
+- [Bug]: Alias filter lost after rollover ([#4499](https://github.com/opensearch-project/OpenSearch/pull/4499))
 
 ### Security
 - CVE-2022-25857 org.yaml:snakeyaml DOS vulnerability ([#4341](https://github.com/opensearch-project/OpenSearch/pull/4341))

buildSrc/src/main/java/org/opensearch/gradle/Jdk.java

@@ -128,7 +128,7 @@ public void setArchitecture(final String architecture) {
                 "unknown architecture [" + jdkArchitecture + "] for jdk [" + name + "], must be one of " + ALLOWED_ARCHITECTURES
             );
         }
-        this.architecture.set(architecture);
+        this.architecture.set(jdkArchitecture);
     }
 
     public String getBaseVersion() {

buildSrc/src/testKit/thirdPartyAudit/build.gradle

@@ -40,7 +40,7 @@ repositories {
 }
 
 dependencies {
-  forbiddenApisCliJar 'de.thetaphi:forbiddenapis:3.3'
+  forbiddenApisCliJar 'de.thetaphi:forbiddenapis:3.4'
   jdkJarHell 'org.opensearch:opensearch-core:current'
   compileOnly "org.${project.properties.compileOnlyGroup}:${project.properties.compileOnlyVersion}"
   implementation "org.${project.properties.compileGroup}:${project.properties.compileVersion}"

buildSrc/version.properties

@@ -1,5 +1,5 @@
 opensearch        = 3.0.0
-lucene            = 9.4.0-snapshot-ddf0d0a
+lucene            = 9.4.0
 
 bundled_jdk_vendor = adoptium
 bundled_jdk = 17.0.4+8

client/rest-high-level/src/test/java/org/opensearch/client/RestHighLevelClientTests.java

@@ -889,7 +889,9 @@ public void testApiNamingConventions() throws Exception {
             "nodes.reload_secure_settings",
             "search_shards",
             "remote_store.restore",
-            "cluster.put_weighted_routing", };
+            "cluster.put_weighted_routing",
+            "cluster.put_decommission_awareness",
+            "cluster.get_decommission_awareness", };
         List<String> booleanReturnMethods = Arrays.asList("security.enable_user", "security.disable_user", "security.change_password");
         Set<String> deprecatedMethods = new HashSet<>();
         deprecatedMethods.add("indices.force_merge");

modules/lang-expression/licenses/lucene-expressions-9.4.0.jar.sha1

@@ -0,0 +1 @@
+19749e264805171009836cbedecc5494b13cd920

modules/transport-netty4/src/main/java/org/opensearch/http/netty4/Netty4HttpServerTransport.java

@@ -332,8 +332,10 @@ public ChannelHandler configureServerChannelHandler() {
         return new HttpChannelHandler(this, handlingSettings);
     }
 
-    static final AttributeKey<Netty4HttpChannel> HTTP_CHANNEL_KEY = AttributeKey.newInstance("opensearch-http-channel");
-    static final AttributeKey<Netty4HttpServerChannel> HTTP_SERVER_CHANNEL_KEY = AttributeKey.newInstance("opensearch-http-server-channel");
+    protected static final AttributeKey<Netty4HttpChannel> HTTP_CHANNEL_KEY = AttributeKey.newInstance("opensearch-http-channel");
+    protected static final AttributeKey<Netty4HttpServerChannel> HTTP_SERVER_CHANNEL_KEY = AttributeKey.newInstance(
+        "opensearch-http-server-channel"
+    );
 
     protected static class HttpChannelHandler extends ChannelInitializer<Channel> {
 

plugins/analysis-icu/licenses/lucene-analysis-icu-9.4.0.jar.sha1

@@ -0,0 +1 @@
+aa0f250558375922f3091820361156e514fe1842

plugins/analysis-kuromoji/licenses/lucene-analysis-kuromoji-9.4.0.jar.sha1

@@ -0,0 +1 @@
+32eb1ad367ab1289804aeed95ea7216711a7764d

plugins/analysis-nori/licenses/lucene-analysis-nori-9.4.0.jar.sha1

@@ -0,0 +1 @@
+63661714be65f882a921d281965b0779fd487b90

plugins/analysis-phonetic/licenses/lucene-analysis-phonetic-9.4.0.jar.sha1

@@ -0,0 +1 @@
+1034d876551fc21f7835b456dab01db21b9a4af6

plugins/analysis-smartcn/licenses/lucene-analysis-smartcn-9.4.0.jar.sha1

@@ -0,0 +1 @@
+f704ee4b14e2fe2622bb983f04b36a32df8fd4a7

plugins/analysis-stempel/licenses/lucene-analysis-stempel-9.4.0.jar.sha1

@@ -0,0 +1 @@
+a95ff17b51da6b3da641fa4053e5ee9ea2ff5daf

plugins/analysis-ukrainian/licenses/lucene-analysis-morfologik-9.4.0.jar.sha1

@@ -0,0 +1 @@
+13e1ae2c760d8c0d7990ffe3296e46d9d8e6f842

plugins/discovery-azure-classic/build.gradle

@@ -59,7 +59,7 @@ dependencies {
   api "com.sun.jersey:jersey-client:${versions.jersey}"
   api "com.sun.jersey:jersey-core:${versions.jersey}"
   api "com.sun.jersey:jersey-json:${versions.jersey}"
-  api 'org.codehaus.jettison:jettison:1.5.0'
+  api 'org.codehaus.jettison:jettison:1.5.1'
   api 'com.sun.xml.bind:jaxb-impl:2.2.3-1'
 
   // HACK: javax.xml.bind was removed from default modules in java 9, so we pull the api in here,

plugins/discovery-azure-classic/licenses/jettison-1.5.1.jar.sha1

@@ -0,0 +1 @@
+d8918f348f234f5046bd39ea1ed9fc91deac402f

plugins/repository-azure/build.gradle

@@ -45,7 +45,7 @@ opensearchplugin {
 
 dependencies {
   api 'com.azure:azure-core:1.31.0'
-  api 'com.azure:azure-storage-common:12.18.0'
+  api 'com.azure:azure-storage-common:12.18.1'
   api 'com.azure:azure-core-http-netty:1.12.4'
   api "io.netty:netty-codec-dns:${versions.netty}"
   api "io.netty:netty-codec-socks:${versions.netty}"
@@ -59,7 +59,7 @@ dependencies {
   api 'io.projectreactor:reactor-core:3.4.23'
   api 'io.projectreactor.netty:reactor-netty:1.0.18'
   api 'io.projectreactor.netty:reactor-netty-core:1.0.22'
-  api 'io.projectreactor.netty:reactor-netty-http:1.0.18'
+  api 'io.projectreactor.netty:reactor-netty-http:1.0.23'
   api "org.slf4j:slf4j-api:${versions.slf4j}"
   api "com.fasterxml.jackson.core:jackson-annotations:${versions.jackson}"
   api "com.fasterxml.jackson.core:jackson-databind:${versions.jackson_databind}"

plugins/repository-azure/licenses/azure-storage-common-12.18.1.jar.sha1

@@ -0,0 +1 @@
+09f5229d0775dff1b21cc3cb2936de751e79b5ac

plugins/repository-azure/licenses/reactor-netty-http-1.0.23.jar.sha1

@@ -0,0 +1 @@
+63932f2b675f451135986b3723a12d45e818b170

plugins/repository-hdfs/build.gradle

@@ -48,7 +48,7 @@ opensearchplugin {
 }
 
 versions << [
-  'hadoop3': '3.3.3'
+  'hadoop3': '3.3.4'
 ]
 
 testFixtures.useFixture ":test:fixtures:krb5kdc-fixture", "hdfs"

plugins/repository-hdfs/licenses/hadoop-client-api-3.3.4.jar.sha1

@@ -0,0 +1 @@
+6339a8f7279310c8b1f7ef314b592d8c71ca72ef

plugins/repository-hdfs/licenses/hadoop-client-runtime-3.3.4.jar.sha1

@@ -0,0 +1 @@
+21f7a9a2da446f1e5b3e5af16ebf956d3ee43ee0

plugins/repository-hdfs/licenses/hadoop-hdfs-3.3.4.jar.sha1

@@ -0,0 +1 @@
+036ef2f86dc44410d2bb5d54ce40435d2484d9a5

rest-api-spec/src/main/resources/rest-api-spec/api/cluster.get_decommission_awareness.json

@@ -0,0 +1,19 @@
+{
+  "cluster.get_decommission_awareness": {
+    "documentation": {
+      "url": "https://opensearch.org/docs/latest/opensearch/rest-api/decommission/",
+      "description": "Get details and status of decommissioned attribute"
+    },
+    "stability": "experimental",
+    "url": {
+      "paths": [
+        {
+          "path": "/_cluster/decommission/awareness/_status",
+          "methods": [
+            "GET"
+          ]
+        }
+      ]
+    }
+  }
+}

rest-api-spec/src/main/resources/rest-api-spec/api/cluster.put_decommission_awareness.json

@@ -0,0 +1,29 @@
+{
+  "cluster.put_decommission_awareness": {
+    "documentation": {
+      "url": "https://opensearch.org/docs/latest/opensearch/rest-api/decommission/",
+      "description": "Decommissions an awareness attribute"
+    },
+    "stability": "experimental",
+    "url": {
+      "paths": [
+        {
+          "path": "/_cluster/decommission/awareness/{awareness_attribute_name}/{awareness_attribute_value}",
+          "methods": [
+            "PUT"
+          ],
+          "parts": {
+            "awareness_attribute_name": {
+              "type": "string",
+              "description": "Awareness attribute name"
+            },
+            "awareness_attribute_value": {
+              "type": "string",
+              "description": "Awareness attribute value"
+            }
+          }
+        }
+      ]
+    }
+  }
+}

server/licenses/lucene-analysis-common-9.4.0.jar.sha1

@@ -0,0 +1 @@
+02fbd4e87241411fcf5d34e92a50bee46ab164dc

server/licenses/lucene-backward-codecs-9.4.0.jar.sha1

@@ -0,0 +1 @@
+259863dfd107645de6146b3c87b4ecee66a4d43d

server/licenses/lucene-core-9.4.0.jar.sha1

@@ -0,0 +1 @@
+cca1116f813c0f0c63acfac4c952baf29d46d76b

server/licenses/lucene-grouping-9.4.0.jar.sha1

@@ -0,0 +1 @@
+51bec1d5acc8ecaf9f50e047d3f86d60c7a958f4

server/licenses/lucene-highlighter-9.4.0.jar.sha1

@@ -0,0 +1 @@
+c8cf8c9308d8fb18a927c7ed267a14ace3990a5f

server/licenses/lucene-join-9.4.0.jar.sha1

@@ -0,0 +1 @@
+99b2d3c8e137a6853a2503456897d47d4f18974b

server/licenses/lucene-memory-9.4.0.jar.sha1

@@ -0,0 +1 @@
+881cb214e79da14de35cb0e8e6779d2722828a96

server/licenses/lucene-misc-9.4.0.jar.sha1

@@ -0,0 +1 @@
+a126123e482e6bf2e7aea670d221a2a39d3277dc

server/licenses/lucene-queries-9.4.0.jar.sha1

@@ -0,0 +1 @@
+fe74dbfe9dba9ee9ee2cb80f151fde97fb4efd12

server/licenses/lucene-queryparser-9.4.0.jar.sha1

@@ -0,0 +1 @@
+13f108a8572fcf0670c7df3ba8dbe1076d0e0dbe

server/licenses/lucene-sandbox-9.4.0.jar.sha1

@@ -0,0 +1 @@
+e7a676a12ea50dcbf64564f4e4022f939f0a627d

server/licenses/lucene-spatial-extras-9.4.0.jar.sha1

@@ -0,0 +1 @@
+84d956d1cb1458c51967af1c4acadd2a1f92634d

server/licenses/lucene-spatial3d-9.4.0.jar.sha1

@@ -0,0 +1 @@
+76887ca708f23b13613e45fb9e307c548b22c6da

server/licenses/lucene-suggest-9.4.0.jar.sha1

@@ -0,0 +1 @@
+406c9c539f262449d3b1e57e7bc4302efeecaf6c

server/src/internalClusterTest/java/org/opensearch/snapshots/DedicatedClusterSnapshotRestoreIT.java

@@ -1475,6 +1475,31 @@ public void testSnapshotDeleteRelocatingPrimaryIndex() throws Exception {
         logger.info("--> done");
     }
 
+    public void testIndexDeletionDuringSnapshotCreationInQueue() throws Exception {
+        assertAcked(prepareCreate("test-idx", 1, indexSettingsNoReplicas(1)));
+        ensureGreen();
+        indexRandomDocs("test-idx", 100);
+        createRepository("test-repo", "fs");
+        createSnapshot("test-repo", "test-snap", Collections.singletonList("test-idx"));
+
+        logger.info("--> create snapshot to be deleted and then delete");
+        createSnapshot("test-repo", "test-snap-delete", Collections.singletonList("test-idx"));
+        clusterAdmin().prepareDeleteSnapshot("test-repo", "test-snap-delete").execute();
+
+        logger.info("--> create snapshot before index deletion during above snapshot deletion");
+        clusterAdmin().prepareCreateSnapshot("test-repo", "test-snap-2")
+            .setWaitForCompletion(false)
+            .setPartial(true)
+            .setIndices("test-idx")
+            .get();
+
+        logger.info("delete index during snapshot creation");
+        assertAcked(admin().indices().prepareDelete("test-idx"));
+
+        clusterAdmin().prepareRestoreSnapshot("test-repo", "test-snap").get();
+        ensureGreen("test-idx");
+    }
+
     private long calculateTotalFilesSize(List<Path> files) {
         return files.stream().mapToLong(f -> {
             try {