Support ES version 7.6.1 (opensearch-project#164)

* 1.6 Upgrade for ES 7.6.1 integTest api name change; comment out all non-existing tasks; comment out jacoco related part * Nearly Finished Upgrade to v1.6 + Jacoco fixed by Drew Baugher - NotificationActionIT test failure * checkout to master to have a working version * Fix null warning Fix URI error Add gradle change * ready to PR * Fix Commit 1 1. Remove Comment 2. Don't use !! in production code, using it in tests is fine, this opens up possibility for NPEs 3. Fix Release * Fix Commit 2 1. repository order change mavenlocal to the last 2. update README 3. --debug-jvm cannot use by now, saying "Unknown command-line option" * Update README * update README * Fix Commit 3 + correct debug settings + add more info in release * optimize release note * github action checkoutv2
ronnaksaxena · Mar 30, 2020 · 64eaace · 64eaace
1 parent 65e4be8
commit 64eaace
Show file tree

Hide file tree

Showing 15 changed files with 148 additions and 112 deletions.
diff --git a/.github/workflows/test-and-build-workflow.yml b/.github/workflows/test-and-build-workflow.yml
@@ -10,7 +10,7 @@ jobs:
   build:
     strategy:
       matrix:
-        java: [12]
+        java: [13]
     # Job name
     name: Build Index Management with JDK ${{ matrix.java }}
     # This job runs on Linux

diff --git a/README.md b/README.md
@@ -41,7 +41,7 @@ Please see our [documentation](https://opendistro.github.io/for-elasticsearch-do
 
 1. Check out this package from version control.
 2. Launch Intellij IDEA, choose **Import Project**, and select the `settings.gradle` file in the root of this package. 
-3. To build from the command line, set `JAVA_HOME` to point to a JDK >= 12 before running `./gradlew`.
+3. To build from the command line, set `JAVA_HOME` to point to a JDK >= 13 before running `./gradlew`.
   - Unix System
     1. `export JAVA_HOME=jdk-install-dir`: Replace `jdk-install-dir` with the JAVA_HOME directory of your system.
     2. `export PATH=$JAVA_HOME/bin:$PATH`
@@ -74,14 +74,16 @@ This project currently uses the Notification subproject from the [Alerting plugi
 4. `./gradlew integTest -Dtests.class=*RestChangePolicyActionIT` runs a single integ class
 5.  `./gradlew integTest -Dtests.class=*RestChangePolicyActionIT -Dtests.method="test missing index"` runs a single integ test method (remember to quote the test method name if it contains spaces)
 
-When launching a cluster using one of the above commands, logs are placed in `build/cluster/run node0/elasticsearch-<version>/logs`. Though the logs are teed to the console, in practices it's best to check the actual log file.
+When launching a cluster using one of the above commands, logs are placed in `build/testclusters/integTest-0/logs`. Though the logs are teed to the console, in practices it's best to check the actual log file.
 
 ### Debugging
 
-Sometimes it's useful to attach a debugger to either the Elasticsearch cluster or the integ tests to see what's going on. When running unit tests, hit **Debug** from the IDE's gutter to debug the tests.  To debug code running in an actual server, run:
+Sometimes it is useful to attach a debugger to either the Elasticsearch cluster or the integ tests to see what's going on. When running unit tests, hit **Debug** from the IDE's gutter to debug the tests.  For the Elasticsearch cluster or the integ tests, first, make sure start a debugger listening on port `5005`. 
+
+To debug the server code, run:
 
 ```
-./gradlew integTest --debug-jvm # to start a cluster and run integ tests
+./gradlew :integTest -Dcluster.debug # to start a cluster with debugger and run integ tests
 ```
 
 OR
@@ -90,15 +92,25 @@ OR
 ./gradlew run --debug-jvm # to just start a cluster that can be debugged
 ```
 
-The Elasticsearch server JVM will launch suspended and wait for a debugger to attach to `localhost:8000` before starting the Elasticsearch server.
+The Elasticsearch server JVM will connect to a debugger attached to `localhost:5005`.
+
+The IDE needs to listen for the remote JVM. If using Intellij you must set your debug configuration to "Listen to remote JVM" and make sure "Auto Restart" is checked.
+You must start your debugger to listen for remote JVM before running the commands.
 
-To debug code running in an integ test (which exercises the server from a separate JVM), run:
+To debug code running in an integration test (which exercises the server from a separate JVM), first, setup a remote debugger listening on port `8000`, and then run:
 
 ```
-./gradlew -Dtest.debug integTest 
+./gradlew :integTest -Dtest.debug
 ```
 
-The test runner JVM will start suspended and wait for a debugger to attach to `localhost:5005` before running the tests.
+The test runner JVM will connect to a debugger attached to `localhost:8000` before running the tests.
+
+Additionally, it is possible to attach one debugger to the cluster JVM and another debugger to the test runner. First, make sure one debugger is listening on port `5005` and the other is listening on port `8000`. Then, run:
+```
+./gradlew :integTest -Dtest.debug -Dcluster.debug
+```
+
+
 
 ## Code of Conduct
 

diff --git a/build-tools/esplugin-coverage.gradle b/build-tools/esplugin-coverage.gradle
@@ -13,9 +13,6 @@
  *   permissions and limitations under the License.
  */
 
-import javax.management.remote.JMXConnectorFactory
-import javax.management.remote.JMXServiceURL
-
 /**
  * ES Plugin build tools don't work with the Gradle Jacoco Plugin to report coverage out of the box.
  * https://github.com/elastic/elasticsearch/issues/28867.
@@ -29,32 +26,39 @@ import javax.management.remote.JMXServiceURL
  *  cluster is stopped and dump it to a file. Luckily our current security policy seems to allow this. This will also probably
  *  break if there are multiple nodes in the integTestCluster. But for now... it sorta works.
  */
+
 apply plugin: 'jacoco'
 
 // Get gradle to generate the required jvm agent arg for us using a dummy tasks of type Test. Unfortunately Elastic's
 // testing tasks don't derive from Test so the jacoco plugin can't do this automatically.
+def jacocoDir = "${buildDir}/jacoco"
 task dummyTest(type: Test) {
     enabled = false
     workingDir = file("/") // Force absolute path to jacoco agent jar
     jacoco {
-        destinationFile = file("${buildDir}/jacoco/test.exec")
+        destinationFile = file("${jacocoDir}/test.exec")
         destinationFile.parentFile.mkdirs()
         jmx = true
     }
 }
 
+
 task dummyIntegTest(type: Test) {
     enabled = false
     workingDir = file("/") // Force absolute path to jacoco agent jar
     jacoco {
-        destinationFile = file("${buildDir}/jacoco/integTest.exec")
+        destinationFile = file("${jacocoDir}/integTest.exec")
         destinationFile.parentFile.mkdirs()
         jmx = true
     }
 }
 
-integTestCluster {
-    jvmArgs += " ${dummyIntegTest.jacoco.getAsJvmArg()}"
+integTest.runner {
+    systemProperty 'jacoco.dir', "${jacocoDir}"
+}
+
+testClusters.integTest {
+    jvmArgs " ${dummyIntegTest.jacoco.getAsJvmArg()}"
     systemProperty 'com.sun.management.jmxremote', "true"
     systemProperty 'com.sun.management.jmxremote.authenticate', "false"
     systemProperty 'com.sun.management.jmxremote.port', "7777"
@@ -65,30 +69,15 @@ integTestCluster {
 jacocoTestReport {
     dependsOn integTest, test
     executionData dummyTest.jacoco.destinationFile, dummyIntegTest.jacoco.destinationFile
-    sourceDirectories = sourceSets.main.allSource
-    classDirectories = sourceSets.main.output
+    sourceDirectories.from = sourceSets.main.allSource
+    classDirectories.from = sourceSets.main.output
     reports {
         html.enabled = true // human readable
         xml.enabled = true // for coverlay
     }
 }
 
-// See https://www.eclemma.org/jacoco/trunk/doc/api/org/jacoco/agent/rt/IAgent.html
-task dumpCoverage {
-    doFirst {
-        def serverUrl = "service:jmx:rmi:///jndi/rmi://127.0.0.1:7777/jmxrmi"
-        def connector = JMXConnectorFactory.connect(new JMXServiceURL(serverUrl))
-        try {
-            def jacocoMBean = new GroovyMBean(connector.MBeanServerConnection, "org.jacoco:type=Runtime")
-            byte[] data = jacocoMBean.getExecutionData(false)
-            file(dummyIntegTest.jacoco.destinationFile).setBytes(data)
-        } finally {
-            connector.close()
-        }
-    }
-}
 project.gradle.projectsEvaluated {
-    dumpCoverage.dependsOn integTestRunner
-    tasks['integTestCluster#stop'].dependsOn dumpCoverage
-    jacocoTestReport.dependsOn dumpCoverage
+    jacocoTestReport.dependsOn integTest.runner
 }
+
diff --git a/build.gradle b/build.gradle
@@ -15,14 +15,14 @@
 
 buildscript {
     ext {
-        es_version = System.getProperty("es.version", "7.4.2")
-        kotlin_version = System.getProperty("kotlin.version", "1.3.31")
+        es_version = System.getProperty("es.version", "7.6.1")
+        kotlin_version = System.getProperty("kotlin.version", "1.3.61")
     }
 
     repositories {
         mavenCentral()
         maven { url "https://plugins.gradle.org/m2/" }
-        // TODO: remove mavenLocal once notification is published to maven
+        // For local publish dependency
         mavenLocal()
     }
 
@@ -36,18 +36,19 @@ buildscript {
 }
 
 plugins {
-    id 'nebula.ospackage' version "5.3.0"
+    id 'nebula.ospackage' version "8.2.0"
 }
 
 apply plugin: 'java'
 apply plugin: 'jacoco'
 apply plugin: 'idea'
 apply plugin: 'elasticsearch.esplugin'
+apply plugin: 'elasticsearch.testclusters'
 apply plugin: 'io.gitlab.arturbosch.detekt'
 apply plugin: 'org.jetbrains.kotlin.jvm'
 apply plugin: 'org.jetbrains.kotlin.plugin.allopen'
 if (!System.properties.containsKey('tests.rest.cluster') && !System.properties.containsKey('tests.cluster')) {
-    apply from: 'build-tools/esplugin-coverage.gradle'
+   apply from: 'build-tools/esplugin-coverage.gradle'
 }
 check.dependsOn jacocoTestReport
 
@@ -73,12 +74,12 @@ detekt {
 
 dependencies {
     compileOnly "org.elasticsearch:elasticsearch:${es_version}"
-    compileOnly "com.amazon.opendistroforelasticsearch:opendistro-job-scheduler-spi:1.4.0.0"
+    compileOnly "com.amazon.opendistroforelasticsearch:opendistro-job-scheduler-spi:1.6.0.0"
     compile "org.jetbrains.kotlin:kotlin-stdlib:${kotlin_version}"
     compile "org.jetbrains.kotlin:kotlin-stdlib-common:${kotlin_version}"
     compile 'org.jetbrains.kotlinx:kotlinx-coroutines-core:1.2.1'
     compile "org.jetbrains:annotations:13.0"
-    compile "com.amazon.opendistroforelasticsearch:notification:1.4.0.0"
+    compile "com.amazon.opendistroforelasticsearch:notification:1.6.0.0"
 
     testCompile "org.elasticsearch.test:framework:${es_version}"
     testCompile "org.jetbrains.kotlin:kotlin-test:${kotlin_version}"
@@ -109,6 +110,7 @@ if (isSnapshot) {
 plugins.withId('java') {
     sourceCompatibility = targetCompatibility = "1.8"
 }
+
 plugins.withId('org.jetbrains.kotlin.jvm') {
     compileKotlin.kotlinOptions.jvmTarget = compileTestKotlin.kotlinOptions.jvmTarget = "1.8"
 }
@@ -125,49 +127,26 @@ test {
     systemProperty 'tests.security.manager', 'false'
 }
 
-integTestRunner {
+testClusters.integTest {
+    testDistribution = "OSS"
+    if (System.getProperty("cluster.debug") != null) {
+        jvmArgs('-agentlib:jdwp=transport=dt_socket,server=n,suspend=y,address=5005')
+    }
+    plugin(fileTree("src/test/resources/job-scheduler").getSingleFile())
+}
+
+integTest.runner {
     systemProperty 'tests.security.manager', 'false'
     systemProperty 'java.io.tmpdir', es_tmp_dir.absolutePath
     // Tell the test JVM if the cluster JVM is running under a debugger so that tests can use longer timeouts for
     // requests. The 'doFirst' delays reading the debug setting on the cluster till execution time.
-    doFirst { systemProperty 'cluster.debug', integTestCluster.debug }
-
-    // The --debug-jvm command-line option makes the cluster debuggable; this makes the tests debuggable
-    if (System.getProperty("test.debug") != null) {
-        jvmArgs '-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005'
+    doFirst {
+        systemProperty 'cluster.debug', getDebug()
     }
-}
-
-integTestCluster {
-    distribution = "oss-zip"
-    // Temporary until job-scheduler is published to Maven
-    setupCommand('installPlugin', 'bin/elasticsearch-plugin', 'install',
-            "file://${fileTree("src/test/resources/job-scheduler").getSingleFile().absolutePath}")
-}
-
-
-run {
-    distribution = "oss-zip"
-    // Temporary until job-scheduler is published to Maven
-    setupCommand('installPlugin', 'bin/elasticsearch-plugin', 'install',
-            "file://${fileTree("src/test/resources/job-scheduler").getSingleFile().absolutePath}")
-}
 
-afterEvaluate {
-    project.tasks.getByName("run#installOpendistroIndexManagementPlugin").dependsOn("run#installPlugin")
-    project.tasks.getByName("run#installPlugin").dependsOn.remove(project.tasks.getByName("run#installOpendistroIndexManagementPlugin"))
-    project.tasks.getByName("run#installPlugin").dependsOn("run#copyPlugins")
-    project.tasks.getByName("run#start").dependsOn.remove(project.tasks.getByName("run#installPlugin"))
-    project.tasks.getByName("run#start").dependsOn("run#installOpendistroIndexManagementPlugin")
-}
-
-tasks.whenTaskAdded { task ->
-    if (task.name == "integTestCluster#wait") {
-        project.tasks.getByName("integTestCluster#installOpendistroIndexManagementPlugin").dependsOn("integTestCluster#installPlugin")
-        project.tasks.getByName("integTestCluster#installPlugin").dependsOn.remove(project.tasks.getByName("integTestCluster#installOpendistroIndexManagementPlugin"))
-        project.tasks.getByName("integTestCluster#installPlugin").dependsOn("integTestCluster#copyPlugins")
-        project.tasks.getByName("integTestCluster#start").dependsOn.remove(project.tasks.getByName("integTestCluster#installPlugin"))
-        project.tasks.getByName("integTestCluster#start").dependsOn("integTestCluster#installOpendistroIndexManagementPlugin")
+    // The -Dcluster.debug option makes the cluster debuggable; this makes the tests debuggable
+    if (System.getProperty("test.debug") != null) {
+        jvmArgs '-agentlib:jdwp=transport=dt_socket,server=n,suspend=y,address=8000'
     }
 }
 
@@ -180,6 +159,7 @@ task ktlint(type: JavaExec, group: "verification") {
     // "--reporter=plain", "--reporter=checkstyle,output=${buildDir}/ktlint.xml"
     // see https://github.com/pinterest/ktlint#usage for more
 }
+
 check.dependsOn ktlint
 
 task ktlintFormat(type: JavaExec, group: "formatting") {

diff --git a/gradle.properties b/gradle.properties
@@ -13,4 +13,4 @@
 #   permissions and limitations under the License.
 #
 
-version = 1.4.0
+version = 1.6.0
diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar
diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties
@@ -1,6 +1,6 @@
-#Mon May 13 18:01:35 PDT 2019
+#Tue Mar 24 11:18:08 PDT 2020
+distributionUrl=https\://services.gradle.org/distributions/gradle-6.2.2-all.zip
 distributionBase=GRADLE_USER_HOME
 distributionPath=wrapper/dists
-zipStoreBase=GRADLE_USER_HOME
 zipStorePath=wrapper/dists
-distributionUrl=https\://services.gradle.org/distributions/gradle-5.6.2-all.zip
+zipStoreBase=GRADLE_USER_HOME
diff --git a/gradlew b/gradlew
@@ -7,7 +7,7 @@
 # you may not use this file except in compliance with the License.
 # You may obtain a copy of the License at
 #
-#      http://www.apache.org/licenses/LICENSE-2.0
+#      https://www.apache.org/licenses/LICENSE-2.0
 #
 # Unless required by applicable law or agreed to in writing, software
 # distributed under the License is distributed on an "AS IS" BASIS,
@@ -125,8 +125,8 @@ if $darwin; then
     GRADLE_OPTS="$GRADLE_OPTS \"-Xdock:name=$APP_NAME\" \"-Xdock:icon=$APP_HOME/media/gradle.icns\""
 fi
 
-# For Cygwin, switch paths to Windows format before running java
-if $cygwin ; then
+# For Cygwin or MSYS, switch paths to Windows format before running java
+if [ "$cygwin" = "true" -o "$msys" = "true" ] ; then
     APP_HOME=`cygpath --path --mixed "$APP_HOME"`
     CLASSPATH=`cygpath --path --mixed "$CLASSPATH"`
     JAVACMD=`cygpath --unix "$JAVACMD"`
@@ -154,19 +154,19 @@ if $cygwin ; then
         else
             eval `echo args$i`="\"$arg\""
         fi
-        i=$((i+1))
+        i=`expr $i + 1`
     done
     case $i in
-        (0) set -- ;;
-        (1) set -- "$args0" ;;
-        (2) set -- "$args0" "$args1" ;;
-        (3) set -- "$args0" "$args1" "$args2" ;;
-        (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
-        (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
-        (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
-        (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
-        (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
-        (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
+        0) set -- ;;
+        1) set -- "$args0" ;;
+        2) set -- "$args0" "$args1" ;;
+        3) set -- "$args0" "$args1" "$args2" ;;
+        4) set -- "$args0" "$args1" "$args2" "$args3" ;;
+        5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
+        6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
+        7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
+        8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
+        9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;
     esac
 fi
 
@@ -175,14 +175,9 @@ save () {
     for i do printf %s\\n "$i" | sed "s/'/'\\\\''/g;1s/^/'/;\$s/\$/' \\\\/" ; done
     echo " "
 }
-APP_ARGS=$(save "$@")
+APP_ARGS=`save "$@"`
 
 # Collect all arguments for the java command, following the shell quoting and substitution rules
 eval set -- $DEFAULT_JVM_OPTS $JAVA_OPTS $GRADLE_OPTS "\"-Dorg.gradle.appname=$APP_BASE_NAME\"" -classpath "\"$CLASSPATH\"" org.gradle.wrapper.GradleWrapperMain "$APP_ARGS"
 
-# by default we should be in the correct project dir, but when run from Finder on Mac, the cwd is wrong
-if [ "$(uname)" = "Darwin" ] && [ "$HOME" = "$PWD" ]; then
-  cd "$(dirname "$0")"
-fi
-
 exec "$JAVACMD" "$@"
diff --git a/gradlew.bat b/gradlew.bat
@@ -5,7 +5,7 @@
 @rem you may not use this file except in compliance with the License.
 @rem You may obtain a copy of the License at
 @rem
-@rem      http://www.apache.org/licenses/LICENSE-2.0
+@rem      https://www.apache.org/licenses/LICENSE-2.0
 @rem
 @rem Unless required by applicable law or agreed to in writing, software
 @rem distributed under the License is distributed on an "AS IS" BASIS,
@@ -29,6 +29,9 @@ if "%DIRNAME%" == "" set DIRNAME=.
 set APP_BASE_NAME=%~n0
 set APP_HOME=%DIRNAME%
 
+@rem Resolve any "." and ".." in APP_HOME to make it shorter.
+for %%i in ("%APP_HOME%") do set APP_HOME=%%~fi
+
 @rem Add default JVM options here. You can also use JAVA_OPTS and GRADLE_OPTS to pass JVM options to this script.
 set DEFAULT_JVM_OPTS="-Xmx64m" "-Xms64m"