Mechanism to close the created Gradle Check AUTOCUT flaky test issues (…

…#448) Signed-off-by: Prudhvi Godithi <[email protected]> (cherry picked from commit 9a7a323) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-project · Jun 26, 2024 · ce3aee0 · ce3aee0
1 parent d710e7c
commit ce3aee0
Show file tree

Hide file tree

Showing 18 changed files with 586 additions and 24 deletions.
diff --git a/.codecov.yml b/.codecov.yml
@@ -6,5 +6,5 @@ coverage:
   status:
     project:
       default:
-        target: auto
-        threshold: 0.2%
+        target: 70%
+        threshold: 3%
diff --git a/.github/workflows/groovy-tests.yml b/.github/workflows/groovy-tests.yml
@@ -22,6 +22,7 @@ jobs:
         run: |
           ./gradlew test --info
       - name: Upload Coverage Report
-        uses: codecov/codecov-action@v3.1.0
+        uses: codecov/codecov-action@v4
         with:
+          token: ${{ secrets.CODECOV_TOKEN }}
           files: ./build/reports/jacoco/test/jacocoTestReport.xml
diff --git a/README.md b/README.md
@@ -38,10 +38,13 @@ lib = library(identifier: 'jenkins@<tag>', retriever: modernSCM([
 | [publishToRubyGems.groovy](./vars/publishToRubyGems.groovy)                                                 | A library to publish gems to [rubygems.org](https://rubygems.org/) with [opensearchproject](https://rubygems.org/profiles/opensearchproject) as the owner. Please note that this library expects the gems to be pre-signed. You can use [PublishToRubyGemsLibTester](./tests/jenkins/lib-testers/PublishToRubyGemsLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToRubyGems_JenkinsFile).                                                                 |
 | [publishToMaven.groovy](./vars/publishToMaven.groovy)                                                       | A library to sign and deploy opensearch maven artifacts to sonatype staging repository, it also has an optional parameter `autoPublish` to auto-release artifacts from staging repo to prod without manual intervention. You can use [PublishToMavenLibTester](./tests/jenkins/lib-testers/PublishToMavenLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToMaven_JenkinsFile).                                                                                  |
 | [publishToNuget.groovy](./vars/publishToNuget.groovy)                                                       | A library to build, sign and publish dotnet artifacts to [Nuget Gallery](https://www.nuget.org/). Please check if the [default docker](https://github.com/opensearch-project/opensearch-build/blob/main/docker/ci/dockerfiles/current/release.centos.clients.x64.arm64.dockerfile) file contains the required dotnet sdk. You can use [PublishToNugetLibTester](./tests/jenkins/lib-testers/PublishToNugetLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToNuget_Jenkinsfile).
-| [publishToArtifactsProdBucket.groovy](./vars/publishToArtifactsProdBucket.groovy)                                                       | This library signs and uploads the artifacts to production S3 bucket which points to artifacts.opensearch.org. Please make sure the role that you use to upload exists and has the right permission. For artifacts of different types like macos, linux and windows, call this lib for each artifact with different signing parameters. You can use [PublishToArtifactsProdBucketLibTester](./tests/jenkins/lib-testers/PublishToArtifactsProdBucketLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToArtifactsProdBucket_Jenkinsfile). 
+| [publishToArtifactsProdBucket.groovy](./vars/publishToArtifactsProdBucket.groovy)                                                     | This library signs and uploads the artifacts to production S3 bucket which points to artifacts.opensearch.org. Please make sure the role that you use to upload exists and has the right permission. For artifacts of different types like macos, linux and windows, call this lib for each artifact with different signing parameters. You can use [PublishToArtifactsProdBucketLibTester](./tests/jenkins/lib-testers/PublishToArtifactsProdBucketLibTester.groovy) to add tests in your repository. See how to use the lib in your [jenkinsFile](./tests/jenkins/jobs/PublishToArtifactsProdBucket_Jenkinsfile). 
 | [buildMessage.groovy](./vars/buildMessage.groovy)                                                       | This library that can parse the jenkins build log based on the user defined input query string. 
-| [closeBuildSuccessGithubIssue.groovy](./vars/closeBuildSuccessGithubIssue.groovy)                                                       | This library that identifies the successfully built components and closes the created [AUTOCUT] issues.
+| [closeBuildSuccessGithubIssue.groovy](./vars/closeBuildSuccessGithubIssue.groovy)                                                     | This library that identifies the successfully built components and closes the created [AUTOCUT] issues.
 | [createGithubIssue.groovy](./vars/createGithubIssue.groovy)                                                       | This library that identifies the failed components and creates the [AUTOCUT] issues.
+| [publishGradleCheckTestResults.groovy](./vars/publishGradleCheckTestResults.groovy)                                                       | This library runs part of Gradle Check and publishes the failed test data to the [OpenSearch Metrics Cluster](https://metrics.opensearch.org/_dashboards/app/dashboards#/view/e5e64d40-ed31-11ee-be99-69d1dbc75083).
+| [gradleCheckFlakyTestDetector.groovy](./vars/gradleCheckFlakyTestDetector.groovy)                                                       | This library detects the flaky tests from [OpenSearch Metrics Cluster](https://metrics.opensearch.org/_dashboards/app/dashboards#/view/e5e64d40-ed31-11ee-be99-69d1dbc75083) and generates a test report.
+| [gradleCheckFlakyTestGitHubIssue.groovy](./vars/gradleCheckFlakyTestGitHubIssue.groovy)                                                       | This library is used in [gradleCheckFlakyTestDetector.groovy](./vars/gradleCheckFlakyTestDetector.groovy) to create/edit the GitHub Issue using the generated test report.
 
 ## Contributing
 

diff --git a/build.gradle b/build.gradle
@@ -127,7 +127,7 @@ jacocoTestReport {
     }
 }
 
-String version = '6.5.1'
+String version = '6.5.2'
 
 task updateVersion {
     doLast {

diff --git a/src/gradlecheck/FetchPostMergeFailedTestClass.groovy b/src/gradlecheck/FetchPostMergeFailedTestClass.groovy
@@ -27,7 +27,7 @@ class FetchPostMergeFailedTestClass {
         this.script = script
     }
 
-    def getQuery() {
+    def getQuery(timeFrame) {
         def queryMap = [
                 size: 200,
                 query: [
@@ -52,11 +52,33 @@ class FetchPostMergeFailedTestClass {
                                                 match: [
                                                         test_status: [
                                                                 query: "FAILED",
-                                                                operator: "OR"
+                                                                operator: "OR",
+                                                                prefix_length: 0,
+                                                                max_expansions: 50,
+                                                                fuzzy_transpositions: true,
+                                                                lenient: false,
+                                                                zero_terms_query: "NONE",
+                                                                auto_generate_synonyms_phrase_query: true,
+                                                                boost: 1
                                                         ]
                                                 ]
                                         ]
-                                ]
+                                ],
+                                filter: [
+                                        [
+                                                range: [
+                                                        build_start_time: [
+                                                                from: "now-${timeFrame}",
+                                                                to: "now",
+                                                                include_lower: true,
+                                                                include_upper: true,
+                                                                boost: 1
+                                                        ]
+                                                ]
+                                        ]
+                                ],
+                                adjust_pure_negative: true,
+                                boost: 1
                         ]
                 ],
                 aggregations: [
@@ -68,13 +90,12 @@ class FetchPostMergeFailedTestClass {
                         ]
                 ]
         ]
-
         def query = JsonOutput.toJson(queryMap)
         return query.replace('"', '\\"')
     }
 
-    def getPostMergeFailedTestClass() {
-         def jsonResponse = new OpenSearchMetricsQuery(metricsUrl,awsAccessKey, awsSecretKey, awsSessionToken, script).fetchMetrics(getQuery())
+    def getPostMergeFailedTestClass(timeFrame) {
+         def jsonResponse = new OpenSearchMetricsQuery(metricsUrl,awsAccessKey, awsSecretKey, awsSessionToken, script).fetchMetrics(getQuery(timeFrame))
          def keys = jsonResponse.aggregations.test_class_keyword_agg.buckets.collect { it.key }
          return keys
     }

diff --git a/src/gradlecheck/MarkdownComparator.groovy b/src/gradlecheck/MarkdownComparator.groovy
@@ -0,0 +1,33 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package gradlecheck
+
+class MarkdownComparator {
+
+    ArrayList<String> testReportMarkDownTable
+    ArrayList<String> gitHubMarkDownTable
+
+    MarkdownComparator(ArrayList<String> testReportMarkDownTable, ArrayList<String> gitHubMarkDownTable) {
+        this.testReportMarkDownTable = testReportMarkDownTable
+        this.gitHubMarkDownTable = gitHubMarkDownTable
+    }
+
+    def markdownComparison() {
+        def differences = testReportMarkDownTable.findAll { ghRow ->
+            !gitHubMarkDownTable.any { compRow ->
+                ghRow['Git Reference'] == compRow['Git Reference'] &&
+                        ghRow['Merged Pull Request'] == compRow['Merged Pull Request'] &&
+                        ghRow['Build Details'] == compRow['Build Details'] &&
+                        ghRow['Test Name'] == compRow['Test Name']
+            }
+        }
+        return differences
+    }
+}
diff --git a/src/gradlecheck/OpenSearchMetricsQuery.groovy b/src/gradlecheck/OpenSearchMetricsQuery.groovy
@@ -26,14 +26,14 @@ class OpenSearchMetricsQuery {
         this.script = script
     }
 
+    // Ensure the alias `gradle-check` is created targeting all the gradle-check-* indices.
     def fetchMetrics(String query) {
         def response = script.sh(
             script: """
             set -e
             set +x
             MONTH_YEAR=\$(date +"%m-%Y")
-            INDEX_NAME="gradle-check-\$MONTH_YEAR"
-            curl -s -XGET "${metricsUrl}/\$INDEX_NAME/_search" --aws-sigv4 "aws:amz:us-east-1:es" --user "${awsAccessKey}:${awsSecretKey}" -H "x-amz-security-token:${awsSessionToken}" -H 'Content-Type: application/json' -d "${query}" | jq '.'
+            curl -s -XGET "${metricsUrl}/gradle-check/_search" --aws-sigv4 "aws:amz:us-east-1:es" --user "${awsAccessKey}:${awsSecretKey}" -H "x-amz-security-token:${awsSessionToken}" -H 'Content-Type: application/json' -d "${query}" | jq '.'
         """,
                 returnStdout: true
         ).trim()

diff --git a/src/gradlecheck/ParseMarkDownTable.groovy b/src/gradlecheck/ParseMarkDownTable.groovy
@@ -0,0 +1,32 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package gradlecheck
+
+class ParseMarkDownTable {
+    String markdown
+
+    ParseMarkDownTable(String markdown) {
+        this.markdown = markdown
+    }
+
+    def parseMarkdownTableRows() {
+        def rows = markdown.split("\\|\\s*\\n")
+        rows = rows[2..-2] // Skipping headers and footer
+        return rows.collect { row ->
+            def cells = row.split("\\|\\s*")
+            return [
+                    'Git Reference': cells[1].trim(),
+                    'Merged Pull Request': cells[2].trim(),
+                    'Build Details': cells[3].trim(),
+                    'Test Name': cells[4].trim()
+            ]
+        }
+    }
+}
diff --git a/tests/gradlecheck/CreateMarkDownTableTest.groovy b/tests/gradlecheck/CreateMarkDownTableTest.groovy
@@ -22,11 +22,8 @@ class CreateMarkDownTableTest  {
                 [gitReference: "def456", pullRequestLink: "https://github.com/opensearch-project/OpenSearch/pull/2", buildDetailLink: "https://ci.opensearch.org/2", testNames: ["test3"]]
         ]
         def additionalPullRequests = ["3", "4"]
-
         def createMarkDownTable = new CreateMarkDownTable(failedTest, tableData, additionalPullRequests)
-
         def result = createMarkDownTable.createMarkdownTable()
-
         def expectedOutput = """
 ## Flaky Test Report for `ExampleTest`
 

diff --git a/tests/gradlecheck/FetchPostMergeFailedTestClassTest.groovy b/tests/gradlecheck/FetchPostMergeFailedTestClassTest.groovy
@@ -71,11 +71,33 @@ class FetchPostMergeFailedTestClassTest {
                                                 match: [
                                                         test_status: [
                                                                 query: "FAILED",
-                                                                operator: "OR"
+                                                                operator: "OR",
+                                                                prefix_length: 0,
+                                                                max_expansions: 50,
+                                                                fuzzy_transpositions: true,
+                                                                lenient: false,
+                                                                zero_terms_query: "NONE",
+                                                                auto_generate_synonyms_phrase_query: true,
+                                                                boost: 1
                                                         ]
                                                 ]
                                         ]
-                                ]
+                                ],
+                                filter: [
+                                        [
+                                                range: [
+                                                        build_start_time: [
+                                                                from: "now-15d",
+                                                                to: "now",
+                                                                include_lower: true,
+                                                                include_upper: true,
+                                                                boost: 1
+                                                        ]
+                                                ]
+                                        ]
+                                ],
+                                adjust_pure_negative: true,
+                                boost: 1
                         ]
                 ],
                 aggregations: [
@@ -88,16 +110,16 @@ class FetchPostMergeFailedTestClassTest {
                 ]
         ]).replace('"', '\\"')
 
-        def result = fetchPostMergeFailedTestClass.getQuery()
+        def result = fetchPostMergeFailedTestClass.getQuery("15d")
 
         assert result == expectedOutput
     }
 
     @Test
     void testGetPostMergeFailedTestClassReturnsKeys() {
         def expectedOutput = ["testClass1", "testClass2"]
-
-        def result = fetchPostMergeFailedTestClass.getPostMergeFailedTestClass()
+        def timeFrame = "15d"
+        def result = fetchPostMergeFailedTestClass.getPostMergeFailedTestClass(timeFrame)
 
         assert result == expectedOutput
     }