Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add OpenSearch failed Test Class and Name fields to test-report.yml #5101

Merged
merged 3 commits into from
Oct 11, 2024
Merged

Add OpenSearch failed Test Class and Name fields to test-report.yml #5101

merged 3 commits into from
Oct 11, 2024

Conversation

peterzhuamazon
Copy link
Member

@peterzhuamazon peterzhuamazon commented Oct 11, 2024 •
edited
Loading

Description

Add OpenSearch failed Test Class and Name fields to test-report.yml

Issues Resolved

opensearch-project/opensearch-metrics#68

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@peterzhuamazon peterzhuamazon self-assigned this Oct 11, 2024
@peterzhuamazon peterzhuamazon changed the title Add OpenSearch Test Class and Name fields to test-report.yml Add OpenSearch failed Test Class and Name fields to test-report.yml Oct 11, 2024
@peterzhuamazon peterzhuamazon mentioned this pull request Oct 11, 2024
Closed
@peterzhuamazon peterzhuamazon merged commit fded093 into opensearch-project:main Oct 11, 2024
10 checks passed
@peterzhuamazon peterzhuamazon deleted the report-test-case branch October 11, 2024 23:26
@prudhvigodithi
Copy link
Member

Coming from here Peter https://build.ci.opensearch.org/job/integ-test/8789/artifact/test-report.yml, for CCR even though the status is pass, can see some failed_test.

- name: cross-cluster-replication
    command: ./test.sh integ-test manifests/2.18.0/opensearch-2.18.0-test.yml --paths
      opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10395/linux/x64/tar
      --component cross-cluster-replication
    repository: https://github.com/opensearch-project/cross-cluster-replication.git
    configs:
      - name: with-security
        status: PASS
        yml: https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/cross-cluster-replication.yml
        test_stdout: https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/stdout.txt
        test_stderr: https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/stderr.txt
        cluster_stdout:
          - https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/local-cluster-logs/id-0/stdout.txt
        cluster_stderr:
          - https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/local-cluster-logs/id-0/stderr.txt
        failed_test:
          - org.opensearch.replication.integ.rest.ResumeReplicationIT.html#test that
            replication fails to resume when custom analyser is not present in follower
          - org.opensearch.replication.integ.rest.ResumeReplicationIT.html#test that
            replication resumes when custom analyser is overridden and present in
            follower
          - org.opensearch.replication.integ.rest.ResumeReplicationIT.html#test that
            replication resumes when custom analyser is present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            replication fails to start when custom analyser is not present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            replication starts successfully when custom analyser is overridden and
            present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            replication starts successfully when custom analyser is present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            snapshot on leader does not affect replication during bootstrap
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test follower
            index unblocked after stop replication
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test stop
            replication in restoring state while shards are starting
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test stop
            replication in restoring state with multiple shards
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test stop
            replication with stale replication settings at leader cluster

@peterzhuamazon
Copy link
Member Author

Coming from here Peter https://build.ci.opensearch.org/job/integ-test/8789/artifact/test-report.yml, for CCR even though the status is pass, can see some failed_test.

- name: cross-cluster-replication
    command: ./test.sh integ-test manifests/2.18.0/opensearch-2.18.0-test.yml --paths
      opensearch=https://ci.opensearch.org/ci/dbc/distribution-build-opensearch/2.18.0/10395/linux/x64/tar
      --component cross-cluster-replication
    repository: https://github.com/opensearch-project/cross-cluster-replication.git
    configs:
      - name: with-security
        status: PASS
        yml: https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/cross-cluster-replication.yml
        test_stdout: https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/stdout.txt
        test_stderr: https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/stderr.txt
        cluster_stdout:
          - https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/local-cluster-logs/id-0/stdout.txt
        cluster_stderr:
          - https://ci.opensearch.org/ci/dbc/integ-test/2.18.0/10395/linux/x64/tar/test-results/8789/integ-test/cross-cluster-replication/with-security/local-cluster-logs/id-0/stderr.txt
        failed_test:
          - org.opensearch.replication.integ.rest.ResumeReplicationIT.html#test that
            replication fails to resume when custom analyser is not present in follower
          - org.opensearch.replication.integ.rest.ResumeReplicationIT.html#test that
            replication resumes when custom analyser is overridden and present in
            follower
          - org.opensearch.replication.integ.rest.ResumeReplicationIT.html#test that
            replication resumes when custom analyser is present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            replication fails to start when custom analyser is not present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            replication starts successfully when custom analyser is overridden and
            present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            replication starts successfully when custom analyser is present in follower
          - org.opensearch.replication.integ.rest.StartReplicationIT.html#test that
            snapshot on leader does not affect replication during bootstrap
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test follower
            index unblocked after stop replication
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test stop
            replication in restoring state while shards are starting
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test stop
            replication in restoring state with multiple shards
          - org.opensearch.replication.integ.rest.StopReplicationIT.html#test stop
            replication with stale replication settings at leader cluster

My logic is not checking the status field but rather check index.html if it shows success or not.
Let me dig a bit into their files.

Thanks.

@peterzhuamazon
Copy link
Member Author

peterzhuamazon commented Oct 14, 2024 •
edited
Loading

I got it, those are ignored test not failed test. Even though the test are showing 100 percent success rate but for the non-ignored test.

I will update my logic to reflect that.

#5108

@peterzhuamazon peterzhuamazon mentioned this pull request Oct 14, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@prudhvigodithi prudhvigodithi prudhvigodithi approved these changes

@dblock dblock Awaiting requested review from dblock dblock is a code owner

@gaiksaya gaiksaya Awaiting requested review from gaiksaya gaiksaya is a code owner

@rishabh6788 rishabh6788 Awaiting requested review from rishabh6788 rishabh6788 is a code owner

@zelinh zelinh Awaiting requested review from zelinh zelinh is a code owner

@Divyaasm Divyaasm Awaiting requested review from Divyaasm Divyaasm is a code owner

@tianleh tianleh Awaiting requested review from tianleh tianleh is a code owner

Assignees

@peterzhuamazon peterzhuamazon

Labels
ci-test-automation cicd distinguished-contributor enhancement New Enhancement integtest jenkins Jenkins related issue test
Projects
Engineering Effectiveness Board
Status: ✅ Done
OpenSearch Engineering Effectiveness
Status: Done
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@peterzhuamazon @prudhvigodithi