[#103][#117] Introduce optional "verbose" test failure reporting and increase test coverage

[calebevans]

Verbose Test Failure Reporting

This pull request introduces a new option in the firewatch CLI: --verbose-test-failure-reporting (and --verbose-test-failure-reporting-ticket-limit). The new option allows users to optionally report on every test failure found in job (up to the limit defined in --verbose-test-failure-reporting-ticket-limit which is 10 by default).

When this functionality is enabled, it also changes the behavior of the duplicate bug search functionality by further specifying the specific test that has failed when searching. This same behavior is observed in the "past bugs" table in the issue description, so all past bugs in the table are relevant to that test name.

This was requested by the RHTAP team and may allow them to start using the tool in their OpenShift CI pipeline.

Example of a bug created using this functionality:

Test Coverage

Not much here, other than increased coverage. We are now at 72.89% coverage. We were previously at 58.69% coverage.

[calebevans]

/retest

[redhat-qe-bot]

retest requires an argument

[redhat-qe-bot]

Report bugs in Issues

The following are automatically added:

• Add reviewers from OWNER file (in the root of the repository) under reviewers section.
• Set PR size label.
• New issue is created for the PR. (Closed when PR is merged/closed)
• Run pre-commit if .pre-commit-config.yaml exists in the repo.

Available user actions:

• To mark PR as WIP comment /wip to the PR, To remove it from the PR comment /wip cancel to the PR.
• To block merging of PR comment /hold, To un-block merging of PR comment /hold cancel.
• To mark PR as verified comment /verified to the PR, to un-verify comment /verified cancel to the PR.
  verified label removed on each new commit push.
• To cherry pick a merged PR comment /cherry-pick <target branch to cherry-pick to> in the PR.
  • Multiple target branches can be cherry-picked, separated by spaces. (/cherry-pick branch1 branch2)
  • Cherry-pick will be started when PR is merged
• To build and push container image command /build-and-push-container in the PR (tag will be the PR number).
• To add a label by comment use /<label name>, to remove, use /<label name> cancel

Supported /retest check runs

• /retest tox: Retest tox
• /retest sonarqube: Retest sonarqube

Supported labels

• hold
• verified
• wip
• lgtm

[calebevans]

@myakove I cannot see the tox failures and cannot acknowledge the sonarqube issues. Should we just move the tox execution to a GitHub action?

[myakove]

@myakove I cannot see the tox failures and cannot acknowledge the sonarqube issues. Should we just move the tox execution to a GitHub action?

Sure, I will remove firewatch from the webhook configuration tomorrow.

[myakove]

/retest sonarqube

[openshift-ci]

@myakove: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test image-images

Use /test all to run all jobs.

In response to this:

/retest sonarqube

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[myakove]

/retest tox

[openshift-ci]

@myakove: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test image-images

Use /test all to run all jobs.

In response to this:

/retest tox

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[myakove]

/retest tox

[openshift-ci]

@myakove: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test image-images

Use /test all to run all jobs.

In response to this:

/retest tox

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[myakove]

/retest sonarqube

[openshift-ci]

@myakove: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test image-images

Use /test all to run all jobs.

In response to this:

/retest sonarqube

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[myakove]

/retest tox

[openshift-ci]

@myakove: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test image-images

Use /test all to run all jobs.

In response to this:

/retest tox

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[redhat-qe-bot]

No tox configured for this repository

[myakove]

/retest sonarqube

[openshift-ci]

@myakove: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test image-images

Use /test all to run all jobs.

In response to this:

/retest sonarqube

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[calebevans]

/verified

[calebevans]

/verified

[Simranpal]

Example of a bug created using this functionality:
I cannot access the ticket, got access denied

[calebevans]

@Simranpal yeah that is in the stage environment, you'll have to set up some proxies. I'll ping you in slack with more info

[calebevans]

/verified

[calebevans]

/retest can-be-merged

[openshift-ci]

@calebevans: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

• /test image-images

Use /test all to run all jobs.

In response to this:

/retest can-be-merged

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.