[#103][#117] Introduce optional "verbose" test failure reporting and …

…increase test coverage (#120)

* Add support for verbose failure reporting

* Remove value from env list

* Reduce duplication

* Linting

* Add a new tox GH action

* Remove tox setup

* Fix poetry config

.pre-commit-config.yaml

@@ -1,3 +1,5 @@
+ci:
+    autofix_prs: false
 repos:
 - repo: https://github.com/pre-commit/pre-commit-hooks
   rev: v4.5.0

README.md

@@ -16,14 +16,19 @@ Thank you for your interest in the firewatch project! Please find some informati
     - This can be used as a "best-guess" for what may have gone wrong during the job's run.
   - Firewatch will search for any past issues created by a specific step that has failed in the same way (pod failure or test failure) and list the 10 most recent ones.
     - This is meant to help the engineer working on the bug to find a solution.
+    - If verbose test failure reporting is enabled, this search is refined further to only search for issues with the same test failure.
   - Firewatch searches for duplicate issues and makes a comment on the issues rather than filing a second issue.
     - This is done using the labels on issues created by firewatch. These labels should consist of the failure type, failed step, and the job name.
     - If the new failures matches the failure type, failed step, and job name; firewatch will make a comment notifying the user that this failure likely happened again.
+    - If verbose test failure reporting is enabled, this search is refined further to only search for issues with the same test failure.
   - If no failures are found, firewatch will search for any open issues on the Jira server provided and add a comment to the issue mentioning that the job has passed since that issue was filed.
     - This functionality also uses the labels on issues created by firewatch.
     - **Note:** If you get a notification on an issue, but would like to continue working on the issue without getting notifications, add the `ignore-passing-notification` label to the issue.
-  - You can optionally report successful runs by defining the `"success_rules"` section in the config.
+  - Firewatch can optionally be configured to report successful runs by defining the `"success_rules"` section in the config.
     - For each rule in this section, a Jira story will be created (with status `closed`) reporting the job success.
+  - Firewatch can optionally be configured to report test failures in a more verbose way (verbose test failure reporting).
+    - When configured to do this, firewatch will report on EVERY test failure in all JUnit files. The issues created from this will specify the failed test name in the title and description.
+    - **This functionality has the potential to create A LOT of tickets if cascading failures occur.** Because of this, firewatch is configured by default to only report up to 10 test failures per run. This value can be overridden, but do so with caution.
 
 ## Getting Started
 
@@ -80,6 +85,17 @@ Remember, when you are using the `firewatch-report-issues` ref, some variables n
   - BEHAVIOR:
     - `"false"`: The `firewatch-report-issues` step will not fail with a non-zero exit code when test failures are found.
     - `"true"`: The `firewatch-report-issues` step will fail with a non-zero exit code when test failures are found.
+- `FIREWATCH_VERBOSE_TEST_FAILURE_REPORTING`
+  - A variable that will determine if firewatch will report on every test failure in all JUnit files (up to the limit defined in `$FIREWATCH_VERBOSE_TEST_FAILURE_REPORTING_LIMIT`).
+  - DEFAULT: `"false"`
+  - BEHAVIOR:
+    - `"false"`: Firewatch will only report on the first test failure found in a JUnit file.
+    - `"true"`: Firewatch will report on every test failure found in a JUnit file.
+- `FIREWATCH_VERBOSE_TEST_FAILURE_REPORTING_LIMIT`
+  - The limit of test failures to report on when verbose test failure reporting is enabled.
+  - DEFAULT: `10`
+  - BEHAVIOR:
+    - If verbose test failure reporting is enabled, firewatch will only report on the first `$FIREWATCH_VERBOSE_TEST_FAILURE_REPORTING_LIMIT` test failures found in a JUnit file.
 
 **OPTIONAL DEFAULT VARIABLES:**
 

cli/jira_config_gen/jira_config_gen.py

@@ -101,6 +101,6 @@ def render_template(
         with open(output_file, "w") as file:
             file.write(rendered_template)
 
-        self.logger.success(f"Config file written to {output_file}")
+        self.logger.info(f"Config file written to {output_file}")
 
         return output_file

cli/objects/configuration.py

@@ -33,6 +33,8 @@ def __init__(
         jira: Jira,
         fail_with_test_failures: bool,
         keep_job_dir: bool,
+        verbose_test_failure_reporting: bool,
+        verbose_test_failure_reporting_ticket_limit: Optional[int] = 10,
         config_file_path: Union[str, None] = None,
     ):
         """
@@ -42,7 +44,8 @@ def __init__(
             jira (Jira): A Jira object used to log in and interact with Jira
             fail_with_test_failures (bool): If a test failure is found, after bugs are filed, firewatch will exit with a non-zero exit code
             keep_job_dir (bool): If true, firewatch will not delete the job directory (/tmp/12345) that is created to hold logs and results for a job following execution.
-            report_success (bool): If true, firewatch will create a Jira story reporting the success. The story will be closed immediately.
+            verbose_test_failure_reporting (bool): If true, firewatch will report all test failures found in the job.
+            verbose_test_failure_reporting_ticket_limit (Optional[int]): Used as a safeguard to prevent firewatch from filing too many bugs. If verbose_test_reporting is set to true, this value will be used to limit the number of bugs filed. Defaults to 10.
             config_file_path (Union[str, None], optional): The firewatch config can be stored in a file or an environment var. Defaults to None.
         """
         self.logger = get_logger(__name__)
@@ -59,6 +62,12 @@ def __init__(
         # Boolean value to decide if firewatch should delete the job directory following execution.
         self.keep_job_dir = keep_job_dir
 
+        # Boolean value to decide if firewatch should report all test failures found in the job.
+        self.verbose_test_failure_reporting = verbose_test_failure_reporting
+        self.verbose_test_failure_reporting_ticket_limit = (
+            verbose_test_failure_reporting_ticket_limit
+        )
+
         # Get the config data
         self.config_data = self._get_config_data(config_file_path=config_file_path)
 
@@ -137,24 +146,6 @@ def _get_default_jira_project(self) -> str:
             )
             exit(1)
 
-    def _get_default_jira_epic(self) -> Optional[str]:
-        """
-        Gets the default Jira epic from the $FIREWATCH_DEFAULT_JIRA_EPIC environment variables and validates that it is a string.
-
-        Returns:
-            Optional[str]: The default Jira project name defined in environment variable.
-        """
-        default_epic = os.getenv("FIREWATCH_DEFAULT_JIRA_EPIC")
-
-        # Verify that value is a string if it exists, return
-        if isinstance(default_epic, str) or not default_epic:
-            return default_epic
-
-        self.logger.error(
-            f'Value for "$FIREWATCH_DEFAULT_JIRA_EPIC" is not a string: "{default_epic}"',
-        )
-        exit(1)
-
     def _get_config_data(self, config_file_path: Optional[str]) -> dict[Any, Any]:
         """
         Gets the config data from either a configuration file or from the FIREWATCH_CONFIG environment variable.

cli/objects/failure.py

@@ -14,38 +14,37 @@
 #    You should have received a copy of the GNU General Public License
 #    along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
+from typing import Optional
+
 from simple_logger.logger import get_logger
 
 
 class Failure:
-    def __init__(self, failed_step: str, failure_type: str):
+    def __init__(
+        self,
+        failed_step: str,
+        failure_type: str,
+        failed_test_name: Optional[str] = None,
+        failed_test_junit_path: Optional[str] = None,
+    ):
         """
         Initializes the Failure object.
 
         Args:
             failed_step (str): The failed step
             failure_type (str): The failure type
+            failed_test_name (Optional[str], optional): The failed test name. Defaults to None.
+            failed_test_junit_path (Optional[str], optional): The path to the failed test's junit file. Defaults to None.
         """
         self.logger = get_logger(__name__)
 
-        self.step = self._get_failed_step(failed_step)
+        self.step = failed_step
         self.failure_type = self._get_failure_type(failure_type)
 
-    def _get_failed_step(self, failed_step: str) -> str:
-        """
-        Gets the name of the failed step. Used to validate the value provided.
-
-        Args:
-            failed_step (str): Failed step name.
-
-        Returns:
-            str: String value representing name of step failed
-        """
-        if isinstance(failed_step, str):
-            return failed_step
-        else:
-            self.logger.error("Failed step must be a string value")
-            exit(1)
+        # if the failure is a test failure, set additional attributes
+        if self.failure_type == "test_failure":
+            self.failed_test_name = failed_test_name
+            self.failed_test_junit_path = failed_test_junit_path
 
     def _get_failure_type(self, failure_type: str) -> str:
         """

cli/objects/jira_base.py

@@ -21,7 +21,6 @@
 from jira import Issue
 from jira import JIRA
 from jira.exceptions import JIRAError
-from jira.resources import User
 from simple_logger.logger import get_logger