Quality Intelligence

testsuite/.rubocop.yml

@@ -138,7 +138,7 @@ Style/TopLevelMethodDefinition:
 Style/Copyright:
   Enabled: true
   AutocorrectNotice: |
-    # Copyright (c) 2023 SUSE LLC.
+    # Copyright (c) 2024 SUSE LLC.
     # Licensed under the terms of the MIT license.
   Notice: '.*Copyright.*SUSE LLC.*'
 

testsuite/Gemfile

@@ -21,6 +21,7 @@ gem 'nokogiri', '~> 1.16'
 gem 'parallel', '~> 1.26'
 gem 'parallel_tests', '~> 4.7'
 gem 'pg', '~> 1.5'
+gem 'prometheus-client', '~> 4.2.3'
 gem 'public_suffix', '~> 6.0'
 gem 'rack', '~> 3.1'
 gem 'rack-test', '~> 2.1'

testsuite/documentation/api.md

@@ -32,3 +32,8 @@ just comment out the API you don't use in this script.
 
 Place the script on the controller, in `spacewalk/testsuite/features/support`,
 make it executable with `chmod +x myscript.rb`, and run it with `./myscript.rb`.
+
+## Choose the API protocol
+
+You can choose the API protocol by setting the `API_PROTOCOL` environment variable. 
+You can set it to `http` to use the HTTP API or `xmlrpc` to use the XML-RPC API.

testsuite/documentation/code-coverage.md

@@ -1,19 +1,39 @@
-# How to check code coverage results in the test suite
+# Code Coverage of the server components after a full test suite execution
 
-## Get the log
+In your test suite you must configure JaCoCo, connected to the java components and run the full test suite, to see which blocks of code are triggered.
 
-* log in to the controller as root;
-* go to `spacewalk/testsuite` directory;
-* copy recursively `coverage` directory to some web server or local directory;
-* open that directory in a web browser.
+## Goals
 
-## Analyze
+- Find features not fully covered through our test-suite
+- Discover obsolete code (methods or classes), which are never triggered in any workflow.
+- Generate a map of features and files(or even piece of code that cover), with the idea of doing reverse engineering and be able to know which Cucumber feature can be broken when we merge a PR.
 
-***IMPORTANT***:
-if you have a lot of failures, the coverage will decrease because the steps were not executed. Ideally, you need to have the test suite results green to evaluate the code coverage correctly and find dead code.
+## How to configure JaCoCo in your java server
 
-Code coverage `result.html` is a great tool, **but** needs a human interpretation of the results.
+Start the app with Jacoco Agent enabled on TCP port 6300:
+For that we need to edit the file `/etc/sysconfig/tomcat` on the server appending
+`-javaagent:/tmp/jacocoagent.jar=output=tcpserver,address=*,port=6300`
+to `JAVA_OPTS` variable, then restart tomcat: `systemctl restart tomcat.service`
 
-## Submit a PR
+## How to run the test suite with JaCoCo
 
-This PR will eliminate dead code or provide a better fix that you had inspired by code coverage.
+1. Run the test suite normally, but with the JaCoCo agent enabled
+    ```bash
+    java -javaagent:/tmp/jacocoagent.jar=output=tcpserver,address=*,port=6300 -jar /tmp/cucumber.jar
+    ```
+2. Then when a test finish we can use JaCoCo CLI to dump results, let's run this from where we have the product code
+    ```bash
+    java -jar jacococli.jar dump --address <server_fqdn> --destfile /tmp/jacoco.exec --port 6300 --reset
+    ```
+3. After that, we can generate a HTML report
+    ```bash
+    java -jar jacococli.jar report /tmp/jacoco.exec --html /srv/www/htdocs/pub/jacoco-cucumber-report --xml /srv/www/htdocs/pub/jacoco-cucumber-report.xml --sourcefiles /tmp/uyuni-master/java/code/src --classfiles /srv/tomcat/webapps/rhn/WEB-INF/lib
+    ```
+4. From the XML report we want to obtain a list of (source_file_path:line_number) for each line of code triggered during a Cucumber feature execution.
+    ```
+    package name
+    + sourcefile name
+        + line nr (if mi == 0)
+    ```
+5. We will have a HashMap stored in Redis handled by KeyValueStore class
+6. On that HashMap, the key being the Cucumber Feature filepath and the value being a list of (source_file_path:line_number) for each line of code triggered during the execution of that feature.

testsuite/documentation/software-components.md

@@ -34,6 +34,25 @@ to run lists of Cucumber features stored in YAML format
 * [`net-ssh`] is a Ruby library that allows to communicate with remote machines via SSH
 * [`net-scp`] is a Ruby library that allows to copy files to and from remote machines via SCP
 
+### NoSQL Database interaction
+
+* [`keyvalue_store`] is a custom Ruby class that allows to interact with a No-SQL database. So you can store a Map of key-value pairs in the database.
+
+### Metrics Collection
+
+* [`metrics_collector_handler`] is a custom Ruby class that allows to push metrics from the system tested to a Metrics Collector.
+  As default this handler is configured to connect to a Prometheus Push Gateway instance located in `nsa.mgr.suse.de:9091`, if you want to change it you need to set the `PROMETHEUS_PUSH_GATEWAY_URL` environment variable.
+
+### Code coverage
+
+* [`code_coverage`] is a custom Ruby class that allows to collect code coverage data from the server components while running our tests. It use Key-Value Store to store the data.
+  In order to enable this feature, you need to set the `CODE_COVERAGE` environment variable to `true`. Additionally, you need to set these environment variables to configure the Key-Value Store: `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, `REDIS_PASSWORD`
+
+### Quality Intelligence
+
+* [`quality_intelligence`] is a custom Ruby class that make use of `MetricsCollectorHandler` in order to monitor fitness functions in the test suite, like for example the time taken to bootstrap or onboard a minion. But it could be extend by embedding a `KeyValueStore` to store other QI data to process.
+In order to enable this feature, you need to set the `QUALITY_INTELLIGENCE` environment variable to `true`.
+
 ### Simulation of user interaction
 
 * [`capybara`](https://github.com/teamcapybara/capybara) simulates user interaction with a web interface.

testsuite/features/init_clients/sle_minion.feature

@@ -18,6 +18,7 @@ Feature: Bootstrap a Salt minion via the GUI
     And I select the hostname of "proxy" from "proxies" if present
     And I click on "Bootstrap"
     And I wait until I see "Bootstrap process initiated." text
+    And I report the bootstrap duration for "sle_minion"
 
   Scenario: Check the new bootstrapped minion in System List page
     When I follow the left menu "Salt > Keys"
@@ -27,6 +28,7 @@ Feature: Bootstrap a Salt minion via the GUI
     And I wait until I see the name of "sle_minion", refreshing the page
     And I wait until onboarding is completed for "sle_minion"
     Then the Salt master can reach "sle_minion"
+    And I report the onboarding duration for "sle_minion"
 
 @susemanager
   Scenario: Use correct kernel image on the SLES minion

testsuite/features/reposync/srv_sync_products.feature

@@ -68,6 +68,7 @@ Feature: Synchronize products in the products page of the Setup Wizard
     And I wait until I see "SUSE Linux Enterprise Server 15 SP4 x86_64" product has been added
     Then the SLE15 SP4 product should be added
     When I wait until all synchronized channels for "sles15-sp4" have finished
+    And I report the synchronization duration for "sles15-sp4"
 
 @scc_credentials
 @uyuni
@@ -100,6 +101,7 @@ Feature: Synchronize products in the products page of the Setup Wizard
     Then the SLE15 SP4 product should be added
     When I use spacewalk-common-channel to add channel "sles15-sp4-devel-uyuni-client" with arch "x86_64"
     And I wait until all synchronized channels for "sles15-sp4" have finished
+    And I report the synchronization duration for "sles15-sp4"
     # TODO: Refactor the scenarios in order to not require a full synchronization of SLES 15 SP4 product in Uyuni
     # When I kill running spacewalk-repo-sync for "sles15-sp4"
 

testsuite/features/step_definitions/system_monitoring_steps.rb

@@ -0,0 +1,25 @@
+# Copyright (c) 2024 SUSE LLC.
+# Licensed under the terms of the MIT license.
+
+### This file contains the definitions for the steps extracting and reporting information from the system
+
+When(/^I report the bootstrap duration for "([^"]*)"$/) do |host|
+  next unless $quality_intelligence_mode
+
+  duration = last_bootstrap_duration(host)
+  $quality_intelligence.push_bootstrap_duration(host, duration)
+end
+
+When(/^I report the onboarding duration for "([^"]*)"$/) do |host|
+  next unless $quality_intelligence_mode
+
+  duration = last_onboarding_duration(host)
+  $quality_intelligence.push_onboarding_duration(host, duration)
+end
+
+When(/^I report the synchronization duration for "([^"]*)"$/) do |product|
+  next unless $quality_intelligence_mode
+
+  duration = synchronization_duration(product)
+  $quality_intelligence.push_synchronization_duration(product, duration)
+end

testsuite/features/support/code_coverage.rb

@@ -1,20 +1,16 @@
-# Copyright (c) 2016-2023 SUSE LLC.
+# Copyright (c) 2024 SUSE LLC.
 # Licensed under the terms of the MIT license.
-require 'redis'
+require_relative 'keyvalue_store'
 require 'nokogiri'
 require 'open-uri'
 
 # CodeCoverage handler to produce, parse and report Code Coverage from the Jave Server to our GitHub PRs
 class CodeCoverage
-  include(Nokogiri::XML)
-  # Initialize a connection with a Redis database
-  def initialize(redis_host, redis_port, redis_username, redis_password)
-    @database = Redis.new(host: redis_host, port: redis_port, username: redis_username, password: redis_password)
-  end
+  include Nokogiri::XML
 
-  # Close the connection with the Redis database
-  def close
-    @database.close
+  # Initialize the CodeCoverage handler
+  def initialize
+    @keyvalue_store = KeyValueStore.new(ENV.fetch('REDIS_HOST', nil), ENV.fetch('REDIS_PORT', nil), ENV.fetch('REDIS_USERNAME', nil), ENV.fetch('REDIS_PASSWORD', nil))
   end
 
   # Parse a JaCoCo XML report, extracting information that will be included in a Set on a Redis database
@@ -36,17 +32,12 @@ def push_feature_coverage(feature_name)
 
           next unless Integer(counter_class.attr('covered').to_s).positive?
 
-          begin
-            @database.sadd("#{package_name}/#{sourcefile_name}", feature_name)
-          rescue StandardError => e
-            warn("#{e.backtrace} > #{package_name}/#{sourcefile_name} : #{feature_name}")
-          end
+          @keyvalue_store.add("#{package_name}/#{sourcefile_name}", feature_name)
         end
       end
     rescue StandardError => e
       warn(e.backtrace)
-    ensure
-      File.delete(filename)
+    ensure File.delete(filename)
     end
   end
 

testsuite/features/support/env.rb

@@ -17,6 +17,7 @@
 require 'set'
 require 'timeout'
 require_relative 'code_coverage'
+require_relative 'quality_intelligence'
 require_relative 'remote_nodes_env'
 require_relative 'commonlib'
 
@@ -29,10 +30,14 @@
   $debug_mode = true
   $stdout.puts('DEBUG MODE ENABLED.')
 end
-if ENV['REDIS_HOST']
+if ENV['REDIS_HOST'] && ENV.fetch('CODE_COVERAGE', false)
   $code_coverage_mode = true
   $stdout.puts('CODE COVERAGE MODE ENABLED.')
 end
+if ENV.fetch('QUALITY_INTELLIGENCE', false)
+  $quality_intelligence_mode = true
+  $stdout.puts('QUALITY INTELLIGENCE MODE ENABLED.')
+end
 
 # Context per feature
 $context = {}
@@ -117,7 +122,10 @@ def capybara_register_driver
 $api_test = new_api_client
 
 # Init CodeCoverage Handler
-$code_coverage = CodeCoverage.new(ENV.fetch('REDIS_HOST', nil), ENV.fetch('REDIS_PORT', nil), ENV.fetch('REDIS_USERNAME', nil), ENV.fetch('REDIS_PASSWORD', nil)) if $code_coverage_mode
+$code_coverage = CodeCoverage.new if $code_coverage_mode
+
+# Init Quality Intelligence Handler
+$quality_intelligence = QualityIntelligence.new if $quality_intelligence_mode
 
 # Define the current feature scope
 Before do |scenario|
@@ -139,7 +147,7 @@ def capybara_register_driver
       print_server_logs
     end
   end
-  page.instance_variable_set(:@touched, false)
+  page.instance_variable_set(:@touched, false) if Capybara::Session.instance_created?
 end
 
 # Test is web session is open

testsuite/features/support/keyvalue_store.rb

@@ -0,0 +1,78 @@
+# Copyright (c) 2024 SUSE LLC.
+# Licensed under the terms of the MIT license.
+require 'redis'
+
+# Key-Value Store to interact with a NoSQL database
+class KeyValueStore
+  # Initialize a connection with a NoSQL database
+  #
+  # @param db_host [String] The hostname of the NoSQL database.
+  # @param db_port [Integer] The port of the NoSQL database.
+  # @param db_username [String] The username to authenticate with the NoSQL database.
+  # @param db_password [String] The password to authenticate with the NoSQL database.
+  # @return [Redis] A connection with the database.
+  def initialize(db_host, db_port, db_username, db_password)
+    begin
+      raise ArgumentError, 'Database host is required' if db_host.nil? || db_host.empty?
+      raise ArgumentError, 'Database port is required' if db_port.nil?
+      raise ArgumentError, 'Database username is required' if db_username.nil? || db_username.empty?
+      raise ArgumentError, 'Database password is required' if db_password.nil? || db_password.empty?
+
+      @database = Redis.new(host: db_host, port: db_port, username: db_username, password: db_password)
+    rescue StandardError => e
+      warn("Error initializing KeyValueStore:\n #{e.full_message}")
+      raise
+    end
+  end
+
+  # Close the connection with the NoSQL database
+  def close
+    @database.close
+  end
+
+  # Add a key-value pair to a Set
+  #
+  # @param key [String] The key to add the value to.
+  # @param value [String] The value to add to the key.
+  # @param database [Integer] Optional: The database number to select (default: 0).
+  def add(key, value, database = 0)
+    begin
+      @database.select(database)
+      @database.sadd(key, value)
+    rescue StandardError => e
+      warn("Error adding a key-value:\n #{e.full_message}")
+      raise
+    end
+  end
+
+  # Get the value of a key
+  #
+  # @param key [String] The key to get the value from.
+  # @param database [Integer] Optional: The database number to select (default: 0).
+  # @return [Array<String>] An array with the values of the key.
+  def get(key, database = 0)
+    begin
+      @database.select(database)
+      @database.smembers(key)
+    rescue StandardError => e
+      warn("Error getting a key-value:\n #{e.full_message}")
+      raise
+    end
+  end
+
+  # Remove a key-value pair from a Set
+  #
+  # @param key [String] The key to remove the value from.
+  # @param value [String] The value to remove from the key.
+  # @param database [Integer] Optional: The database number to select (default: 0).
+  # @return [Integer] The number of members that were successfully removed
+  def remove(key, value, database = 0)
+    begin
+      @database.select(database)
+      @database.srem(key, value)
+    rescue StandardError => e
+      warn("Error removing a key-value:\n #{e.full_message}")
+      raise
+    end
+  end
+end

testsuite/features/support/metrics_collector_handler.rb

@@ -0,0 +1,34 @@
+# Copyright (c) 2024 SUSE LLC.
+# Licensed under the terms of the MIT license.
+
+require 'net/http'
+require 'prometheus/client'
+require 'prometheus/client/push'
+require 'uri'
+
+# Metrics Collector handler to push metrics to the Metrics Collector
+class MetricsCollectorHandler
+  def initialize(metrics_collector_url)
+    @metrics_collector_url = metrics_collector_url
+    @metrics_collector = Prometheus::Client::Registry.new
+  end
+
+  # Push a metric to the Metrics Collector, raising an error if the Metrics Collector request fails.
+  #
+  # @param job_name [String] the job name to push the metric to
+  # @param metric_name [String] the metric name to push
+  # @param metric_value [Integer] the metric value to push
+  # @param labels [Hash] the labels to add to the metric
+  # @return [void]
+  # @raise [SystemCallError] if the Prometheus request fails
+  def push_metrics(job_name, metric_name, metric_value, labels = {})
+    begin
+      gauge = @metrics_collector.get(metric_name.to_sym) || @metrics_collector.gauge(metric_name.to_sym, docstring: job_name, labels: labels.keys)
+      gauge.set(metric_value, labels: labels)
+      Prometheus::Client::Push.new(job: job_name, gateway: @metrics_collector_url).add(@metrics_collector)
+    rescue StandardError => e
+      warn("Error pushing the metric #{metric_name} with value #{metric_value}:\n#{e.full_message}")
+      raise
+    end
+  end
+end

testsuite/features/support/namespaces/system.rb

@@ -183,6 +183,25 @@ def get_system_errata(system_id)
   def get_systems_errata(system_ids)
     @test.call('system.getRelevantErrata', sessionKey: @test.token, sids: system_ids)
   end
+
+  # Returns the event history for a system.
+  #
+  # @param system_id [String] The ID of the system.
+  # @param offset [Integer] Number of results to skip
+  # @param limit [Integer] Maximum number of results
+  # @return [Array<Hash>] An array of events
+  def get_event_history(system_id, offset, limit)
+    @test.call('system.getEventHistory', sessionKey: @test.token, sid: system_id, offset: offset, limit: limit)
+  end
+
+  # Returns the event details for a system.
+  #
+  # @param system_id [String] The ID of the system.
+  # @param event_id [String] The ID of the event.
+  # @return [Hash] The event details
+  def get_event_details(system_id, event_id)
+    @test.call('system.getEventDetails', sessionKey: @test.token, sid: system_id, eid: event_id)
+  end
 end
 
 # System Configuration namespace