Enhance StatisticMonitor with API support #4970

Deblintrake09 · 2024-02-16T15:36:20Z

Closes Refactor StatesMonitoring class to utilize API instead of .state files #4890

Description

This PR enhances the StatisticMonitor class to enable using the API for analysisd events and remoted stats.

New parameter use_api added called with -a and --use_api. It is used in with remoted to get remote data from the API.
analysis_events are recovered by default from the API.

Testing performed

OS	Package used
Ubuntu 22	4.8.0-beta1

Validation	Jenkins	Local	OS	Commit	Notes
Artifacts are generated	⚫⚫	local_artifacts.zip	Ubuntu22		Nothing to highlight

Rebits

Good jobs. Some changes are required

Rebits · 2024-02-21T11:21:36Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

+urllib3.disable_warnings()
+
+API_URL="https://localhost:55000"
+DAEMONS_ENDPOINT="/manager/daemons/stats?daemons_list=wazuh-analysisd,wazuh-remoted,wazuh-db"


Hardcoded daemons endpoint

Fixed in 8128e8a

Rebits · 2024-02-21T11:21:55Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

-import wazuh_testing.tools as tls
+urllib3.disable_warnings()
+
+API_URL="https://localhost:55000"


Hardcoded api url and port

Fixed in 8128e8a

Rebits · 2024-02-21T11:22:49Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

+
+
+


Suggested change

Fixed in 8128e8a

Rebits · 2024-02-21T11:24:04Z

deps/wazuh_testing/wazuh_testing/scripts/wazuh_statistics.py

@@ -26,13 +26,16 @@ def get_script_arguments():
                                     formatter_class=argparse.RawTextHelpFormatter)
    parser.add_argument('-t', '--target', dest='target_list', required=True, type=str, nargs='+', action='store',
                        help='Type the statistics target to collect separated by whitespace. '
-                             'Targets: agent, logcollector, remote and analysis.')
+                             'Targets: agent, logcollector, remote, analysis_events, analysisd_state and wazuhdb')


This could affect other QA process. Did it only affect test_cluster https://github.com/wazuh/wazuh-jenkins/pull/6276/files?

Rebits · 2024-02-21T11:24:36Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

@@ -40,44 +52,56 @@ class StatisticMonitor:
        dst_dir (str): directory to store the CSVs. Defaults to temp directory.
        csv_file (str): path to the CSV file.
        target (str): target file to monitor.
+        parse_json (bool): Determine if the file is a JSON file. Default False.


Suggested change

parse_json (bool): Determine if the file is a JSON file. Default False.

Rebits · 2024-02-21T11:29:46Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

+        data = daemons_response.json()['data']['affected_items']
+        self._write_csv(data, self.target, self.csv_file)
+
+


Suggested change

Rebits · 2024-02-21T11:32:12Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

+        if response.status_code != 200:
+            logging.info("Retrying get API data, status code {}".format(response.status_code))
+            return self._parse_api_data()
+
+        daemons_response = requests.get(API_URL + DAEMONS_ENDPOINT, verify=False, headers={'Authorization': 'Bearer ' + response.json()['data']['token']})
+        if daemons_response.status_code != 200:
+            logging.info("Retrying get API data, status code {}".format(response.status_code))
+            return self._parse_api_data()


Infinite loop. We should include some minor logic in order to prevent credential and others errors. In addition, logging should be error instead of info

fixed in 135c652

Rebits · 2024-02-21T11:32:45Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

+            timestamp = datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')
+
+            if self.use_api:
+


Suggested change

fixed in 135c652

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

Rebits · 2024-02-21T11:35:21Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

+                if target == "analysis_events":
+                    data = data[0]
+                elif target == "remote":
+                    data = data[1]
+                elif target == "wazuhdb":
+                    data = data[2]


Hardcoded elements. We should change the monitored process through the API

fixed in 135c652

Rebits · 2024-02-21T15:17:35Z

deps/wazuh_testing/wazuh_testing/scripts/wazuh_statistics.py

+    parser.add_argument('-u', '--use_state_file', action='store_true', default=False,
+                        help="Determine if the state files should be used to collect the for analysisd and remoted."
+                             "Use with remoted and analysis to get data from state files. Default False.")
+    parser.add_argument('-i', '--ip', dest='ip', action='store', default='localhost',
+                        help=f"IP for the API. Default localhost.")
+    parser.add_argument('-p', '--port', dest='port', action='store', default='55000',
+                        help=f"port for the API. Default localhost.")


Suggested change

parser.add_argument('-u', '--use_state_file', action='store_true', default=False,

help="Determine if the state files should be used to collect the for analysisd and remoted."

"Use with remoted and analysis to get data from state files. Default False.")

parser.add_argument('-i', '--ip', dest='ip', action='store', default='localhost',

help=f"IP for the API. Default localhost.")

parser.add_argument('-p', '--port', dest='port', action='store', default='55000',

help=f"port for the API. Default localhost.")

parser.add_argument('-u', '--use_state_file', action='store_true', default=False,

help="Use state files for analysis and remote operations."

"When used with 'remote' and 'analysis', data will be collected from state files; otherwise, the API will be used. Default False.")

parser.add_argument('-i', '--ip', dest='ip', action='store', default='localhost',

help=f"Specify the IP address for the API. Default is 'localhost'.")

parser.add_argument('-p', '--port', dest='port', action='store', default='55000',

help=f"Specify the port for the API. Default is '55000'.")

Fixed in a9192a5

Rebits · 2024-02-21T15:20:53Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

        else:
            raise ValueError(f'The target {self.target} is not a valid one.')

-        state_file = splitext(basename(self.statistics_file))[0]
-        self.csv_file = join(self.dst_dir, f'{state_file}_stats.csv')
+        if self.use_state_file == True:


Suggested change

if self.use_state_file == True:

if self.use_state_file:

Fixed in a1d0448

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

Rebits · 2024-02-21T15:27:20Z

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

+        if target == "analysis":
+            csv_header = headers.analysisd_header if self.use_state_file == True else headers.analysisd_events_header
+        elif target == "logcollector":
+            csv_header = headers.logcollector_header
+        elif target == "remote":
+            csv_header = headers.remoted_header if self.use_state_file == True  else headers.remoted_api_header
+        elif target == "wazuhdb":
+            csv_header = headers.wazuhdb_header
+        else:
+            csv_header = headers.agentd_header


Suggested change

if target == "analysis":

csv_header = headers.analysisd_header if self.use_state_file == True else headers.analysisd_events_header

elif target == "logcollector":

csv_header = headers.logcollector_header

elif target == "remote":

csv_header = headers.remoted_header if self.use_state_file == True else headers.remoted_api_header

elif target == "wazuhdb":

csv_header = headers.wazuhdb_header

else:

csv_header = headers.agentd_header

if target == "analysis":

csv_header = headers.analysisd_header if self.use_state_file else headers.analysisd_events_header

elif target == "logcollector":

csv_header = headers.logcollector_header

elif target == "remote":

csv_header = headers.remoted_header if self.use_state_file else headers.remoted_api_header

elif target == "wazuhdb":

csv_header = headers.wazuhdb_header

else:

csv_header = headers.agentd_header

Fixed in b48cd5b

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py

Deblintrake09 added 2 commits February 16, 2024 16:34

add params to wazuh_statistics

a404dee

add api support to statistic.py

197f6e5

Deblintrake09 requested a review from Rebits February 16, 2024 15:36

Deblintrake09 self-assigned this Feb 16, 2024

Deblintrake09 linked an issue Feb 16, 2024 that may be closed by this pull request

Refactor StatesMonitoring class to utilize API instead of .state files #4890

Closed

3 tasks

Deblintrake09 added 6 commits February 16, 2024 17:33

move statistic headers to file

cc403ec

add wazuhdb header

3712d7c

add missing headers

a1d4008

add wazuhdb parsing

3486a9f

change remoted parameter to remote

2ac54a6

fix target argument documentation

e128396

Selutario mentioned this pull request Feb 21, 2024

Compare wazuh-db stats in loaded env wazuh/wazuh#21937

Closed

4 tasks

enhance documentation

b0ce46f

Rebits requested changes Feb 21, 2024

View reviewed changes

Deblintrake09 added 6 commits February 21, 2024 13:27

add port and ip for API

8128e8a

add port and IP for API

135c652

fix documentation

15d80c8

replace use_api parameter for use_state_file

0da9d00

fix file naming

17aac75

fix undo name change

68a79c1

Rebits requested changes Feb 21, 2024

View reviewed changes

deps/wazuh_testing/wazuh_testing/tools/performance/statistic.py Show resolved Hide resolved

Deblintrake09 added 5 commits February 21, 2024 17:15

enhance wazuh-statistic docu

a9192a5

add wait for complete_to_api endpoint

a1d0448

remove unnecesary checks

b48cd5b

fix typo

edac452

fix query typo

56c798a

Deblintrake09 requested a review from Rebits February 21, 2024 16:53

davidjiglesias merged commit 2702be3 into 4.8.0 Feb 22, 2024
2 of 4 checks passed

davidjiglesias deleted the enhancement/4890-wazuh-statistics-scripts branch February 22, 2024 07:57

GGP1 mentioned this pull request Feb 23, 2024

Update the data visualization script to support API statistics files #5022

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance StatisticMonitor with API support #4970

Enhance StatisticMonitor with API support #4970

Deblintrake09 commented Feb 16, 2024

Rebits left a comment

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Rebits Feb 21, 2024

Rebits Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

Rebits Feb 21, 2024

Deblintrake09 Feb 21, 2024

		data = daemons_response.json()['data']['affected_items']
		self._write_csv(data, self.target, self.csv_file)

		timestamp = datetime.fromtimestamp(time()).strftime('%Y-%m-%d %H:%M:%S')

		if self.use_api:

Enhance StatisticMonitor with API support #4970

Enhance StatisticMonitor with API support #4970

Conversation

Deblintrake09 commented Feb 16, 2024

Description

Testing performed

Rebits left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment