diff --git a/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/forwarder_monitoring.png b/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/forwarder_monitoring.png new file mode 100644 index 0000000000..c5bc1b8498 Binary files /dev/null and b/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/forwarder_monitoring.png differ diff --git a/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/forwarder_monitoring_2.png b/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/forwarder_monitoring_2.png new file mode 100644 index 0000000000..a30f86784b Binary files /dev/null and b/docs/assets/operation_center/ingestion_methods/sekoiaio_forwarder/forwarder_monitoring_2.png differ diff --git a/docs/integration/categories/applicative/sekoiaio_forwarder_logs.md b/docs/integration/categories/applicative/sekoiaio_forwarder_logs.md new file mode 100644 index 0000000000..d11927d257 --- /dev/null +++ b/docs/integration/categories/applicative/sekoiaio_forwarder_logs.md @@ -0,0 +1,26 @@ +uuid: 915a119c-2ec8-4482-a3c6-4d4cae62b671 +name: Sekoia.io forwarder logs +type: intake + +## Overview +- **Vendor**: Sekoia +- **Plan**: Defend Core & Defend Prime +- **Detection based on**: Audit +- **Supported application or feature**: +Sekoia.io forwarder logs collect all statictics coming from Sekoia forwarder instances. It helps to monitor the forwarder health: + + - resource usage + - queue size + - number of messages received by the forwarder + - number of messages sent by the forwarder + +## Configure + +To monitor forwarder health, create a new intake `Sekoia.io forwarer logs` in your community. Once the intake is enabled, please follow [this documentation](/integration/ingestion_methods/syslog/sekoiaio_forwarder/#monitor-your-concentrator) in order to activate metrics on the forwarder side. You can find also details about the generated metrics + +{!_shared_content/operations_center/integrations/generated/915a119c-2ec8-4482-a3c6-4d4cae62b671fc_sample.md!} + +{!_shared_content/integration/detection_section.md!} + +{!_shared_content/operations_center/detection/generated/suggested_rules_915a119c-2ec8-4482-a3c6-4d4cae62b671fc_do_not_edit_manually.md!} +{!_shared_content/operations_center/integrations/generated/915a119c-2ec8-4482-a3c6-4d4cae62b671fc.md!} diff --git a/docs/integration/ingestion_methods/syslog/sekoiaio_forwarder.md b/docs/integration/ingestion_methods/syslog/sekoiaio_forwarder.md index a506e11406..85c373fae5 100644 --- a/docs/integration/ingestion_methods/syslog/sekoiaio_forwarder.md +++ b/docs/integration/ingestion_methods/syslog/sekoiaio_forwarder.md @@ -324,6 +324,60 @@ tls_cert_name: server.crt tls_ca_name: server.crt ``` +## Monitor your concentrator + +The forwarder is a critical component in the architecture between your information system and the Sekoia platform. A prolonged service interruption could lead to data loss, potentially causing missed detection of an attack within your environment. + +In this context, please find below the instructions to enable monitoring of your forwarder. +This will allow health status information of the component to be transmitted to Sekoia, enabling you to set up alerts based on the values of the transmitted metrics. + +### Create the forwarder logs intake + +The first step is to create the intake on the Sekoia platform and save the associated intake key + +For detailed information on this process, please refer to the following [documentation](/integration/categories/applicative/sekoiaio_forwarder_logs/) + +### Configuration of the intake.yml file + +To activate the monitoring mode, simply defined a new intake and add `stats: True` in the `intakes.yaml` file with the associated `intake_key` coming from the right format: + +```yaml +--- +intakes: +- name: Monitoring + stats: True + intake_key: INTAKE_KEY_FOR_FORWARDER_LOGS +``` + +!!! Note + This intake is special because logs are auto-generated by the forwarder. As a consequency, you don't need to define an input port neither the protocol. + +### Understanding concentrator metrics + +The concentrator is based on a rsyslog instance so to monitor the forwarder, we decided against developing our own metrics. Instead, we opted for leveraging a standard implementation provided by rsyslog direclty: the [impstats module](https://www.rsyslog.com/doc/configuration/modules/impstats.html) + +By enabling this internal module, rsyslog generates numerous low-level metrics, which are essential for us to comprehend in order to understand the forwarder metrics. Therefore, it is crucial to grasp the operational workflow of rsyslog. [Here](https://www.rsyslog.com/doc/configuration/basic_structure.html) you can find more details about rsyslog principles. + +The forwarder is configured with one input module per intake, as specified in the `intake.yaml` file. Each input module is paired with a corresponding ruleset, action, and specific queue. When monitoring is enabled, dedicated metrics for each module will be transferred to Sekoia, identified by a specific name, as shown below: + +![image](/assets/operation_center/ingestion_methods/sekoiaio_forwarder/forwarder_monitoring_2.png){: style="max-width:100%"} + +By leveraging these metrics, you can easily define custom rules to detect specific behaviors such as service interruptions, full queues, discarded events, and more. This monitoring capability is crucial for maintaining optimal performance and ensuring the reliability of the forwarder in your system. + +!!! Note + To understand the detailed meaning of each counter, please refer to the [associated rsyslog documentation](https://www.rsyslog.com/doc/configuration/rsyslog_statistic_counter.html). + +### Extract concentrator metrics in case of outage + +In extreme cases, the forwarder may cease to function entirely, and as a result, it will also stop sending its metrics to Sekoia (e.g., a full queue). While an alert from Sekoia will notify you of this issue, you will still need to investigate and understand the root cause to resolve the problem. + +In addition to continuously sending its metrics to Sekoia, the forwarder also retains a raw copy of its metrics locally. To retrieve these logs on your host for debugging purpose, you can use the following command: + + +```bash + sudo docker compose cp rsyslog:/var/log/rsyslog-stats.log rsyslog-stats.log +``` + ## Start the concentrator To start the concentrator, run the following command in the folder where `docker-compose.yml` and `intakes.yaml` are stored: diff --git a/mkdocs.yml b/mkdocs.yml index e3c20ccc05..4912fbbaa2 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -328,6 +328,7 @@ nav: - Microsoft IIS: integration/categories/applicative/microsoft_iis.md - Salesforce: integration/categories/applicative/salesforce.md - Sekoia.io activity logs: integration/categories/applicative/sekoiaio_activity_logs.md + - Sekoia.io forwarder logs: integration/categories/applicative/sekoiaio_forwarder_logs.md - Systancia Cleanroom: integration/categories/applicative/systancia_cleanroom.md - Veeam Backup: integration/categories/applicative/veeam_backup.md - Email: @@ -646,6 +647,7 @@ plugins: xdr/features/collect/integrations/application/openvpn.md: integration/categories/network/openvpn.md xdr/features/collect/integrations/application/rsa_securid.md: integration/categories/iam/rsa_securid.md xdr/features/collect/integrations/application/sekoiaio_activity_logs.md: integration/categories/applicative/sekoiaio_activity_logs.md + xdr/features/collect/integrations/application/sekoiaio_forwarder_logs.md: integration/categories/applicative/sekoiaio_forwarder_logs.md xdr/features/collect/integrations/application/systancia_cleanroom.md: integration/categories/applicative/systancia_cleanroom.md xdr/features/collect/integrations/application/unbound.md: integration/categories/network/unbound.md xdr/features/collect/integrations/application/veeam_backup.md: integration/categories/applicative/veeam_backup.md