Merge branch 'master' into updates-to-azure-jul29

docs/_include/general-shipping/k8s-all-data.md

@@ -1,28 +1,28 @@
-## All telemetry (logs, metrics, traces and security reports) at once
+## Send All Telemetry Data (logs, metrics, traces and security reports)
 
 
-To enjoy the full Kubernetes 360 experience, you can send all your telemetry data to Logz.io using one single Helm chart:
+Send all of your telemetry data using one single Helm chart:
+
 
 ```sh
 helm repo add logzio-helm https://logzio.github.io/logzio-helm
 helm repo update
 helm install  -n monitoring --create-namespace \
 --set logs.enabled=true \
---set logzio-logs-collector.secrets.logzioLogsToken="<<LOG-SHIPPING-TOKEN>>" \  
---set logzio-logs-collector.secrets.logzioRegion="<<LOGZIO-REGION>>" \  
---set logzio-logs-collector.secrets.env_id="<<CLUSTER-NAME>>" \  
---set logzio-fluentd.enabled=false \
+--set logzio-logs-collector.secrets.logzioLogsToken="<<LOG-SHIPPING-TOKEN>>" \
+--set logzio-logs-collector.secrets.logzioRegion="<<LOGZIO-REGION>>" \
+--set logzio-logs-collector.secrets.env_id="<<CLUSTER-NAME>>" \
 --set metricsOrTraces.enabled=true \
 --set logzio-k8s-telemetry.metrics.enabled=true \
---set logzio-k8s-telemetry.secrets.MetricsToken="<<PROMETHEUS-METRICS-SHIPPING-TOKEN>>" \
+--set logzio-k8s-telemetry.secrets.MetricsToken="<<METRICS-SHIPPING-TOKEN>>" \
 --set logzio-k8s-telemetry.secrets.ListenerHost="https://<<LISTENER-HOST>>:8053" \
 --set logzio-k8s-telemetry.secrets.p8s_logzio_name="<<ENV-ID>>" \
 --set logzio-k8s-telemetry.traces.enabled=true \
 --set logzio-k8s-telemetry.secrets.TracesToken="<<TRACING-SHIPPING-TOKEN>>" \
 --set logzio-k8s-telemetry.secrets.LogzioRegion="<<LOGZIO-REGION>>" \
 --set logzio-k8s-telemetry.spm.enabled=true \
 --set logzio-k8s-telemetry.secrets.env_id="<<ENV-ID>>" \
---set logzio-k8s-telemetry.secrets.SpmToken="<<PROMETHEUS-METRICS-SHIPPING-TOKEN>>" \
+--set logzio-k8s-telemetry.secrets.SpmToken="<<SPM-ACCOUNT-SHIPPING-TOKEN>>" \
 --set logzio-k8s-telemetry.serviceGraph.enabled=true \
 --set logzio-k8s-telemetry.k8sObjectsConfig.enabled=true \
 --set logzio-k8s-telemetry.secrets.k8sObjectsLogsToken="<<LOG-SHIPPING-TOKEN>>" \
@@ -41,9 +41,8 @@ logzio-monitoring logzio-helm/logzio-monitoring
 | --- | --- |
 | `<<LOG-SHIPPING-TOKEN>>` | Your [logs shipping token](https://app.logz.io/#/dashboard/settings/general). |
 | `<<LISTENER-HOST>>` | Your account's [listener host](https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping?product=logs). |
-| `<<PROMETHEUS-METRICS-SHIPPING-TOKEN>>` | Your [metrics shipping token](https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping). |
-| `<<P8S-LOGZIO-NAME>>` | The name for the environment's metrics, to easily identify the metrics for each environment. |
+| `<<METRICS-SHIPPING-TOKEN>>` | Your [metrics shipping token](https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping). |
+| `<<SPM-ACCOUNT-SHIPPING-TOKEN>>` | Your [SPM account shipping token](https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping) |
 | `<<ENV-ID>>` | The name for your environment's identifier, to easily identify the telemetry data for each environment. |
 | `<<TRACING-SHIPPING-TOKEN>>` | Your [traces shipping token](https://app.logz.io/#/dashboard/settings/manage-tokens/data-shipping). |
-| `<<LOGZIO-REGION>>` | Name of your Logz.io traces region e.g `us`, `eu`... |
-
+| `<<LOGZIO-REGION>>` | Your Logz.io [region code](https://docs.logz.io/docs/user-guide/admin/hosting-regions/account-region/#available-regions) |

docs/_include/log-shipping/certificate.md

@@ -1,4 +1,4 @@
-##### Download the Logz.io public certificate to your credentials server
+### Download the Logz.io public certificate
 
 For HTTPS shipping, download the Logz.io public certificate to your certificate authority folder.
 

docs/_include/log-shipping/filebeat-installed-port5015-begin.md

@@ -1,3 +1,3 @@
 * [Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-installation.html) installed
-* Port 5015 open
+* Port 5015 open to outgoing traffic
 * Root access

docs/_include/log-shipping/filebeat-installed-port5015-end.md

@@ -1,2 +1,3 @@
+:::note
 While support for [Filebeat 6.3 and later versions](https://www.elastic.co/guide/en/beats/filebeat/6.7/filebeat-installation.html) is available, Logz.io recommends that you use the latest stable version
-* Destination port 5015 open to outgoing traffic
+:::

docs/_include/log-shipping/filebeat-ssl.md

@@ -1,14 +1,11 @@
-###### Disabling SSL for Filebeat log shipping
+#### Disabling SSL
 
-By default, Filebeat uses SSL/TLS to secure the communication between Filebeat and Logz.io. However, if you want to disable SSL, you can modify the Filebeat configuration accordingly.
+Filebeat uses SSL/TLS to secure the communication between Filebeat and Logz.io. To disable SSL, modify the Filebeat configuration accordingly:
 
-To ship logs without using SSL in Filebeat:
+1. Open the Filebeat configuration file, typically located at `/etc/filebeat/filebeat.yml` (Linux) or `C:\ProgramData\Filebeat\filebeat.yml` (Windows).
 
-1. Open the Filebeat configuration file for editing. The configuration file's location may vary depending on your operating system, but it is commonly located at `/etc/filebeat/filebeat.yml` (Linux) or `C:\ProgramData\Filebeat\filebeat.yml` (Windows).
+2. Find the `output.logstash` section in the file.
 
-2. Look for the `output.logstash` section in the configuration file.
+3. Remove the # character at the beginning of the #ssl.enabled line to disable SSL. The line should now look like this: `#ssl.enabled: false`
 
-3. Uncomment the # character at the beginning of the #ssl.enabled line to disable SSL. The line should now look like this:
-  `#ssl.enabled: false`
-
-4. Save the changes to the configuration file and restart the Filebeat service to apply the changes.
+4. Save the changes and restart the Filebeat service to apply the changes.

docs/_include/log-shipping/filebeat-wizard.html

@@ -1 +1 @@
-Log into your Logz.io account, and go to the [Filebeat log shipping page](https://app.logz.io/#/dashboard/send-your-data/log-sources/filebeat) to use the dedicated Logz.io Filebeat configuration wizard. It's the simplest way to configure Filebeat for your use case.
+Log in to Logz.io and navigate to the [Filebeat log shipping page](https://app.logz.io/#/dashboard/integrations/Filebeat-data).

docs/_include/log-shipping/filebeat-wizard.md

@@ -1,19 +1,24 @@
-###### Adding log sources to the configuration file
-
-For each of the log types you plan to send to Logz.io, fill in the following:
+#### Adding log sources to the configuration file
 
 * Select your operating system - **Linux** or **Windows**.
-* Specify the full **Path** to the logs.
-* Select a log **Type** from the list or select **Other** and give it a name of your choice to specify a custom log type. 
+* Specify the full log **Path**.
+* Select a log **Type** from the list or select **Other** to create and specify a custom log type. 
   * If you select a log type from the list, the logs will be automatically parsed and analyzed. [List of types available for parsing by default](https://docs.logz.io/docs/user-guide/data-hub/log-parsing/default-parsing/#built-in-log-types).
-  * If you select **Other**, contact support to request custom parsing assistance. Don't be shy, it's included in your plan!
+  * If you select **Other**, contact support for custom parsing assistance.
 * Select the log format - **Plaintext** or **Json**.
-* (_Optional_) Enable the **Multiline** option if your log messages span
+* (Optional) Enable the **Multiline** option if your log messages span
 multiple lines. You’ll need to give a regex that
 identifies the beginning line of each log.
-* (_Optional_) Add a custom field. Click **+ Add a field** to add additional fields.
+* (Optional) Add a custom field. Click **+ Add a field** to add additional fields.
+
+:::note
+The wizard makes it simple to add multiple log types to a single configuration file. So to add additional sources, click **+ Add a log type** to fill in the details for another log type. Repeat as necessary.
+:::
+
+#### Filebeat 8.1+
+If you're running Filebeat 8.1+, there are some adjustment you need to make in the config file:
 
-If you're running Filebeat 8.1+, the `type` of the `filebeat.inputs` is `filestream` instead of `logs`:
+1. Change `type` of the `filebeat.inputs` to `filestream` instead of `logs`:
 
 ```yaml
 filebeat.inputs:
@@ -22,7 +27,16 @@ filebeat.inputs:
     - /var/log/*.log
 ```
 
-###### Add additional sources (_Optional_)
-
-The wizard makes it simple to add multiple log types to a single configuration file. Click **+ Add a log type** to fill in the details for another log type. Repeat as necessary.
+2. **To configure multiline** nest the multiline settings under `parsers`:
 
+```yaml
+- type: filestream
+  paths:
+    - /var/log/*.log
+  parsers:
+  - multiline:
+      type: pattern
+      pattern: '^\d{4}-'
+      negate: true
+      match: after
+```

docs/_include/log-shipping/lambda-xtension-tablink-indox.html

@@ -1 +1 @@
-in the [Environment Variables & ARNs](https://docs.logz.io/docs/shipping/Compute/Lambda-extensions#environment-variables) tab
+(https://docs.logz.io/docs/shipping/Compute/Lambda-extensions#environment-variables)

docs/_include/log-shipping/lambda-xtension-tablink.md

@@ -1 +1 @@
-(https://app.logz.io/#/dashboard/send-your-data/log-sources/lambda-extensions?type=tables)
+(https://docs.logz.io/docs/shipping/aws/lambda-extensions/#arns)

docs/_include/log-shipping/rsyslog-troubleshooting.md

@@ -1,20 +1,22 @@
-This section contains some guidelines for handling errors that you may encounter when trying to collect logs for Rsyslog - SELinux configuration.
+## Troubleshooting
 
-SELinux is a Linux feature that allows you to implement access control security policies in Linux systems. In distributions such as Fedora and RHEL, SELinux is in Enforcing mode by default.
+This section provides guidelines for handling errors when collecting logs for Rsyslog with SELinux configuration.
 
-Rsyslog is one of the system processes protected by SELinux. This means that rsyslog by default is not allowed to send to a port other than 514/udp (the standard syslog port) has limited access to other files and directories outside of their initial configurations.
+SELinux is a Linux feature for implementing access control security policies. In distributions like Fedora and RHEL, SELinux is enabled in Enforcing mode by default.
 
-To send information to Logz.io properly in a SELinux environment, it is necessary to add exceptions to allow:
+Rsyslog, a system process protected by SELinux, is restricted by default to sending data only to port 514/udp (the standard syslog port) and has limited access to files and directories beyond its initial configuration.
 
-* rsyslog to communicate with logz.io through the desired port
-* rsyslog to access the files and directories needed for it to work properly
+To send data to Logz.io in a SELinux environment, you need to add exceptions to allow:
 
+* rsyslog to communicate with logz.io through the desired port.
+* rsyslog to access the necessary files and directories.
 
-##### Possible cause - issue not related to SELinux
+
+### Issue not related to SELinux
 
 The issue may not be caused by SELinux.
 
-###### Suggested remedy
+**Suggested remedy**
 
 Disable SELinux temporarily and see if that solves the problem.
 
@@ -55,23 +57,20 @@ SELINUX=disabled
 SELINUX=permissive 
 ```
 
-##### Possible cause - need exceptions to SELinux for Logz.io
+### Need to add exceptions
 
 You may need to add exception to SELinux configuration to enable Logz.io.
 
-###### Suggested remedy
-
+**Suggested remedy**
 
-###### Install the policycoreutils and the setroubleshoot packages
+1. Install the policycoreutils and the setroubleshoot packages:
 
 ```shell
 # Installing policycoreutils & setroubleshoot packages
 $ sudo yum install policycoreutils setroubleshoot
 ```
 
-###### Check which syslog ports are allowed by SELinux
-
-Run the command as in the example below:
+2. Check which syslog ports are allowed by SELinux:
 
 ```shell
 $ sudo semanage port -l| grep syslog
@@ -80,14 +79,14 @@ output:
 syslogd_port_t udp 514
 ```
 
-###### Add a new port to policy for Logz.io
+3. Add a new port to policy for Logz.io:
 
 ```shell
 # Adding a port to SELinux policies
 $ sudo semanage port -m -t syslogd_port_t -p tcp 5000
 ```
 
-###### Authorize Rsyslog directory
+4. Authorize Rsyslog directory:
 
 
 ```shell
@@ -96,7 +95,7 @@ $ sudo semanage fcontext -a -t syslogd_var_lib_t "/var/spool/rsyslog/*"
 $ sudo restorecon -R -v /var/spool/rsyslog
 ```
 
-Depending on the distribution, run the following command:
+5. Depending on the distribution, run the following command:
 
 ```shell
 # instructing se to authorize /etc/rsyslog.d/*
@@ -109,7 +108,7 @@ $ sudo semanage fcontext -a -t etc_t "/etc/rsyslog.d"
 $ sudo restorecon -v /etc/rsyslog.d
 ```
 
-###### Restart Rsyslog
+6. Restart Rsyslog:
 
 ```shell
 $ sudo service rsyslog restart

docs/_include/log-shipping/stack.md

@@ -1,4 +1,4 @@
-##### Create new stack
+#### Create new stack
 
 To deploy this project, click the button that matches the region you wish to deploy your Stack to:
 
@@ -23,7 +23,7 @@ To deploy this project, click the button that matches the region you wish to dep
 | `ca-central-1`   | [![Deploy to AWS](https://dytvr9ot2sszz.cloudfront.net/logz-docs/lights/LightS-button.png)](https://console.aws.amazon.com/cloudformation/home?region=ca-central-1#/stacks/create/review?templateURL=https://logzio-aws-integrations-ca-central-1.s3.amazonaws.com/s3-hook/0.4.2/sam-template.yaml&stackName=logzio-s3-hook&param_logzioToken=<<LOG-SHIPPING-TOKEN>>&param_logzioListener=https://<<LISTENER-HOST>>:8071)     | 
 
 
-##### Specify stack details
+#### Specify stack details
 
 Specify the stack details as per the table below, check the checkboxes and select **Create stack**.
 
@@ -37,39 +37,31 @@ Specify the stack details as per the table below, check the checkboxes and selec
 | `pathToFields`   | Fields from the path to your logs directory that you want to add to the logs. For example, `org-id/aws-type/account-id` will add each of the fields `ord-id`, `aws-type` and `account-id` to the logs that are fetched from the directory that this path refers to.  | -                  |
 
 
-##### Add trigger
+#### Add trigger
 
-Give the stack a few minutes to be deployed.
+After deploying the stack, wait a few minutes for it to complete. Once your Lambda function is ready, you'll need to manually add a trigger due to CloudFormation limitations:
 
-Once your Lambda function is ready, you'll need to manually add a trigger. This is due to Cloudformation limitations.
+1. Navigate to the function's page and click on **Add trigger**.
 
-Go to the function's page, and click on **Add trigger**.
+2. Choose **S3** as a trigger, and fill in:
 
-![Step 5 screenshot](https://dytvr9ot2sszz.cloudfront.net/logz-docs/control-tower/s3-hook-stack-05.png)
+    - **Bucket**: Your bucket name.
+    - **Event type**: Select `All object create events`.
+    - **Prefix** and **Suffix**: Leave these fields empty.
 
-Then, choose **S3** as a trigger, and fill in:
+    Confirm the checkbox, and click **Add**.
 
-- **Bucket**: Your bucket name.
-- **Event type**: Choose option `All object create events`.
-- Prefix and Suffix should be left empty.
 
-Confirm the checkbox, and click **Add*.
+#### Send logs
 
-![Step 5 screenshot](https://dytvr9ot2sszz.cloudfront.net/logz-docs/control-tower/s3-hook-stack-06.png)
+Your function is now configured. When you upload new files to your bucket, the function will be triggered, and the logs will be sent to your Logz.io account.
 
-##### Send logs
+#### Parsing
 
-That's it. Your function is configured.
-Once you upload new files to your bucket, it will trigger the function, and the logs will be sent to your Logz.io account.
+The S3 Hook will automatically parse logs if the object's path contains the phrase `cloudtrail` (case insensitive).
 
-###### Parsing
+#### Check your logs
 
-S3 Hook will automatically parse logs in the following cases:
+Allow some time for data ingestion, then check your [OpenSearch Dashboards](https://app.logz.io/#/dashboard/osd/discover/).
 
-- The object's path contains the phrase `cloudtrail` (case insensitive).
-
-##### Check Logz.io for your logs
-
-Give your logs some time to get from your system to ours, and then open [OpenSearch Dashboards](https://app.logz.io/#/dashboard/osd/discover/).
-
-If you still don't see your logs, see Log shipping troubleshooting.
+For troubleshooting, refer to our [log shipping troubleshooting](https://docs.logz.io/docs/user-guide/log-management/troubleshooting/log-shipping-troubleshooting/) guide.

docs/_include/log-shipping/validate-yaml.md

@@ -1,7 +1,7 @@
-##### Download and validate the file
+#### Download and validate confiuration
 
 When you're done adding your sources, click **Make the config file** to download it.
 
 You can compare it to our [sample configuration](https://raw.githubusercontent.com/logzio/logz-docs/master/shipping-config-samples/logz-filebeat-config.yml) if you have questions.
 
-If you've edited the file manually, it's a good idea to run it through a YAML validator to rule out indentation errors, clean up extra characters, and check if your yml file is valid. ([Yamllint.com](http://www.yamllint.com/) is a great choice.)
+Validate the file using a YAML validator tool, such as ([Yamllint.com](http://www.yamllint.com/).