diff --git a/docs/assets/operation_center/intakes/intakes-card-menu.png b/docs/assets/operation_center/intakes/intakes-card-menu.png new file mode 100644 index 0000000000..f264cf70b4 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-card-menu.png differ diff --git a/docs/assets/operation_center/intakes/intakes-card.png b/docs/assets/operation_center/intakes/intakes-card.png new file mode 100644 index 0000000000..05f28c7a50 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-card.png differ diff --git a/docs/assets/operation_center/intakes/intakes-connector.png b/docs/assets/operation_center/intakes/intakes-connector.png new file mode 100644 index 0000000000..86eb7b64a1 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-connector.png differ diff --git a/docs/assets/operation_center/intakes/intakes-critical.png b/docs/assets/operation_center/intakes/intakes-critical.png new file mode 100644 index 0000000000..e71b7d2db1 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-critical.png differ diff --git a/docs/assets/operation_center/intakes/intakes-date-selector.png b/docs/assets/operation_center/intakes/intakes-date-selector.png new file mode 100644 index 0000000000..9d9a63964f Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-date-selector.png differ diff --git a/docs/assets/operation_center/intakes/intakes-details-configure.png b/docs/assets/operation_center/intakes/intakes-details-configure.png new file mode 100644 index 0000000000..c2a71e7ccb Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-details-configure.png differ diff --git a/docs/assets/operation_center/intakes/intakes-details-entity.png b/docs/assets/operation_center/intakes/intakes-details-entity.png new file mode 100644 index 0000000000..08410591a9 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-details-entity.png differ diff --git a/docs/assets/operation_center/intakes/intakes-details-error.png b/docs/assets/operation_center/intakes/intakes-details-error.png new file mode 100644 index 0000000000..d742e64664 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-details-error.png differ diff --git a/docs/assets/operation_center/intakes/intakes-details-menu.png b/docs/assets/operation_center/intakes/intakes-details-menu.png new file mode 100644 index 0000000000..f2fc007a58 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-details-menu.png differ diff --git a/docs/assets/operation_center/intakes/intakes-details-name.png b/docs/assets/operation_center/intakes/intakes-details-name.png new file mode 100644 index 0000000000..ab2e13a08b Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-details-name.png differ diff --git a/docs/assets/operation_center/intakes/intakes-details-rename.png b/docs/assets/operation_center/intakes/intakes-details-rename.png new file mode 100644 index 0000000000..bab9b6e171 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-details-rename.png differ diff --git a/docs/assets/operation_center/intakes/intakes-details.png b/docs/assets/operation_center/intakes/intakes-details.png new file mode 100644 index 0000000000..2d3e20a282 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-details.png differ diff --git a/docs/assets/operation_center/intakes/intakes-error.png b/docs/assets/operation_center/intakes/intakes-error.png new file mode 100644 index 0000000000..ed6916c0ea Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-error.png differ diff --git a/docs/assets/operation_center/intakes/intakes-event-filter.png b/docs/assets/operation_center/intakes/intakes-event-filter.png new file mode 100644 index 0000000000..335cfd5e3c Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-event-filter.png differ diff --git a/docs/assets/operation_center/intakes/intakes-graph-hover.png b/docs/assets/operation_center/intakes/intakes-graph-hover.png new file mode 100644 index 0000000000..b3f5c3d1b4 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-graph-hover.png differ diff --git a/docs/assets/operation_center/intakes/intakes-graph.png b/docs/assets/operation_center/intakes/intakes-graph.png new file mode 100644 index 0000000000..1a3d2e4241 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-graph.png differ diff --git a/docs/assets/operation_center/intakes/intakes-last-events.png b/docs/assets/operation_center/intakes/intakes-last-events.png new file mode 100644 index 0000000000..b50f7cd20d Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-last-events.png differ diff --git a/docs/assets/operation_center/intakes/intakes-list-card.png b/docs/assets/operation_center/intakes/intakes-list-card.png new file mode 100644 index 0000000000..fcf46d14c0 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-list-card.png differ diff --git a/docs/assets/operation_center/intakes/intakes-list-diff.png b/docs/assets/operation_center/intakes/intakes-list-diff.png new file mode 100644 index 0000000000..4b08011720 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-list-diff.png differ diff --git a/docs/assets/operation_center/intakes/intakes-list-notification.png b/docs/assets/operation_center/intakes/intakes-list-notification.png new file mode 100644 index 0000000000..22c63d9c2e Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-list-notification.png differ diff --git a/docs/assets/operation_center/intakes/intakes-list-order.png b/docs/assets/operation_center/intakes/intakes-list-order.png new file mode 100644 index 0000000000..af4d13a7bc Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-list-order.png differ diff --git a/docs/assets/operation_center/intakes/intakes-listing.png b/docs/assets/operation_center/intakes/intakes-listing.png index b2baccda73..c6553fbc6a 100644 Binary files a/docs/assets/operation_center/intakes/intakes-listing.png and b/docs/assets/operation_center/intakes/intakes-listing.png differ diff --git a/docs/assets/operation_center/intakes/intakes-log-error.png b/docs/assets/operation_center/intakes/intakes-log-error.png new file mode 100644 index 0000000000..3ba4af4347 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-log-error.png differ diff --git a/docs/assets/operation_center/intakes/intakes-new-notification.png b/docs/assets/operation_center/intakes/intakes-new-notification.png new file mode 100644 index 0000000000..4f03c494e7 Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-new-notification.png differ diff --git a/docs/assets/operation_center/intakes/intakes-running.png b/docs/assets/operation_center/intakes/intakes-running.png new file mode 100644 index 0000000000..b84be85aeb Binary files /dev/null and b/docs/assets/operation_center/intakes/intakes-running.png differ diff --git a/docs/xdr/features/collect/intakes.md b/docs/xdr/features/collect/intakes.md index 8c2696b07d..1eae43d06f 100644 --- a/docs/xdr/features/collect/intakes.md +++ b/docs/xdr/features/collect/intakes.md @@ -1,39 +1,113 @@ # Intakes -Intakes correspond to data sources sent to Sekoia.io. They are identified by a name, a log format, an entity and an intake key. +Intakes correspond to data sources sent to Sekoia.io. They are identified by a name, an intake format, an entity and an intake key. You can configure as much intakes as you need in order to increase Sekoia.io knowledge of your infrastructure. -All features related to intakes are visible through the “Intakes” menu on the Operations Center. +All features related to intakes are visible through the “Intakes” menu. + ## Intakes listing -The intakes homepage allows you to view the list of intakes already created within the community but also to create new intakes. +The intakes homepage allows you to view and manage the list of intakes already created within the community but also to create new intakes. ![intakes_listing](/assets/operation_center/intakes/intakes-listing.png){: style="max-width:100%"} On this page, you can find: -- The created intakes in your community -- The entity associated with the intake -- The number of events sent to Sekoia.io for the past 7 days -- The number of events in error -- The intake key for each intake that you can copy directly from the table to your clipboard -- The creator as well as the creation date -- A delete button that implies the events produced by the intake will no longer be analyzed +- The created intakes in your community with their associated entity +- The number of `valid events` +- The number of `invalid events` +- The number of `events in warning` + +![intakes_card](/assets/operation_center/intakes/intakes-card.png){: style="max-width:100%"} + +### Event types + +#### Invalid event +An event is invalid when the data does not match the expected format (JSON, Key value, Grok, Date, Delimiter separated value). + +#### Event in warning +A warning can be generated in 2 cases: + +- The parsing of the event was a success but no data were extracted +- Some values were incompatibles with their expected data type and could not be extracted + +### Events date range +Use the date range selector to view the number of events in different timeframes (24h, 7 days or 30 days). This date range selector will help you to assess if the volume of ingested logs has the expected behavior. + +![intakes_date_selector](/assets/operation_center/intakes/intakes-date-selector.png){: style="max-width:100%"} + +### Intake types + +The Integration catalog contains two different types of intakes. These 2 types differs on the method of collecting the events. + +#### Push intake + +With Push intakes, logs must be forwarded by the user to Sekoia with one of our forwarding methods (Syslog, Sekoia concentrator, etc..). + +#### Pull intake + +With Pull intakes, logs are collected directly by Sekoia with a connector. The connector usually uses a REST API to collect data. In other words Sekoia SOC platform is fetching the logs for the user this case. + +![intakes_list_diff](/assets/operation_center/intakes/intakes-list-diff.png){: style="max-width:100%"} + +Pull intakes are distinguishable by their On/Off button and Connect log tab. + +##### Start/Stop a Pull intake + +![intakes_running](/assets/operation_center/intakes/intakes-running.png){: style="max-width:100%"} + +Pull intakes have a On/Off button which allows you to enable or disable the collection of data. Pull intakes have a `connector` that regularly pulls the logs from your data source. + +##### Error status + +When a Pull intake is fetching logs, it can encounter different kind of issues. The error status will help you understand the current state of the `connector`. +![intakes_error](/assets/operation_center/intakes/intakes-error.png){: style="max-width:100%"} +![intakes_critical](/assets/operation_center/intakes/intakes-critical.png){: style="max-width:100%"} + +| Status | Description | +| --- | --- | +| **Error during data collection** | Some errors were encountered during data collection but the connector is still running. **A verification by the user may be needed to fix the issues**. | +| **Stopped by critical error** | The connector was stopped because of a critical error. **An urgent action is required by the user to investigate and fix the issues**. Note that if too many errors are encountered, the connector will consider them as a critical error. | + +To troubleshoot the origin of these errors and restore data collection, you can use the `connector log` tab. + +### Filters + +Multiple filters are available to search for specific intakes. + +| Filter | Description | +| --- | --- | +| Created at | Filter intakes by their date of creation (date range, relative dates…) | +| Entity | Filter intakes by associated entities | +| Intake format | Filter intakes by their intake formats. An intake format corresponds to a specific product | +| Intake key | Filter intakes by their intake key. An intake key is the unique identifier used to identify to which intake a log belong | +| Status | Filter intakes by their status. Available status are: Running, Stopped, Error during data collection and Stopped by critical error | +| UUID | Filter intakes by their UUID. A UUID is a unique identifier of an intake. | + +### Ordering + +The list of intakes can be ordered by: + +- Creation date (in descending order) +- Error status (in this order: Stopped by critical error, Error during data collection) +- Name (in alphabetical order) + +![intakes_list_order](/assets/operation_center/intakes/intakes-list-order.png){: style="max-width:100%"} ## Create an intake from our integrations catalog -To configure a new source of events in your community, you can rely on our list of continuously growing integrations that are constantly developed and enhanced by Sekoia.io’s team. -![intakes_types](/assets/operation_center/intakes/intakes-types.png){: style="max-width:100%"} +To configure a new data source in your community, you can rely on our list of continuously growing integrations that are constantly developed and enhanced by Sekoia.io’s team. +![intakes_list_card](/assets/operation_center/intakes/intakes-list-card.png){: style="max-width:100%"} To create an intake, you have to: 1. Click on the `+ Intake` from the Intakes homepage -2. Choose the source of events that suits your needs (you can filter these sources by clicking on the categories’ tags) -3. Click on the `Create` button in the source card +2. Choose the data source that suits your needs (you can filter these data sources by clicking on the categories’ tags) +3. Click on the `Create` button in the card 4. Provide an intelligible name 5. Select the entity to which you want to associate the corresponding data 6. Click on `Create` -7. Find your newly created source of events as well as the intake key in the Intakes homepage +7. (optional) For Pull intakes which requires a connector, you will need to provide configuration parameters to access the data source ![intakes_creation](/assets/operation_center/intakes/modal-intake-creation.png){: style="max-width:100%"} @@ -43,42 +117,173 @@ To create an intake, you have to: ## Configure a notification to report on inactive intake An inactive intake may have devastating consequences on your security monitoring. To prevent incidents from happening, you can set up notifications to get alerted when an intake stops sending events to Sekoia.io. -To set up your notification, you will have to: +To set up your notification, you can: -1. Go to the Intakes listing page and click on: ![Notification logo](/assets/operation_center/intakes/Monitor_intake_logo.png){: style="max-width:10%"} on the right side of the table +1. Go to the Intakes listing page and click on: ![Card menu](/assets/operation_center/intakes/intakes-card-menu.png){: style="max-width:10%"} on the right side of the card and click on the `Notifications` menu 2. Specify how long the intake should be inactive before sending a notification. The duration can go from 15 min to 24 hours. 3. Select how you want to be notified. Triggered actions that are available are mentioned in the page [Turn on notifications](/getting_started/notifications-Listing_Creation). !!! note - You can also set up this notification from the User Center > Notifications by selecting the trigger `No events are received`. + You can also set up this notification from the **intake details page** or the **User Center** > Notifications by selecting the trigger `No events are received`. + +![Create notification](/assets/operation_center/intakes/intakes-list-notification.png){: style="max-width:100%"} -![Create notification](/assets/operation_center/intakes/Monitor_intake_create_notification.png){: style="max-width:100%"} +## Create a custom intake +To learn more about how to create a custom intake, please refer to this [section.](/../../../integration/develop_integration/formats/create_a_format/) -## Notification when an intake is not receiving events +## Intake details page -You can set a notification when an intake stops receiving events: +The details page of an intake provides key infos to check the health state of your data source collection. These infos are: -- On the Intake's page by clicking on the bell next to the delete button -- On the `Notifications` page of the User Center +- The volume of ingested events in a visual graph +- The volume of events invalid or in warning +- An event delivery metric which indicates the lag of event delivery +- The latest ingested events +- The logs of the connector (for Pull intakes only) -![Intakenotification](/assets/operation_center/intakes/2022-12-30_16h35_06.png) +![intakes_details](/assets/operation_center/intakes/intakes-details.png){: style="max-width:100%"} -Here is the needed configuration setup: +### Events graph -- Trigger: `No Events are received` -- Period: Period of time when the intake selected did not receive any events -- Intake -- Actions to perform: Choose a channel where the notification will be sent -- Notification name +![intakes_graph](/assets/operation_center/intakes/intakes-graph.png){: style="max-width:100%"} -## Create a custom intake +The events graph provide metrics on the ingestion of your logs. The metrics are calculated on the selected period (24h, 7 days or 30 days). -To learn more about how to create a custom intake, please refer to this [section.](/../../../integration/develop_integration/formats/create_a_format/) +!!! warning + Events are displayed by their **reception date** in the graph. By using the reception date, you are guaranteed to see events received recently in graph and avoid issues with wrongly dated events or events with important lags. + +- **Events**: total number of events ingested in the selected period +- **Events in warning**: number of events in warning +- **Invalid events**: number of invalid events +- **Valid events**: number of valid events +- **Event delivery**: represents the average latency between the **creation date** of the original event and the **reception date** of the event at Sekoia + +!!! tip + Hover the mouse on the graph to view the number of events per time unit. + +![intakes_graph_hover](/assets/operation_center/intakes/intakes-graph-hover.png){: style="max-width:100%"} + +#### Event delivery + +The event delivery metric is a lag indicator that computes an average value based on the elapse time between the moment the event is created in the data source and the moment it is received in Sekoia. +Different factors can lead to an abnormal value: + +| Potential issue | Description | +| --- | --- | +| Event wrongly dated | If events in the data source are wrongly dated, the event delivery indicator can have a high value or negative value | +| Data source timezone | A timezone issue can be at the origin of a high value or negative value | +| Data source unavailable | If the data source is unavailable for a long time, the lag value will increase because of the delay to collect events | +| Network/Bandwidth issues | Network and bandwidth issues will generate higher lag | +| Events burst | An unusual high burst of events can overload the ingestion and increase lag | + +### List of recent events + +Below the events graph, you have access to the list of recents events. This list help you verify that you received your expected events and allow you to check the content and parsing of these events. Events are displayed by their reception date. Click on the `Show more` button to display additional events. + +#### Invalid & Warning messages + +![intakes_event_filter](/assets/operation_center/intakes/intakes-event-filter.png){: style="max-width:100%"} + +Use the filter above the list of recent events to display the list of invalid or warning messages. This list contains the different type of parsing issues encountered. For each issue is displayed: + +- The issue parsing message +- The last date when the issue was seen +- The number of occurrence of the issue + +The last occurrence of the event can be consulted to troubleshoot issues. + +![intakes_details_error](/assets/operation_center/intakes/intakes-details-error.png){: style="max-width:100%"} + +!!! info + To preserve storage cost, only last occurrence of invalid event is stored in Sekoia. + +### Connector log + +The `Connector log` tab is only available for Pull intakes. + +![intakes_connector](/assets/operation_center/intakes/intakes-connector.png){: style="max-width:100%"} + +In this tab, you will find the latest logs of the connector. These logs help you check that the connector is functioning properly by checking the `Info` level messages. +But they also help you troubleshoot issues by checking the `Error` level messages. + +Use the filter button to filter `Error` logs. + +![intakes_log_error](/assets/operation_center/intakes/intakes-log-error.png){: style="max-width:100%"} + +If the error is related to a configuration issue, read the next section to modify the configuration of the connector. + +### Intake menu + +The intake menu allows you to perform different kind of actions: + +- Edit intake entity +- Configure connector parameters (only for Pull intakes) +- Configure notifications on the intake +- Rename the intake +- Search events of the intake in the Events page +- Consult the documentation page of the intake +- Copy the intake key +- Copy the intake UUID +- Delete the intake + +![intakes_details_menu](/assets/operation_center/intakes/intakes-details-menu.png){: style="max-width:100%"} + +#### Edit entity + +To modify the entity of the intake: + +1. Click on `Edit entity` in the menu +2. Select a new entity in the list +3. Save your modification + +![intakes_details_entity](/assets/operation_center/intakes/intakes-details-entity.png){: style="max-width:100%"} + +#### Configure pull intakes + +This menu is only available for Pull intakes. Use this menu to modify the parameters of the connector. You can change the account used for authentication or change parameter values. + +1. Click on `Configure` in the menu +2. Modify the configuration +3. Save your modification + +![intakes_details_configure](/assets/operation_center/intakes/intakes-details-configure.png){: style="max-width:100%"} + +!!! info + The modifications are applied instantly. You do not need to restart the connector to take your modifications into account. + +#### Notifications + +To create a new notification on the intake: + +1. Click on `Notifications` in the menu +2. Configure the notification settings +3. Save your modification + +#### Rename intake + +To rename the intake: + +1. Click on `Rename` in the menu +2. Enter the new name of the intake +3. Save your modification + +![intakes_details_rename](/assets/operation_center/intakes/intakes-details-rename.png){: style="max-width:100%"} + +#### Delete intake + +To delete the intake: + +1. Click on `Delete` in the menu +2. Confirm the deletion + +!!! warning + The deletion of an intake do not remove the events already ingested. However, note that the intake key will no longer be usable. You will need to deploy a new intake key in your infrastructure if you wish to restore the same kind of intake. --- !!! note Do not hesitate to contact your support if: + - The settings’ recommendations provided are not sufficient or not applicable to your system. We can then see with you how to transfer your events in the best conditions. - The format of the logs you want to send us is not in the list. We regularly add new formats and we can let you know when yours is available in our catalog.