Add Endace Integration #10308

jamesagarside · 2024-07-01T11:11:58Z

Adding support for Endace.

The integration contains two inputs, Netflow for use with Endace Flow https://www.endace.com/EndaceFlow and Packbeat Flow Logs for use with Endace vProbe or where systems dont have another way of generating flows externally.

The integration takes information from the flow event and uses it to construct a pivot link which loads the flow in the Endace platform and stores the link under event.reference.

elasticmachine · 2024-07-01T13:11:53Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

packages/endace/changelog.yml

packages/endace/manifest.yml

packages/endace/_dev/build/docs/README.md

packages/endace/_dev/deploy/docker/docker-compose.yml

packages/endace/data_stream/log/agent/stream/netflow.yml.hbs

packages/endace/data_stream/log/elasticsearch/ingest_pipeline/default.yml

Co-authored-by: Michael Wolf <[email protected]>

.github/CODEOWNERS

packages/endace/data_stream/log/elasticsearch/ingest_pipeline/endace-netflow.yml

packages/endace/data_stream/log/fields/ecs.yml

packages/endace/data_stream/flow/elasticsearch/ingest_pipeline/default.yml

packages/endace/data_stream/flow/elasticsearch/ingest_pipeline/endace.yml

packages/endace/data_stream/flow/elasticsearch/ingest_pipeline/geoip.yml

jamesagarside · 2024-07-15T16:00:14Z

@taylor-swanson @pkoutsovasilis
Thank you both for reviewing this. Im happy to fix these here but its worth noting the pipelines other than endace.yml we're copied from the following Packages.
https://github.com/elastic/integrations/tree/main/packages/network_traffic/data_stream/flow/elasticsearch/ingest_pipeline
https://github.com/elastic/integrations/tree/main/packages/netflow/data_stream/log/elasticsearch/ingest_pipeline
These changes will need to be reflected here to ensure parity.

…/default.yml Co-authored-by: Taylor Swanson <[email protected]>

…#10489) * Able to create P2V when just one IP is present - Fixed start&end time * ngest_pipeline/endace-netflow.yml changed to handle single IP events * updating expected.json after command elastic-package test pipeline --generate

… datastream

jamesagarside · 2024-07-23T15:49:22Z

ER for this PR Issue #9579

jamesagarside · 2024-07-23T15:50:51Z

There is a bug in Kibana which means when rendering the event.reference pivot field ampersands are escaped which causes issues with the Endace appliance. Two Kibana issues have been raised to fix this behaviour.
elastic/kibana#188975
elastic/kibana#188973

taylor-swanson · 2024-07-25T16:22:05Z

Adding @elastic/sec-linux-platform team as they will be the owners of the flow data stream.

taylor-swanson

Few other comments, mostly nit ones.

The bigger concern I had was around the ip and network.type handling in the log data stream. Checking that a v4 address is null or 0.0.0.0 may help that situation, but not sure we if can assume that an ip of 0.0.0.0 is considered an "unset" ip in every circumstance.

taylor-swanson · 2024-07-25T13:33:08Z

packages/endace/_dev/build/docs/README.md

+# Endace
+
+Endace is a company known for its network recording, traffic capture, and analysis technology. Endace's solutions are often used for network security, performance monitoring, and troubleshooting.
+This integration allows users to ingest Network flow data from either Endace Flow via syslog input or use Elastic Agent to generate and ship Network Flow data to an Elastic deployment. Both of these methods add the `event.reference` field to each event when ingested into Elasticsearch which is a URL used to pivot to Endace.   


I don't see a syslog input used? Only netflow and packet are used.

The Netflow integration leverages the Syslog input under the hood I thought

They are completely different technologies (Netflow is a binary protocol)

taylor-swanson · 2024-07-25T14:03:05Z

packages/endace/data_stream/log/_dev/test/pipeline/test-netflow-log-events.json-expected.json

+        {
+            "@timestamp": "2018-04-15T03:30:00Z",
+            "destination": {
+                "ip": "0.0.0.0",
+                "locality": "internal",
+                "port": 135
+            },
+            "ecs": {
+                "version": "8.11.0"
+            },
+            "event": {
+                "action": "netflow_flow",
+                "category": [
+                    "network"
+                ],
+                "kind": "event",
+                "type": [
+                    "connection"
+                ]
+            },
+            "flow": {
+                "id": "GYmhjYyvaAI",
+                "locality": "internal"
+            },
+            "netflow": {
+                "bgp_destination_as_number": 0,
+                "bgp_source_as_number": 0,
+                "destination_ipv4_address": "0.0.0.0",
+                "destination_ipv6_address": "2a02:cf40::2",
+                "destination_transport_port": 135,
+                "exporter": {
+                    "address": "81.2.69.144:4444",
+                    "source_id": 2875616939,
+                    "timestamp": "2018-04-15T03:30:00Z",
+                    "uptime_millis": 0,
+                    "version": 10
+                },
+                "flow_end_seconds": "2018-04-15T03:29:02Z",
+                "flow_start_seconds": "2018-04-15T03:28:44Z",
+                "procera_base_service": "IP protocol 58 (IPv6-ICMP)",
+                "procera_content_categories": "",
+                "procera_flow_behavior": "INITIAL,SERVER_IS_LOCAL,BEGINNING,ESTABLISHED",
+                "procera_http_content_type": "",
+                "procera_http_file_length": 0,
+                "procera_http_location": "",
+                "procera_http_url": "",
+                "procera_incoming_octets": 86,
+                "procera_outgoing_octets": 78,
+                "procera_service": "IP protocol 58 (IPv6-ICMP)",
+                "procera_subscriber_identifier": "",
+                "procera_template_name": "IPFIX",
+                "protocol_identifier": 58,
+                "source_ipv4_address": "0.0.0.0",
+                "source_ipv6_address": "2a02:cf40::1",
+                "source_transport_port": 136,
+                "type": "netflow_flow"
+            },
+            "network": {
+                "community_id": "1:vK+Zeop1Y3GHxfFGVF2/COcNBWw=",
+                "direction": "internal",
+                "iana_number": "58",
+                "transport": "ipv6-icmp",
+                "type": [
+                    "ipv4",
+                    "ipv6"
+                ]
+            },
+            "observer": {
+                "ip": [
+                    "81.2.69.144"
+                ]
+            },
+            "related": {
+                "ip": [
+                    "0.0.0.0"
+                ]
+            },
+            "source": {
+                "ip": "0.0.0.0",
+                "locality": "internal",
+                "port": 136
+            }
+        },


This event looks wrong. The protocol is ipv6-icmp, but only 0.0.0.0 is populated for source.ip, related.ip, and destination.ip.

Likewise, network.ip is both ipv4 and ipv6, but seems like it should only be ipv6 in this case.

Perhaps we have handle 0.0.0.0 in a special manner.

Could we add IPV6 support at a later date?
We could add a check for type on the default pipeline.

packages/endace/data_stream/log/elasticsearch/ingest_pipeline/default.yml

taylor-swanson · 2024-07-25T14:15:19Z

packages/endace/data_stream/log/elasticsearch/ingest_pipeline/endace-netflow.yml

Need on_failure processor like in default.yml.

taylor-swanson · 2024-07-25T14:23:21Z

packages/endace/validation.yml

This entire file can be removed, only applies to kibana dashboards which aren't present here.

packages/endace/manifest.yml

Co-authored-by: Taylor Swanson <[email protected]>

elastic-sonarqube · 2024-07-26T16:20:23Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
97.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

elasticmachine · 2024-07-26T16:20:49Z

💚 Build Succeeded

Buildkite Build
Commit: f1405b0

History

💚 Build #13849 succeeded 505622d
💔 Build #13846 failed fafac50
💔 Build #13843 failed d6d4672
💚 Build #13829 succeeded d4ab3f8
💔 Build #13678 failed 7d07ffb
💔 Build #13384 failed d5afd1e

botelastic · 2024-08-25T16:59:58Z

Hi! We just realized that we haven't looked into this PR in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!

botelastic · 2024-09-24T17:37:41Z

Hi! This PR has been stale for a while and we're going to close it as part of our cleanup procedure. We appreciate your contribution and would like to apologize if we have not been able to review it, due to the current heavy load of the team. Feel free to re-open this PR if you think it should stay open and is worth rebasing. Thank you for your contribution!

jamesagarside and others added 7 commits May 31, 2024 10:36

base intergation

c307a2c

Netflow tests complete

c01b125

Added tests

d72723d

Fixed tests for flow

14ece87

Merge branch 'main' into endace

7e7dd97

Modified owner in manifest

b4bb6a8

Removed test data scripts

76544db

jamesagarside mentioned this pull request Jul 1, 2024

[New Integration] Endace #9579

Open

15 tasks

Extended test coverage

2ccdfd8

mjwolf reviewed Jul 2, 2024

View reviewed changes

jamesagarside and others added 5 commits July 4, 2024 11:22

Update changelog.yml

03697e0

Update changelog.yml

ec2ddea

Update packages/endace/_dev/deploy/docker/docker-compose.yml

e945227

Co-authored-by: Michael Wolf <[email protected]>

Fixed proposed changes

540056a

Updated README

d5afd1e

mjwolf requested a review from a team July 8, 2024 19:45

pkoutsovasilis reviewed Jul 9, 2024

View reviewed changes

.github/CODEOWNERS Outdated Show resolved Hide resolved

pkoutsovasilis reviewed Jul 9, 2024

View reviewed changes

packages/endace/data_stream/log/elasticsearch/ingest_pipeline/endace-netflow.yml Outdated Show resolved Hide resolved

pkoutsovasilis reviewed Jul 9, 2024

View reviewed changes

packages/endace/data_stream/log/elasticsearch/ingest_pipeline/endace-netflow.yml Show resolved Hide resolved

pkoutsovasilis reviewed Jul 9, 2024

View reviewed changes

packages/endace/data_stream/log/fields/ecs.yml Show resolved Hide resolved

taylor-swanson reviewed Jul 9, 2024

View reviewed changes

Update packages/endace/data_stream/flow/elasticsearch/ingest_pipeline…

7d07ffb

…/default.yml Co-authored-by: Taylor Swanson <[email protected]>

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Jul 19, 2024

rcesar999 and others added 5 commits July 22, 2024 11:14

Fixed duplicate tags

0c95395

Corrected typo in agent config

7e1735f

Updated owners of this package to include upstream owener of the Flow…

667677b

… datastream

updated docs

04cb6d4

jamesagarside added 3 commits July 22, 2024 20:00

Added reference field

d6d4672

Fixed pipeline tags, operator grouping and tests

154d45f

Removed redundant ? operators

fafac50

jamesagarside requested review from mjwolf and pkoutsovasilis July 22, 2024 22:07

Update CODEOWNERS

505622d

mjwolf approved these changes Jul 24, 2024

View reviewed changes

taylor-swanson requested a review from a team July 25, 2024 16:21

taylor-swanson reviewed Jul 25, 2024

View reviewed changes

Add failure

f1405b0

Co-authored-by: Taylor Swanson <[email protected]>

botelastic bot added the Stalled label Aug 25, 2024

botelastic bot closed this Sep 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Endace Integration #10308

Add Endace Integration #10308

jamesagarside commented Jul 1, 2024

elasticmachine commented Jul 1, 2024

jamesagarside commented Jul 15, 2024

jamesagarside commented Jul 23, 2024

jamesagarside commented Jul 23, 2024

taylor-swanson commented Jul 25, 2024

taylor-swanson left a comment

taylor-swanson Jul 25, 2024

jamesagarside Jul 26, 2024

taylor-swanson Jul 26, 2024

taylor-swanson Jul 25, 2024

jamesagarside Jul 26, 2024

taylor-swanson Jul 25, 2024

taylor-swanson Jul 25, 2024

elastic-sonarqube bot commented Jul 26, 2024

elasticmachine commented Jul 26, 2024

botelastic bot commented Aug 25, 2024

botelastic bot commented Sep 24, 2024

Add Endace Integration #10308

Add Endace Integration #10308

Conversation

jamesagarside commented Jul 1, 2024

elasticmachine commented Jul 1, 2024

🚀 Benchmarks report

jamesagarside commented Jul 15, 2024

jamesagarside commented Jul 23, 2024

jamesagarside commented Jul 23, 2024

taylor-swanson commented Jul 25, 2024

taylor-swanson left a comment

Choose a reason for hiding this comment

taylor-swanson Jul 25, 2024

Choose a reason for hiding this comment

jamesagarside Jul 26, 2024

Choose a reason for hiding this comment

taylor-swanson Jul 26, 2024

Choose a reason for hiding this comment

taylor-swanson Jul 25, 2024

Choose a reason for hiding this comment

jamesagarside Jul 26, 2024

Choose a reason for hiding this comment

taylor-swanson Jul 25, 2024

Choose a reason for hiding this comment

taylor-swanson Jul 25, 2024

Choose a reason for hiding this comment

elastic-sonarqube bot commented Jul 26, 2024

Quality Gate passed

elasticmachine commented Jul 26, 2024

💚 Build Succeeded

History

botelastic bot commented Aug 25, 2024

botelastic bot commented Sep 24, 2024