[Entity Analytics] Add Field Retention Enrich Policy and Ingest Pipeline to Entity Engine

[hop-dev]

Summary

Add the "Ouroboros" part of the entity engine:

• an enrich policy is created for each engine
• the enrich policy is executed every 30s by a kibana task, this will be 1h once we move to a 24h lookback
• create an ingest pipeline for the latest which performs the specified field retention operations (for more detail see below)

Expand for example host entity

``` { "@timestamp": "2024-10-01T12:10:46.000Z", "host": { "name": "host9", "hostname": [ "host9" ], "domain": [ "test.com" ], "ip": [ "1.1.1.1", "1.1.1.2", "1.1.1.3" ], "risk": { "calculated_score": "70.0", "calculated_score_norm": "27.00200653076172", "calculated_level": "Low" }, "id": [ "1234567890abcdef" ], "type": [ "server" ], "mac": [ "AA:AA:AA:AA:AA:AB", "aa:aa:aa:aa:aa:aa", "AA:AA:AA:AA:AA:AC" ], "architecture": [ "x86_64" ] }, "asset": { "criticality": "low_impact" }, "entity": { "name": "host9", "id": "kP/jiFHWSwWlO7W0+fGWrg==", "source": [ "risk-score.risk-score-latest-default", ".asset-criticality.asset-criticality-default", ".ds-logs-testlogs1-default-2024.10.01-000001", ".ds-logs-testlogs2-default-2024.10.01-000001", ".ds-logs-testlogs3-default-2024.10.01-000001" ], "type": "host" } } ```

Field retention operators

First some terminology:

• latest value - the value produced by the transform which represents the latest vioew of a given field in the transform lookback period
• enrich value - the value added to the document by the enrich policy, this represents the last value of a field outiside of the transform lookback window

We hope that this will one day be merged into the entity manager framework so I've tried to abstract this as much as possible. A field retention operator specifies how we should choose a value for a field when looking at the latest value and the enrich value.

Collect values

Collect unique values in an array, first taking from the latest values and then filling with enrich values up to maxLength.

{
  operation: 'collect_values',
  field: 'host.ip',
  maxLength: 10
}

Prefer newest value

Choose the latest value if present, otherwise choose the enrich value.

{
  operation: 'prefer_newest_value',
  field: 'asset.criticality'
}

Prefer oldest value

Choose the enrich value if it is present, otherwise choose latest.

{
  operation: 'prefer_oldest_value',
  field: 'first_seen_timestamp'
}

Test instructions

We currently require extra permissions for the kibana system user for this to work, so we must

1. Get Elasticsearch running from source

This prototype requires a custom branch of elasticsearch in order to give the kibana system user more privileges.

Step 1 - Clone the prototype branch

The elasticsearch branch is at https://github.com/elastic/elasticsearch/tree/entity-store-permissions.

Or you can use github command line to checkout my draft PR:

gh pr checkout 113942

Step 2 - Install Java

Install homebrew if you do not have it.

brew install openjdk@21
sudo ln -sfn /opt/homebrew/opt/openjdk@21/libexec/openjdk.jdk /Library/Java/JavaVirtualMachines/openjdk-21.jdk

Step 3 - Run elasticsearch

This makes sure your data stays between runs of elasticsearch, and that you have platinum license features

./gradlew run --data-dir /tmp/elasticsearch-repo --preserve-data -Drun.license_type=trial

2. Get Kibana Running

Step 1 - Connect kibana to elasticsearch

Set this in your kibana config:

elasticsearch.username: elastic-admin
elasticsearch.password: elastic-password

Now start kibana and you should have connected to the elasticsearch you made.

3. Initialise entity engine and send data!

• Initialise the host or user engine (or both)

curl -H 'Content-Type: application/json' \
      -X POST \   
      -H 'kbn-xsrf: true' \
      -H 'elastic-api-version: 2023-10-31' \
      -d '{}' \
      http:///elastic:changeme@localhost:5601/api/entity_store/engines/host/init 

• use your favourite data generation tool to create data, maybe https://github.com/elastic/security-documents-generator

[tiansivive]

Maybe a gist with the script?

[jloleysens]

Thanks for answering my question @hop-dev , lgtm!

[tiansivive]

I think we have it ✅

[hop-dev]

@elasticmachine merge upstream

[hop-dev]

@elasticmachine merge upstream

[hop-dev]

@elasticmachine merge upstream

[elasticmachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: c708692

Failed CI Steps

• Jest Tests #19

Test Failures

• [job] [logs] Jest Tests #19 / Header rendering it renders the data providers when show is true

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	20.7MB	20.7MB	-159.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	538	539	+1

Total ESLint disabled count

id	before	after	diff
securitySolution	623	624	+1

History

• 💔 Build #241523 failed 3a3f4e9
• 💔 Build #241510 failed 79b88fc
• 💔 Build #241288 failed 1f09e08
• 💔 Build #241194 failed aa48ca3
• 💔 Build #240847 failed 3fb6d60
• 💔 Build #240300 failed f7b109c7829c5ac0a2db95a54152cc9dd730c2bf

cc @hop-dev

[machadoum]

I am confused. So why do we have it? ❓ 😕 ❓

[machadoum]

Could we make it optional?

[kibanamachine]

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11293803178

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation