[SecuritySolution][SIEM migrations] Implement background task API

[semd]

Summary

It implements the background task to execute the rule migrations and the API to manage them. It also contains a basic implementation of the langGraph agent workflow that will perform the migration using generative AI.

Note

This feature needs siemMigrationsEnabled experimental flag enabled to work. Otherwise, the new API routes won't be registered, and the SiemRuleMigrationsService setup won't be called. So no migration task code can be reached, and no data stream/template will be installed to ES.

The rule migration task implementation:

• Retrieve a batch of N rule migration documents (50 rules initially, we may change that later) with status: pending.
• Update those documents to status: processing.
• Execute the migration for each of the N migrations in parallel.
• If there is any error update the document with status: error.
• For each rule migration that finishes we set the result to the storage, and also update status: finished.
• When all the batch of rules is finished the task will check if there are still migration documents with status: pending if so it will process the next batch with a delay (10 seconds initially, we may change that later).
• If the task is stopped (via API call or server shut-down), we do a bulk update for all the status: processing documents back to status: pending.

Task API

• POST /internal/siem_migrations/rules (implemented here) -> Creates the migration on the backend and stores the original rules. It returns the migration_id
• GET /internal/siem_migrations/rules/stats -> Retrieves the stats for all the existing migrations, aggregated by migration_id.
• GET /internal/siem_migrations/rules/{migration_id} -> Retrieves all the migration rule documents of a specific migration.
• PUT /internal/siem_migrations/rules/{migration_id}/start -> Starts the background task for a specific migration.
• GET /internal/siem_migrations/rules/{migration_id}/stats -> Retrieves the stats of a specific migration task. The UI will do polling to this endpoint.
• PUT /internal/siem_migrations/rules/{migration_id}/stop -> Stops the execution of a specific migration running task. When a migration is stopped, the executing task is aborted and all the rules in the batch being processed are moved back to pending, all finished rules will remain stored. When the Kibana server shuts down all the running migrations are stopped automatically. To resume the migration we can call {migration_id}/start again and it will take it from the same rules batch it was left.

Stats (UI polling) response example:

{
    "status": "running",
    "rules": {
        "total": 34,
        "finished": 20,
        "pending": 4,
        "processing": 10,
        "failed": 0
    },
    "last_updated_at": "2024-10-29T15:04:49.618Z"
}

LLM agent Graph

The initial implementation of the agent graph that is executed per rule:

The first node tries to match the original rule with an Elastic prebuilt rule. If it does not succeed, the second node will try to translate the query as a custom rule using the ES|QL knowledge base, this composes previous PoCs:

• [Security Solution][Siem Migrations] PoC prebuilt rule matching API  #193900
• [Security Solution][Siem Migrations] POC: SPL rule translation to ES|QL #196651

Testing locally

Enable the flag

xpack.securitySolution.enableExperimental: ['siemMigrationsEnabled']

cURL request examples:

Rules migration `create` POST request

curl --location --request POST 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '[
    {
        "id": "f8c325ea-506e-4105-8ccf-da1492e90115",
        "vendor": "splunk",
        "title": "Linux Auditd Add User Account Type",
        "description": "The following analytic detects the suspicious add user account type. This behavior is critical for a SOC to monitor because it may indicate attempts to gain unauthorized access or maintain control over a system. Such actions could be signs of malicious activity. If confirmed, this could lead to serious consequences, including a compromised system, unauthorized access to sensitive data, or even a wider breach affecting the entire network. Detecting and responding to these signs early is essential to prevent potential security incidents.",
        "query": "sourcetype=\"linux:audit\" type=ADD_USER \n| rename hostname as dest \n| stats count min(_time) as firstTime max(_time) as lastTime by exe pid dest res UID type \n| `security_content_ctime(firstTime)` \n| `security_content_ctime(lastTime)`\n| search *",
        "query_language":"spl",
        "mitre_attack_ids": [
            "T1136"
        ]
    },
    {
        "id": "7b87c556-0ca4-47e0-b84c-6cd62a0a3e90",
        "vendor": "splunk",
        "title": "Linux Auditd Change File Owner To Root",
        "description": "The following analytic detects the use of the '\''chown'\'' command to change a file owner to '\''root'\'' on a Linux system. It leverages Linux Auditd telemetry, specifically monitoring command-line executions and process details. This activity is significant as it may indicate an attempt to escalate privileges by adversaries, malware, or red teamers. If confirmed malicious, this action could allow an attacker to gain root-level access, leading to full control over the compromised host and potential persistence within the environment.",
        "query": "`linux_auditd` `linux_auditd_normalized_proctitle_process`\r\n| rename host as dest \r\n| where LIKE (process_exec, \"%chown %root%\") \r\n| stats count min(_time) as firstTime max(_time) as lastTime by process_exec proctitle normalized_proctitle_delimiter dest \r\n| `security_content_ctime(firstTime)` \r\n| `security_content_ctime(lastTime)`\r\n| `linux_auditd_change_file_owner_to_root_filter`",
        "query_language": "spl",
        "mitre_attack_ids": [
            "T1222"
        ]
    }
]'

Rules migration `start` task request

• Assuming the connector azureOpenAiGPT4o is already created in the local environment.
• Using the {{migration_id}} from the first POST request response

curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/start' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' \
--header 'Content-Type: application/json' \
--data '{
    "connectorId": "azureOpenAiGPT4o"
}'

Rules migration `stop` task request

• Using the {{migration_id}} from the first POST request response.

curl --location --request PUT 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stop' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' 

Rules migration task `stats` request

• Using the {{migration_id}} from the first POST request response.

curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}/stats' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' 

Rules migration rules documents request

• Using the {{migration_id}} from the first POST request response.

curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/{{migration_id}}' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' 

Rules migration all stats request

curl --location --request GET 'http://elastic:changeme@localhost:5601/internal/siem_migrations/rules/stats' \
--header 'kbn-xsrf;' \
--header 'x-elastic-internal-origin: security-solution' \
--header 'elastic-api-version: 1' 

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[angorayc]

Can we have some comments about how StateGraph works? A url link will do too.

[semd]

Of course @angorayc, the LangGraph TS docs entry point is here and the StateGraph reference is here.
If you want to check the Python reference, which is better organized, it's here.

[gsoldevila]

LGTM inference is ai-infra so part of platform group (valid dependency).

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b5008c9

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
securitySolution	119	120	+1

Unknown metric groups

API count

id	before	after	diff
securitySolution	187	188	+1

History

• 💚 Build #248729 succeeded 0c119ed
• 💔 Build #248245 failed d3ac344
• 💔 Build #248172 failed 5ff35b1

cc @semd

[kibanamachine]

Starting backport for target branches: 8.18, 8.x

https://github.com/elastic/kibana/actions/runs/11708722593

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.18	The branch "8.18" does not exist
❌	8.x	Backport failed because of merge conflicts You might need to backport the following PRs to 8.x: - [SecuritySolution] Check user permissions before initialising entity engine (#198661)

Manual backport

To create the backport manually run:

node scripts/backport --pr 197997

Questions ?

Please refer to the Backport tool documentation

[semd]

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation