[Security Solution][CSPM][Fleet] Cannot execute ILM policy delete step on CSPM findings logs

[frconil]

Kibana Version: 8.14.3

Elasticsearch Version: 8.14.3 (but still present in main)

Describe the bug:

The kibana_system role does not have enough permissions to delete the system indices for logs-cloud_security_posture.findings as configured in https://github.com/elastic/integrations/blob/main/packages/cloud_security_posture/data_stream/findings/elasticsearch/ilm/default_policy.json

{
"failed_step": "delete",
"step_info": {
"type": "security_exception",
"reason": "action [indices:admin/delete] is unauthorized for user [kibana_system] with effective roles [kibana_system] on indices [.ds-logs-cloud_security_posture.findings-default-2024.03.15-000001], this action is granted by the index privileges [delete_index,manage,all]"
}
}

This is similar to the issue with endpoint/APM indices #121244

Looking at https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/KibanaOwnedReservedRoleDescriptors.java#L407-L421

it looks like we have defined the permissions for logs-cloud_security_posture.findings_latest-default* but not logs-cloud_security_posture.findings-default*

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[maxcold]

Thanks for reporting the bug, I will bring it to the team!

[Omolola-Akinleye]

Thanks @frconil Looks like our ILM policy have a delete configuration to remove index after 180 days.
logs-cloud_security_posture.findings index is missing delete privileges.

You're correct to fix this issue, we just need to add delete privileges. Before applying a fix, I will talked to the team to understand why we didn't add delete privileges to logs-cloud_security_posture.findings and what's the impact if we decide to apply delete privileges fix.

Some of my team members are on holiday and will be back Monday. I will report back with an update then.

[Omolola-Akinleye]

@frconil After Investigating [Security Solution][CSPM][Fleet] Cannot execute ILM policy delete step on CSPM findings logs bug. We can confirm we are missing privileges for our ILM policy.
Cause: We defined a deletion phase in our policy and we also added the datastream .ds-logs-cloud_security_posture.findings-defaultunder kibana_system role so when index life cycle is over 180 days, then deletion step occurs.

Since we define .ds-logs-cloud_security_posture.findings with kibana system role privilege, we are missing some ILM privilege requirements which include delete privileges and missing a few Fleet privileges installing and upgrade package.

Couple of solutions:
In the short term, I can add privileges to findings datastream to fix this bug. We will work on a fix for the 8.17 release.
In the long term, ILM is not Serverless compatible so we have additonal ticket to investigate migrate ILM to DLM and possibly migrate ILM to DLM.)

[maxcold]

@smriti0321 I want to bring up an option that I mentioned on Slack as well. Maybe we should remove the delete phase of the ILM instead of fixing it. Reasons:

• it looks like it was added without a specific product requirement at the very beginning of the CSP because some other integration had it
• this bug makes me wonder if it ever actually worked, it looks like it never worked
• we don't have any ILM for vulnerability while the data volume there is usually significantly higher
• ILM is not supported on serverless, for serverless we need to use a new method: DLM. My understanding is that eventually, DLM will also be the way to do it in ESS as well

Maybe it makes sense instead of spending time fixing this bug, spend the time on introducing data retention via DLM on both findings and vulnerabilities. If we even want to. Because many other integrations don't have any default ILM/DLM

[smriti0321]

Thanks @maxcold for the details, I do agree on DLM being the path forward, but I need more time and discussion with other stakeholders to understand the roadmap there and @Omolola-Akinleye for investigation and proposal of short term solution.
I have an open question for @frconil, is this reported by a user/customer? Is this issue being reported often by users? Or this was a bug observed internally by our support team?

This will help product decide the path forward.

[frconil]

This was reported by one of our users, yes. I assume further CSPM adoption will highlight similar issues, especially since it takes at least 6 months for the problem to appear.

[smriti0321]

@Omolola-Akinleye we should go ahead with your proposal for short term solution, which is going to fix the issue for this customers and potential customers who have not yet hit 6 month mark as stated by @frconil
Regarding long term plan for DLM vs ILM, we will revisit this in future. I had a discussion with @maxcold and we do have some open questions for product to get insights on future of ILM etc., before we make a decision.

[Omolola-Akinleye]

Okay I will push a fix today! Thanks everyone!