[Cloud Security][Dashboard Enhancements] Update Compliance Dashboard Data API to include Benchmarks Cloud Assets Data

[Omolola-Akinleye]

Motivation

To see the Compliance Dashboard View with benchmarks UI, We will need to update the API that returns cloud assets by benchmark data so that I can view an updated and aggregated trend of scores by benchmark ID.

1. Create score_by_benchmark_id aggregation
   The score_by_aggregration will query for the following fields:

• aggregate by rule.benchmark.id and rule.benchmark.version
• passed findings count
• failed findings count
• total findings

2. Add score_by_benchmark_id_version aggregation into findings stats Benchmark score index. See x-pack/plugins/cloud_security_posture/server/tasks/findings_stats_task.ts

esClient.index({
      index: BENCHMARK_SCORE_INDEX_DEFAULT_NS,
      document: {
        policy_template: policyTemplateTrend.key,
        passed_findings: policyTemplateTrend.passed_findings.doc_count,
        failed_findings: policyTemplateTrend.failed_findings.doc_count,
        total_findings: policyTemplateTrend.total_findings.value,
        score_by_cluster_id: clustersStats,
        // add score_by_benchmark_id_version 
      },
    })

3. The logs-cloud_security_posture.scores-default Index Response
   Sample Output

policy_template: policyTemplateTrend.key,
        passed_findings: policyTemplateTrend.passed_findings.doc_count,
        failed_findings: policyTemplateTrend.failed_findings.doc_count,
        total_findings: policyTemplateTrend.total_findings.value,
        score_by_cluster_id: clustersStats,
scores_by_benchmark_id: [
{
   'cis_aws': {
    'v1_3_0':{
      total_findings: 123,
      passed_findings: 100,
      failed_findings: 23
    }
 },
 'cis_gcp': {
   `v1_2_0`:{
    total_findings: 123,
    passed_findings: 100,
    failed_findings: 23
     },
  `v1_3_0`:{
    total_findings: 123,
    passed_findings: 100,
    failed_findings: 23
   }
},
// add benchmark ID records...
]

4. Integrate score_by_benchmark_id trend aggregation with the benchmark aggregation results.
5. Update the ComplianceDashboardData API endpoint/Contract to include a list of benchmarks

interface BenchmarkData {
  meta: {
    benchmarkId: CspFinding['rule']['benchmark']['id'];
    benchmarkVersion: CspFinding['rule']['benchmark']['version'];
    benchmarkName: CspFinding['rule']['benchmark']['name'];
    assetCount: number;
  };
  stats: Stats;
  groupedFindingsEvaluation: GroupedFindingsEvaluation[];
  trend: PostureTrend[];
}`


export interface ComplianceDashboardData {
  stats: Stats;
  groupedFindingsEvaluation: GroupedFindingsEvaluation[];
  clusters: Cluster[];
  trend: PostureTrend[];
  benchmark: BenchmarkData[]
}

Definition of Done

Preview Give feedback

1. Create scores_by_benchmark_id and sublevel benchmark_version aggregation
2. [Cloud Security][Dashboard Enhancements] Create Aggregation query to get benchmarks #170750
   4 of 4
   8.12 candidate Feature:Cloud-Security Team:Cloud Security verified +4
   
3. Update ComplianceDashboad Data API request body/contract to return benchmarks with scores
4. Update and test mapping
5. Create api integration tests

Epic - https://github.com/elastic/security-team/issues/7621

[elasticmachine]

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

[kfirpeled]

I do not think we have to change that much, if we already collect data per account, we can easily add the benchmark details of that account. As later on sum up all the data into a single row.

wdyt? would that work? And if so, it could be a much simpler solution.

[kfirpeled]

Also, please take into account that both the benchmark id and the benchmark version matter here. As we discussed in the kick-off. It is being planned to be able to separate between benchmarks versions once we would support that.

[Omolola-Akinleye]

I do not think we have to change that much, if we already collect data per account, we can easily add the benchmark details of that account. As later on sum up all the data into a single row.

wdyt? would that work? And if so, it could be a much simpler solution.

@kfirpeled Do you mind elaborating? Correct me if I'm wrong but I thought the Benchmark section Compliance scores will focus on collecting scores data per benchmark. If we continue collecting data per account and add benchmark id and rule version, wouldn't still see the same scores?

Here is example if i add benchmark id and version to each account.

"score_by_cluster_id": {
            "439975565995": {
              "total_findings": 1408,
              "passed_findings": 228,
              "failed_findings": 1180,
              "benchmark_id": 'cis_gcp',
              "benchmark_version: "1.3.0"
            },
            "704479110758": {
              "total_findings": 2000,
              "passed_findings": 1000,
              "failed_findings": 1000
               "benchmark_id": 'cis_aws',
              "benchmark_version: "1.5.0"
            },
             "704479110754": {
              "total_findings": 3000,
              "passed_findings":2000,
              "failed_findings": 1000
               "benchmark_id": 'cis_aws',
              "benchmark_version: "1.5.0"
            },
           "704479110757": {
              "total_findings": 3000,
              "passed_findings":2000,
              "failed_findings": 1000
               "benchmark_id": 'cis_aws',
              "benchmark_version: "1.6.0"
            }

vs a separate benchmark scores

"score_by_benchmark_id": {
            "cis_gcp_v1.2.0": {
              "total_findings": 1408,
              "passed_findings": 228,
              "failed_findings": 1180,
            },
            "cis_aws_v1.5.0": {
              "total_findings": 5000,
              "passed_findings": 3000
              "failed_findings": 2000
            },
             "cis_aws_v1.6.0": {
              "total_findings": 3000
              "passed_findings": 1000,
              "failed_findings":2000
            }```

[animehart]

2 Benchmarks with Same ID but Different version shows up as different element. Verified benchmark id, benchmark version, failed findings, passed findings, resource type are shown in the Benchmark response and can also be seen in the UI : Verified