Compliance dashboard UI and API #171312

Omolola-Akinleye · 2023-11-15T14:59:50Z

Summary

Summarize your PR. If it involves visual changes include a screenshot or gif.

Add benchmarks to Compliance Dashboard API.
Add score_by_benchmark_id to the Benchmark Scores Index this will show posture stats for each benchmark id
Add benchmark aggregation query using benchmark id and version
Add BWC API versioning
STATS API V1 should show clusters
STATS API V2 should show benchmarks
Add unit tests
Added integration tests with API versioning test cases.

To test PR with API versioning, in Kibana client - x-pack/plugins/cloud_security_posture/public/common/api/use_stats_api.ts

Change version value to 1 to see clusters

http.get<ComplianceDashboardData>(getStatsRoute(CSPM_POLICY_TEMPLATE), { version: '1' }),

Change version value to 2 to see versions

http.get<ComplianceDashboardData>(getStatsRoute(CSPM_POLICY_TEMPLATE), { version: '2' })

elasticmachine · 2023-11-22T01:46:31Z

Pinging @elastic/kibana-cloud-security-posture (Team:Cloud Security)

maxcold · 2023-11-24T09:25:53Z

@Omolola-Akinleye Is the PR ready for review? the build seems to be failing ~~and for some reason I can't access the results of the build~~ due to the type check failures

…ance-dashboard-data-api

Omolola-Akinleye · 2023-11-28T23:54:48Z

@Omolola-Akinleye Is the PR ready for review? the build seems to be failing ~~and for some reason I can't access the results of the build~~ due to the type check failures

@maxcold PR is now ready for review sorry for the hold up.

maxcold

I started with the CR, but couldn't finish it in one go, will continue tomorrow

maxcold · 2023-11-29T15:00:57Z

x-pack/plugins/cloud_security_posture/server/lib/mapping_field_util.ts

@@ -0,0 +1,18 @@
+/*


let's add some basic unit tests for these utils

also maybe some comments on why we need these mapping utils, it's was a bit hard to figure it out just from looking at the code

maxcold · 2023-11-29T15:21:05Z

x-pack/plugins/cloud_security_posture/server/lib/mapping_field_util.ts

+export const MAPPING_VERSION_DELIMITER = '_';
+
+export const toBenchmarkDocFieldKey = (benchmarkId: string, benchmarkVersion: string) => {
+  if (benchmarkVersion.includes(MAPPING_VERSION_DELIMITER))


nit: seems like the linter is ok with it but I'm personally not a big fan of if without {}, it's confusing to read and prone to errors. I'd either have it like

if (benchmarkVersion.includes(MAPPING_VERSION_DELIMITER)) { return `${benchmarkId};${benchmarkVersion.replaceAll('_', '.')}`; } return `${benchmarkId};${benchmarkVersion}`;

or have a ternary in the return

…ance-dashboard-data-api

…inleye/kibana into compliance-dashboard-data-api

CohenIdo · 2023-11-30T10:09:19Z

...ck/plugins/cloud_security_posture/server/routes/compliance_dashboard/compliance_dashboard.ts

+              getStats(logger, esClient, query, pitId, runtimeMappings),
+              getGroupedFindingsEvaluation(logger, esClient, query, pitId, runtimeMappings),
+              getClusters(logger, esClient, query, pitId, runtimeMappings),
+              getTrends(logger, esClient, policyTemplate),


nit: logger usually passes as the last argument in our plugin.

CohenIdo · 2023-11-30T10:34:02Z

x-pack/plugins/cloud_security_posture/server/routes/compliance_dashboard/get_benchmarks.ts

+      if (!Array.isArray(version.aggs_by_benchmark_name.buckets))
+        throw new Error('missing aggs by benchmark name');
+
+      if (version.aggs_by_benchmark_name && version.aggs_by_benchmark_name.buckets.length > 0) {
+        benchmarkName = version.aggs_by_benchmark_name.buckets[0].key;
+      }


let's check if there are missing value at once

Suggested change

if (!Array.isArray(version.aggs_by_benchmark_name.buckets))

throw new Error('missing aggs by benchmark name');

if (version.aggs_by_benchmark_name && version.aggs_by_benchmark_name.buckets.length > 0) {

benchmarkName = version.aggs_by_benchmark_name.buckets[0].key;

}

if (!Array.isArray(version.aggs_by_benchmark_name.buckets))

throw new Error('missing aggs by benchmark name');

if (version.aggs_by_benchmark_name && version.aggs_by_benchmark_name.buckets.length > 0) {

benchmarkName = version.aggs_by_benchmark_name.buckets[0].key;

}

if (!Array.isArray(version.aggs_by_benchmark_name.buckets))

throw new Error('missing aggs by benchmark name');

if (version.aggs_by_benchmark_name && version.aggs_by_benchmark_name.buckets.length > 0) {

benchmarkName = version.aggs_by_benchmark_name.buckets[0].key;

}

if (!Array.isArray(resourcesTypesAggs))

throw new Error('missing aggs by resource type per benchmark');

@CohenIdo I'm not sure what you are asking here. Do you mind elaborating?

CohenIdo · 2023-11-30T10:38:17Z

x-pack/plugins/cloud_security_posture/server/routes/compliance_dashboard/get_benchmarks.ts

+      const resourcesTypesAggs = version.aggs_by_resource_type.buckets;
+      if (!Array.isArray(resourcesTypesAggs))
+        throw new Error('missing aggs by resource type per benchmark');
+      const groupedFindingsEvaluation = getFailedFindingsFromAggs(resourcesTypesAggs);


getFailedFindingsFromAggs is confusing name for a function that returns

{ name: bucket.key, totalFindings: bucket.doc_count, totalFailed, totalPassed, postureScore: calculatePostureScore(totalPassed, totalFailed), }

I know it's not part of the PR, but is a good chance to rename it for more meaningful name.

CohenIdo · 2023-11-30T10:40:28Z

...cloud_security_posture/server/routes/compliance_dashboard/get_grouped_findings_evaluation.ts

+  };
+  aggs_by_benchmark_name: Aggregation<KeyDocCount>;
+}
+export interface FailedFindingsBucket extends KeyDocCount {


Could you please rename the interface name? The bucket not only includes failed findings.

CohenIdo · 2023-11-30T10:47:35Z

x-pack/plugins/cloud_security_posture/server/routes/compliance_dashboard/get_trends.ts

+  score_by_benchmark_id: Record<
+    string,
+    Record<
+      string,
+      {
+        total_findings: number;
+        passed_findings: number;
+        failed_findings: number;
+      }
+    >
+  >;


I'm suggesting to refactor the interface in order to avoid code duplication and using ovject instead of Record:

interface FindingsDetails { total_findings: number; passed_findings: number; failed_findings: number; } interface ScoreByClusterId { [clusterId: string]: FindingsDetails; } interface ScoreByBenchmarkId { [benchmarkId: string]: { [key: string]: FindingsDetails; }; } export interface ScoreTrendDoc { '@timestamp': string; total_findings: number; passed_findings: number; failed_findings: number; score_by_cluster_id: ScoreByClusterId; score_by_benchmark_id: ScoreByBenchmarkId; }

CohenIdo · 2023-11-30T10:55:01Z

x-pack/plugins/cloud_security_posture/server/routes/compliance_dashboard/get_trends.ts

-  // large number that should be sufficient for 24 hours considering we write to the score index every 5 minutes
-  size: 999,
+  // Amount of samples of the last 24 hours (accounting that we take a sample every 5 minutes)
+  size: (24 * 60) / 5,


Let's use please the identical parameter currently utilized in the task manager configuration to stay synchronized with any future modifications. It might necessitate confirming that the calculation result is an integer by rounding the outcome.

Hey @CohenIdo, this is a score trends aggregation query which will return the size of the number of trend documents. I don't see how I can apply the 5m interval here.

…inleye/kibana into compliance-dashboard-data-api

kibana-ci · 2023-11-30T15:39:31Z

💚 Build Succeeded

Buildkite Build
Commit: a134050

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`cloudSecurityPosture`	432.1KB	434.5KB	+2.3KB

History

💔 Build #180274 failed 0591bf5
💔 Build #180237 failed f8111c5
💔 Build #180215 failed f6ad4ae
💛 Build #180105 was flaky df1a906
💚 Build #179821 succeeded 0e2c06a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @Omolola-Akinleye

Omolola-Akinleye · 2023-11-30T16:52:34Z

Addressed PR changes @CohenIdo @maxcold

opauloh

LGTM, just some nits that can be addressed in the future

opauloh · 2023-11-30T17:16:45Z

x-pack/plugins/cloud_security_posture/server/routes/compliance_dashboard/get_trends.ts

-  // large number that should be sufficient for 24 hours considering we write to the score index every 5 minutes
-  size: 999,
+  // Amount of samples of the last 24 hours (accounting that we take a sample every 5 minutes)
+  size: (24 * 60) / CSPM_FINDINGS_STATS_INTERVAL,


nice improvement 🚀

opauloh · 2023-11-30T17:18:50Z

x-pack/plugins/cloud_security_posture/server/tasks/findings_stats_task.ts


 const CSPM_FINDINGS_STATS_TASK_ID = 'cloud_security_posture-findings_stats';
 const CSPM_FINDINGS_STATS_TASK_TYPE = 'cloud_security_posture-stats_task';
-const CSPM_FINDINGS_STATS_INTERVAL = '5m';
+export const CSPM_FINDINGS_STATS_INTERVAL = 5;


nit: move that to the common/constants.ts file

opauloh · 2023-11-30T17:21:06Z

...urity_posture/public/pages/compliance_dashboard/compliance_charts/compliance_score_chart.tsx

+  </>
+);
+
+const NonCompactPercentageLabels = ({


nit: this could be called PercentageLabels instead of NonCompactPercentageLabels

## Summary Summarize your PR. If it involves visual changes include a screenshot or gif. Address nit comments from [Compliance Dashboard UI/API PR](#171312)

Omolola-Akinleye added 3 commits November 14, 2023 17:12

scores by benchmark aggs

09a00e0

add benchmark aggregation query

81b7c2c

first pass compliance dashboard api update

190f5b8

Omolola-Akinleye linked an issue Nov 15, 2023 that may be closed by this pull request

[Cloud Security][Dashboard Enhancements] Update Compliance Dashboard Data API to include Benchmarks Cloud Assets Data #170795

Closed

Omolola-Akinleye added 3 commits November 20, 2023 16:59

fix benchmarks api

eaebc40

add utils for mapping field conversion

58485f1

add benchmark tests and unit tests

f91fd17

Omolola-Akinleye self-assigned this Nov 22, 2023

Omolola-Akinleye added Team:Cloud Security Cloud Security team related v8.12.0 release_note:skip Skip the PR/issue when compiling release notes labels Nov 22, 2023

Omolola-Akinleye marked this pull request as ready for review November 22, 2023 01:46

Omolola-Akinleye requested a review from a team as a code owner November 22, 2023 01:46

add logger and tests

c03ee4e

Omolola-Akinleye added 2 commits November 27, 2023 09:51

add remove the unecessary mock items

f33f65e

Merge branch 'main' of github.com:Omolola-Akinleye/kibana into compli…

a2d884d

…ance-dashboard-data-api

Omolola-Akinleye force-pushed the compliance-dashboard-data-api branch from c3934a9 to a2d884d Compare November 27, 2023 14:52

Omolola-Akinleye added 3 commits November 27, 2023 10:30

add kspm test case

e4c84fe

remove dynamic property

605c38f

add versioning and update integration tests

622da00

Merge branch 'main' into compliance-dashboard-data-api

0e2c06a

Omolola-Akinleye requested review from opauloh, JordanSh, CohenIdo and maxcold November 29, 2023 15:36

Merge branch 'main' into compliance-dashboard-data-api

df1a906

maxcold reviewed Nov 29, 2023

View reviewed changes

add dashboard ui with benchmarks

1ce83b7

This was linked to issues Nov 29, 2023

[Cloud Security][Dashboard Enhancements] Benchmark Table Styles #170796

Closed

[Cloud Security][Dashboard Enhancements] Metadata Footer Logos/Tooltips #170802

Closed

Merge branch 'main' of github.com:Omolola-Akinleye/kibana into compli…

bde5b2e

…ance-dashboard-data-api

Omolola-Akinleye linked an issue Nov 29, 2023 that may be closed by this pull request

[Cloud Security][Dashboard Enhancements] Update Summary Section Compliance Score UX #170809

Closed

Omolola-Akinleye added 2 commits November 29, 2023 13:56

fix border styles

92394f7

Merge branch 'compliance-dashboard-data-api' of github.com:Omolola-Ak…

f6ad4ae

…inleye/kibana into compliance-dashboard-data-api

Omolola-Akinleye changed the title ~~Compliance dashboard data api~~ Compliance dashboard data UI and API Nov 29, 2023

fix extra padding

f8111c5

Omolola-Akinleye changed the title ~~Compliance dashboard data UI and API~~ Compliance dashboard UI and API Nov 29, 2023

fix alignment issue

0591bf5

CohenIdo reviewed Nov 30, 2023

View reviewed changes

Omolola-Akinleye added 2 commits November 30, 2023 08:43

address pr comments

09a914a

Merge branch 'main' into compliance-dashboard-data-api

d14b4da

Omolola-Akinleye enabled auto-merge (squash) November 30, 2023 13:58

Omolola-Akinleye added 2 commits November 30, 2023 09:15

use constant interval and make code more readable

8239363

Merge branch 'compliance-dashboard-data-api' of github.com:Omolola-Ak…

a134050

…inleye/kibana into compliance-dashboard-data-api

Omolola-Akinleye requested a review from CohenIdo November 30, 2023 17:13

opauloh approved these changes Nov 30, 2023

View reviewed changes

Omolola-Akinleye merged commit af28af8 into elastic:main Nov 30, 2023
30 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Nov 30, 2023

Omolola-Akinleye mentioned this pull request Dec 7, 2023

[Cloud Security] address pr nit comments #172853

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Compliance dashboard UI and API #171312

Compliance dashboard UI and API #171312

Omolola-Akinleye commented Nov 15, 2023 •

edited

Loading

elasticmachine commented Nov 22, 2023

maxcold commented Nov 24, 2023 •

edited

Loading

Omolola-Akinleye commented Nov 28, 2023

maxcold left a comment

maxcold Nov 29, 2023

maxcold Nov 29, 2023

maxcold Nov 29, 2023

CohenIdo Nov 30, 2023

CohenIdo Nov 30, 2023

Omolola-Akinleye Nov 30, 2023

CohenIdo Nov 30, 2023

CohenIdo Nov 30, 2023

CohenIdo Nov 30, 2023

CohenIdo Nov 30, 2023

Omolola-Akinleye Nov 30, 2023

kibana-ci commented Nov 30, 2023

Omolola-Akinleye commented Nov 30, 2023

opauloh left a comment

opauloh Nov 30, 2023

opauloh Nov 30, 2023

opauloh Nov 30, 2023

Compliance dashboard UI and API #171312

Compliance dashboard UI and API #171312

Conversation

Omolola-Akinleye commented Nov 15, 2023 • edited Loading

Summary

elasticmachine commented Nov 22, 2023

maxcold commented Nov 24, 2023 • edited Loading

Omolola-Akinleye commented Nov 28, 2023

maxcold left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Nov 30, 2023

💚 Build Succeeded

Metrics [docs]

Async chunks

History

Omolola-Akinleye commented Nov 30, 2023

opauloh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Omolola-Akinleye commented Nov 15, 2023 •

edited

Loading

maxcold commented Nov 24, 2023 •

edited

Loading