Skip to content

Commit

Permalink
[8.16] [Security Solution][Data Quality Dashboard][Serverless] add st…
Browse files Browse the repository at this point in the history
…art/end time support for latest_results (elastic#199248) (elastic#200635)

# Backport

This will backport the following commits from `main` to `8.16`:
- [[Security Solution][Data Quality Dashboard][Serverless] add start/end
time support for latest_results
(elastic#199248)](elastic#199248)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Karen
Grigoryan","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-07T17:42:31Z","message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (elastic#199248)\n\naddresses elastic#191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70","branchLabelMapping":{"^v9.0.0$":"main","^v8.17.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team:Threat
Hunting","Team:Threat
Hunting:Explore","backport:prev-minor","ci:cloud-deploy","ci:project-deploy-security","v8.17.0"],"number":199248,"url":"https://github.com/elastic/kibana/pull/199248","mergeCommit":{"message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (elastic#199248)\n\naddresses elastic#191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","labelRegex":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/199248","number":199248,"mergeCommit":{"message":"[Security
Solution][Data Quality Dashboard][Serverless] add start/end time support
for latest_results (elastic#199248)\n\naddresses elastic#191053\r\n\r\n- Introduce
`defaultStartTime` and `defaultEndTime` props across data\r\nquality
context and panels for fetching latest_results and align them\r\nwith
serverless default time range of last week\r\n- Update hooks to handle
new time parameters and include them in storage\r\nresults queries.\r\n-
Modify server-side helpers and routes to process and filter
indices\r\nbased on the provided time range.\r\n- Update related tests
to accommodate the new time
parameters.","sha":"1df04aef8d5859507c85a2ad37594075e9054b70"}},{"branch":"8.x","label":"v8.17.0","labelRegex":"^v8.17.0$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/199385","number":199385,"state":"MERGED","mergeCommit":{"sha":"de09e3af76d4bd1ce029830bd866e20b46e3aa9e","message":"[8.x]
[Security Solution][Data Quality Dashboard][Serverless] add start/end
time support for latest_results (elastic#199248) (elastic#199385)\n\n#
Backport\n\nThis will backport the following commits from `main` to
`8.x`:\n- [[Security Solution][Data Quality Dashboard][Serverless] add
start/end\ntime support for
latest_results\n(elastic#199248)](https://github.com/elastic/kibana/pull/199248)\n\n<!---
Backport version: 9.4.3 -->\n\n### Questions ?\nPlease refer to the
[Backport
tool\ndocumentation](https://github.com/sqren/backport)\n\n<!--BACKPORT
[{\"author\":{\"name\":\"Karen\nGrigoryan\",\"email\":\"[email protected]\"},\"sourceCommit\":{\"committedDate\":\"2024-11-07T17:42:31Z\",\"message\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time support\nfor
latest_results (elastic#199248)\\n\\naddresses elastic#191053\\r\\n\\r\\n-
Introduce\n`defaultStartTime` and `defaultEndTime` props across
data\\r\\nquality\ncontext and panels for fetching latest_results and
align them\\r\\nwith\nserverless default time range of last week\\r\\n-
Update hooks to handle\nnew time parameters and include them in
storage\\r\\nresults queries.\\r\\n-\nModify server-side helpers and
routes to process and filter\nindices\\r\\nbased on the provided time
range.\\r\\n- Update related tests\nto accommodate the new
time\nparameters.\",\"sha\":\"1df04aef8d5859507c85a2ad37594075e9054b70\",\"branchLabelMapping\":{\"^v9.0.0$\":\"main\",\"^v8.17.0$\":\"8.x\",\"^v(\\\\d+).(\\\\d+).\\\\d+$\":\"$1.$2\"}},\"sourcePullRequest\":{\"labels\":[\"release_note:skip\",\"v9.0.0\",\"Team:Threat\nHunting\",\"Team:Threat\nHunting:Explore\",\"backport:prev-minor\",\"ci:cloud-deploy\",\"ci:project-deploy-security\"],\"title\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time
support\nfor\nlatest_results\",\"number\":199248,\"url\":\"https://github.com/elastic/kibana/pull/199248\",\"mergeCommit\":{\"message\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time support\nfor
latest_results (elastic#199248)\\n\\naddresses elastic#191053\\r\\n\\r\\n-
Introduce\n`defaultStartTime` and `defaultEndTime` props across
data\\r\\nquality\ncontext and panels for fetching latest_results and
align them\\r\\nwith\nserverless default time range of last week\\r\\n-
Update hooks to handle\nnew time parameters and include them in
storage\\r\\nresults queries.\\r\\n-\nModify server-side helpers and
routes to process and filter\nindices\\r\\nbased on the provided time
range.\\r\\n- Update related tests\nto accommodate the new
time\nparameters.\",\"sha\":\"1df04aef8d5859507c85a2ad37594075e9054b70\"}},\"sourceBranch\":\"main\",\"suggestedTargetBranches\":[],\"targetPullRequestStates\":[{\"branch\":\"main\",\"label\":\"v9.0.0\",\"branchLabelMappingKey\":\"^v9.0.0$\",\"isSourceBranch\":true,\"state\":\"MERGED\",\"url\":\"https://github.com/elastic/kibana/pull/199248\",\"number\":199248,\"mergeCommit\":{\"message\":\"[Security\nSolution][Data
Quality Dashboard][Serverless] add start/end time support\nfor
latest_results (elastic#199248)\\n\\naddresses elastic#191053\\r\\n\\r\\n-
Introduce\n`defaultStartTime` and `defaultEndTime` props across
data\\r\\nquality\ncontext and panels for fetching latest_results and
align them\\r\\nwith\nserverless default time range of last week\\r\\n-
Update hooks to handle\nnew time parameters and include them in
storage\\r\\nresults queries.\\r\\n-\nModify server-side helpers and
routes to process and filter\nindices\\r\\nbased on the provided time
range.\\r\\n- Update related tests\nto accommodate the new
time\nparameters.\",\"sha\":\"1df04aef8d5859507c85a2ad37594075e9054b70\"}}]}]\nBACKPORT-->\n\nCo-authored-by:
Karen Grigoryan <[email protected]>"}}]}] BACKPORT-->
  • Loading branch information
kapral18 authored Nov 18, 2024
1 parent c8b46e8 commit b900fa0
Show file tree
Hide file tree
Showing 25 changed files with 642 additions and 58 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,8 @@ const ContextWrapper: FC<PropsWithChildren<unknown>> = ({ children }) => (
},
]}
setSelectedIlmPhaseOptions={jest.fn()}
defaultStartTime="now-7d"
defaultEndTime="now"
>
{children}
</DataQualityProvider>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,8 @@ export interface DataQualityProviderProps {
ilmPhases: string[];
selectedIlmPhaseOptions: EuiComboBoxOptionOption[];
setSelectedIlmPhaseOptions: (options: EuiComboBoxOptionOption[]) => void;
defaultStartTime: string;
defaultEndTime: string;
}

const DataQualityContext = React.createContext<DataQualityProviderProps | undefined>(undefined);
Expand All @@ -67,6 +69,8 @@ export const DataQualityProvider: React.FC<PropsWithChildren<DataQualityProvider
ilmPhases,
selectedIlmPhaseOptions,
setSelectedIlmPhaseOptions,
defaultStartTime,
defaultEndTime,
}) => {
const value = useMemo(
() => ({
Expand All @@ -90,6 +94,8 @@ export const DataQualityProvider: React.FC<PropsWithChildren<DataQualityProvider
ilmPhases,
selectedIlmPhaseOptions,
setSelectedIlmPhaseOptions,
defaultStartTime,
defaultEndTime,
}),
[
httpFetch,
Expand All @@ -112,6 +118,8 @@ export const DataQualityProvider: React.FC<PropsWithChildren<DataQualityProvider
ilmPhases,
selectedIlmPhaseOptions,
setSelectedIlmPhaseOptions,
defaultStartTime,
defaultEndTime,
]
);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ const ContextWrapper: React.FC<{ children: React.ReactNode; isILMAvailable: bool
},
]}
setSelectedIlmPhaseOptions={jest.fn()}
defaultStartTime={'now-7d'}
defaultEndTime={'now'}
>
{children}
</DataQualityProvider>
Expand Down Expand Up @@ -159,6 +161,8 @@ describe('useIlmExplain', () => {
},
]}
setSelectedIlmPhaseOptions={jest.fn()}
defaultStartTime={'now-7d'}
defaultEndTime={'now'}
>
{children}
</DataQualityProvider>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,8 @@ const ContextWrapper: FC<PropsWithChildren<unknown>> = ({ children }) => (
},
]}
setSelectedIlmPhaseOptions={jest.fn()}
defaultStartTime={'now-7d'}
defaultEndTime={'now'}
>
{children}
</DataQualityProvider>
Expand Down Expand Up @@ -119,6 +121,8 @@ const ContextWrapperILMNotAvailable: FC<PropsWithChildren<unknown>> = ({ childre
},
]}
setSelectedIlmPhaseOptions={jest.fn()}
defaultStartTime={'now-7d'}
defaultEndTime={'now'}
>
{children}
</DataQualityProvider>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,10 @@ import { notificationServiceMock } from '@kbn/core-notifications-browser-mocks';
import { getHistoricalResultStub } from '../../../../stub/get_historical_result_stub';
import { useStoredPatternResults } from '.';

const startTime = 'now-7d';
const endTime = 'now';
const isILMAvailable = true;

describe('useStoredPatternResults', () => {
const httpFetch = jest.fn();
const mockToasts = notificationServiceMock.createStartContract().toasts;
Expand All @@ -21,7 +25,16 @@ describe('useStoredPatternResults', () => {

describe('when patterns are empty', () => {
it('should return an empty array and not call getStorageResults', () => {
const { result } = renderHook(() => useStoredPatternResults([], mockToasts, httpFetch));
const { result } = renderHook(() =>
useStoredPatternResults({
patterns: [],
toasts: mockToasts,
httpFetch,
isILMAvailable,
startTime,
endTime,
})
);

expect(result.current).toEqual([]);
expect(httpFetch).not.toHaveBeenCalled();
Expand All @@ -45,7 +58,14 @@ describe('useStoredPatternResults', () => {
});

const { result, waitFor } = renderHook(() =>
useStoredPatternResults(patterns, mockToasts, httpFetch)
useStoredPatternResults({
patterns,
toasts: mockToasts,
httpFetch,
isILMAvailable,
startTime,
endTime,
})
);

await waitFor(() => result.current.length > 0);
Expand Down Expand Up @@ -104,5 +124,63 @@ describe('useStoredPatternResults', () => {
},
]);
});

describe('when isILMAvailable is false', () => {
it('should call getStorageResults with startDate and endDate', async () => {
const patterns = ['pattern1-*', 'pattern2-*'];

httpFetch.mockImplementation((path: string) => {
if (path === '/internal/ecs_data_quality_dashboard/results_latest/pattern1-*') {
return Promise.resolve([getHistoricalResultStub('pattern1-index1')]);
}

if (path === '/internal/ecs_data_quality_dashboard/results_latest/pattern2-*') {
return Promise.resolve([getHistoricalResultStub('pattern2-index1')]);
}

return Promise.reject(new Error('Invalid path'));
});

const { result, waitFor } = renderHook(() =>
useStoredPatternResults({
patterns,
toasts: mockToasts,
httpFetch,
isILMAvailable: false,
startTime,
endTime,
})
);

await waitFor(() => result.current.length > 0);

expect(httpFetch).toHaveBeenCalledTimes(2);

expect(httpFetch).toHaveBeenCalledWith(
'/internal/ecs_data_quality_dashboard/results_latest/pattern1-*',
{
method: 'GET',
signal: expect.any(AbortSignal),
version: '1',
query: {
startDate: startTime,
endDate: endTime,
},
}
);
expect(httpFetch).toHaveBeenCalledWith(
'/internal/ecs_data_quality_dashboard/results_latest/pattern2-*',
{
method: 'GET',
signal: expect.any(AbortSignal),
version: '1',
query: {
startDate: startTime,
endDate: endTime,
},
}
);
});
});
});
});
Original file line number Diff line number Diff line change
Expand Up @@ -10,13 +10,34 @@ import { HttpHandler } from '@kbn/core-http-browser';
import { isEmpty } from 'lodash/fp';

import { DataQualityCheckResult } from '../../../../types';
import { formatResultFromStorage, getStorageResults } from '../../utils/storage';
import {
GetStorageResultsOpts,
formatResultFromStorage,
getStorageResults,
} from '../../utils/storage';

export const useStoredPatternResults = (
patterns: string[],
toasts: IToasts,
httpFetch: HttpHandler
) => {
export interface UseStoredPatternResultsOpts {
patterns: string[];
toasts: IToasts;
httpFetch: HttpHandler;
isILMAvailable: boolean;
startTime: string;
endTime: string;
}

export type UseStoredPatternResultsReturnValue = Array<{
pattern: string;
results: Record<string, DataQualityCheckResult>;
}>;

export const useStoredPatternResults = ({
patterns,
toasts,
httpFetch,
isILMAvailable,
startTime,
endTime,
}: UseStoredPatternResultsOpts): UseStoredPatternResultsReturnValue => {
const [storedPatternResults, setStoredPatternResults] = useState<
Array<{ pattern: string; results: Record<string, DataQualityCheckResult> }>
>([]);
Expand All @@ -28,17 +49,29 @@ export const useStoredPatternResults = (

const abortController = new AbortController();
const fetchStoredPatternResults = async () => {
const requests = patterns.map((pattern) =>
getStorageResults({ pattern, httpFetch, abortController, toasts }).then((results = []) => ({
const requests = patterns.map(async (pattern) => {
const getStorageResultsOpts: GetStorageResultsOpts = {
pattern,
httpFetch,
abortController,
toasts,
};

if (!isILMAvailable) {
getStorageResultsOpts.startTime = startTime;
getStorageResultsOpts.endTime = endTime;
}

return getStorageResults(getStorageResultsOpts).then((results) => ({
pattern,
results: Object.fromEntries(
results.map((storageResult) => [
storageResult.indexName,
formatResultFromStorage({ storageResult, pattern }),
])
),
}))
);
}));
});

const patternResults = await Promise.all(requests);
if (patternResults?.length) {
Expand All @@ -47,7 +80,7 @@ export const useStoredPatternResults = (
};

fetchStoredPatternResults();
}, [httpFetch, patterns, toasts]);
}, [endTime, httpFetch, isILMAvailable, patterns, startTime, toasts]);

return storedPatternResults;
};
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@ describe('useResultsRollup', () => {

const patterns = ['auditbeat-*', 'packetbeat-*'];
const isILMAvailable = true;
const startTime = 'now-7d';
const endTime = 'now';

const useStoredPatternResultsMock = useStoredPatternResults as jest.Mock;

Expand All @@ -52,6 +54,8 @@ describe('useResultsRollup', () => {
patterns,
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand Down Expand Up @@ -94,10 +98,19 @@ describe('useResultsRollup', () => {
patterns: ['auditbeat-*'],
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

expect(useStoredPatternResultsMock).toHaveBeenCalledWith(['auditbeat-*'], toasts, httpFetch);
expect(useStoredPatternResultsMock).toHaveBeenCalledWith({
patterns: ['auditbeat-*'],
toasts,
httpFetch,
isILMAvailable,
startTime,
endTime,
});

expect(result.current.patternRollups).toEqual({
'auditbeat-*': {
Expand All @@ -119,6 +132,8 @@ describe('useResultsRollup', () => {
patterns,
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand All @@ -144,6 +159,8 @@ describe('useResultsRollup', () => {
patterns,
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand Down Expand Up @@ -180,6 +197,8 @@ describe('useResultsRollup', () => {
patterns,
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand Down Expand Up @@ -369,6 +388,8 @@ describe('useResultsRollup', () => {
patterns,
isILMAvailable: false,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand Down Expand Up @@ -532,6 +553,8 @@ describe('useResultsRollup', () => {
patterns,
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand Down Expand Up @@ -592,6 +615,8 @@ describe('useResultsRollup', () => {
patterns,
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand Down Expand Up @@ -654,6 +679,8 @@ describe('useResultsRollup', () => {
patterns: ['packetbeat-*', 'auditbeat-*'],
isILMAvailable,
telemetryEvents: mockTelemetryEvents,
startTime,
endTime,
})
);

Expand Down
Loading

0 comments on commit b900fa0

Please sign in to comment.