Skip to content

Commit

Permalink
[Synthetics] Fix heatmap on monitor detail/history page for very larg…
Browse files Browse the repository at this point in the history 
…e doc counts (elastic#184177)

## Summary

Resolves elastic#180076.

### User-facing updates

#### We can now display the heatmap for "very large" document counts

##### This PR

_note: The first bucket in this image is empty because that's when my
monitor actually started._

<img width="2083" alt="image"
src="https://github.com/elastic/kibana/assets/18429259/72daf378-586e-4c7d-a1b0-a05bf95fb09a">

##### Previously

<img width="1420" alt="image"
src="https://github.com/elastic/kibana/assets/18429259/94f46913-debc-4bce-9794-600241fc4d6f">

#### We display appropriate bucketed data

This is a weird behavior I only noticed once I had made the new query.
For some reason, the chart didn't display all the data it should. This
may be related to the initial issue, but even in ranges less than 10k
docs I was seeing this behavior. I manually checked this by querying for
the bucket spans and ensuring the right number of docs were showing up,
which is what I'd expect, because it's highly unlikely there would be a
bug like this in the date histogram agg.

##### This PR

<img width="1420" alt="image"
src="https://github.com/elastic/kibana/assets/18429259/8620f709-dd4f-4a52-b39d-a3ff778ff7b4">

##### Previously

<img width="1420" alt="image"
src="https://github.com/elastic/kibana/assets/18429259/745fa36d-db35-46a4-be1d-330bed799884">

### Why are you doing this?

We have noticed in a few SDH that in cases where users are querying very
large document tallies, like a month's worth of data for a monitor that
runs every minute, they're not seeing their status heatmap fill up.

~This patch would introduce a multi-stage query mechanism to the pings
query endpoint. The PR is in draft state, so the procedure performs the
query in this manner all the time, but we may want to add a new param to
the endpoint that allows the client to opt into this behavior, or simply
cap at the default 10k documents like it does today.~

~I intend to do a good amount of benchmark testing on this fix locally
and in Cloud before I try to merge it. We may also need to explore
imposing a hard upper limit on the number of stages we are willing to
execute. Right now the query is unbounded and a user could try to query
datasets that would result in Kibana retrieving and trying to respond
with hundreds of MB of data.~

### What changed

This PR modifies the backend query to rely on a Date Histogram agg,
rather than querying the documents directly for this purpose. This agg
uses the same interval that the client calculates based on the width of
the page. I have kept the existing time bin logic on the client side,
and modified the function that builds the data structure to pour the
buckets from the histogram into the bins that we serve the chart that
the user sees.

One drawback is because the buckets are computed by Elasticsearch, we
need to make API calls to retrieve more data, whereas on resizes
previously, all the data was already present and would get re-bucketed
on the client. I think this is acceptable because of the ability to
handle larger datasets and given that the query should be very fast
still.

Additionally:

- I have made additional modifications to the frontend logic to prevent
unnecessary API calls. Before, the component would always make at least
two calls on an initial load. I have also switched from `lodash`
`throttle` to `useDebounce` from `react-use`, as this seems to more
properly drop intermediate resize events where we would want to skip
hitting the API.
- I have also changed the render logic. We used to use a default value
to build out an initial set of time bins. Unfortunately, with all these
changes in place, this results in a weird behavior where we show an
empty scale while waiting for data, and the bins then snap to the proper
size once the element's width has been determined and the bins get
scaled out properly. Instead of this, I skip rendering the chart until
the initial width is available via a `ref` applied to the wrapper
element. At that point, we send the init width to the hook and use that
value for our initial query and bin calculations, so we only ever show
bins on the charts that will correspond to the bucketed data we
eventually display.
- I added a progress indicator to the wrapper element so the user
receives an indication that the data is being refreshed.
- I added a quiet fetch similar to how the overview page works, so that
after an initial render the chart will not have its data dropped until
we have new data to replace it with. Along those lines, the hook now
watches the page location and if it changes (i.e. to from History to
Overview), state management will destroy the heatmap data so we don't
have a flicker of other data during a new fetch.

##  Testing this PR

Unfortunately, to truly test this PR you'd need a monitor with over 10k
documents, as that's the criteria specified in the initial issue. I did
this by running a monitor over about 10 days on a 1-run-per-minute
schedule. You could create a cloud deployment from this PR, or create
dummy documents in some way (this can be hard to verify though). I am
pretty confident this works and fixes the issue based on my real-world
testing.

---------

Co-authored-by: shahzad31 <[email protected]>
(cherry picked from commit 50f8bd6)

# Conflicts:
#	x-pack/plugins/observability_solution/synthetics/common/constants/synthetics/rest_api.ts
#	x-pack/plugins/observability_solution/synthetics/public/apps/synthetics/state/root_effect.ts
#	x-pack/plugins/observability_solution/synthetics/server/routes/index.ts
#	x-pack/plugins/observability_solution/synthetics/server/routes/pings/get_ping_statuses.ts
  • Loading branch information
@justinkambic
justinkambic committed Oct 14, 2024
1 parent 74f0571 commit 8fbf86b
Show file tree
Hide file tree
Showing 26 changed files with 515 additions and 530 deletions.
2 changes: 2 additions & 0 deletions x-pack/plugins/observability_solution/synthetics/common/constants/synthetics/rest_api.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,8 @@ export enum SYNTHETICS_API_URLS {
SYNTHETICS_OVERVIEW = '/internal/synthetics/overview',
PINGS = '/internal/synthetics/pings',
PING_STATUSES = '/internal/synthetics/ping_statuses',
MONITOR_STATUS_HEATMAP = '/internal/synthetics/ping_heatmap',
OVERVIEW_TRENDS = '/internal/synthetics/overview_trends',
OVERVIEW_STATUS = `/internal/synthetics/overview_status`,
INDEX_SIZE = `/internal/synthetics/index_size`,
AGENT_POLICIES = `/internal/synthetics/agent_policies`,
Expand Down
41 changes: 14 additions & 27 deletions x-pack/plugins/observability_solution/synthetics/common/runtime_types/ping/ping.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -255,40 +255,13 @@ export const PingStateType = t.type({
export type Ping = t.TypeOf<typeof PingType>;
export type PingState = t.TypeOf<typeof PingStateType>;

export const PingStatusType = t.intersection([
t.type({
timestamp: t.string,
docId: t.string,
config_id: t.string,
locationId: t.string,
summary: t.partial({
down: t.number,
up: t.number,
}),
}),
t.partial({
error: PingErrorType,
}),
]);

export type PingStatus = t.TypeOf<typeof PingStatusType>;

export const PingsResponseType = t.type({
total: t.number,
pings: t.array(PingType),
});

export type PingsResponse = t.TypeOf<typeof PingsResponseType>;

export const PingStatusesResponseType = t.type({
total: t.number,
pings: t.array(PingStatusType),
from: t.string,
to: t.string,
});

export type PingStatusesResponse = t.TypeOf<typeof PingStatusesResponseType>;

export const GetPingsParamsType = t.intersection([
t.type({
dateRange: DateRangeType,
Expand All @@ -306,3 +279,17 @@ export const GetPingsParamsType = t.intersection([
]);

export type GetPingsParams = t.TypeOf<typeof GetPingsParamsType>;

export const MonitorStatusHeatmapBucketType = t.type({
doc_count: t.number,
down: t.type({
value: t.number,
}),
up: t.type({
value: t.number,
}),
key: t.number,
key_as_string: t.string,
});

export type MonitorStatusHeatmapBucket = t.TypeOf<typeof MonitorStatusHeatmapBucketType>;
90 changes: 0 additions & 90 deletions .../synthetics/public/apps/synthetics/components/monitor_details/hooks/use_ping_statuses.tsx
View file

This file was deleted.

40 changes: 19 additions & 21 deletions ...s/public/apps/synthetics/components/monitor_details/monitor_status/monitor_status_data.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ import {
COLOR_MODES_STANDARD,
} from '@elastic/eui';
import type { BrushEvent } from '@elastic/charts';
import { PingStatus } from '../../../../../../common/runtime_types';
import { MonitorStatusHeatmapBucket } from '../../../../../../common/runtime_types';

export const SUCCESS_VIZ_COLOR = VISUALIZATION_COLORS[0];
export const DANGER_VIZ_COLOR = VISUALIZATION_COLORS[VISUALIZATION_COLORS.length - 1];
Expand Down Expand Up @@ -114,28 +114,26 @@ export function createTimeBuckets(intervalMinutes: number, from: number, to: num

export function createStatusTimeBins(
timeBuckets: MonitorStatusTimeBucket[],
pingStatuses: PingStatus[]
heatmapData: MonitorStatusHeatmapBucket[]
): MonitorStatusTimeBin[] {
let iPingStatus = 0;
return (timeBuckets ?? []).map((bucket) => {
const currentBin: MonitorStatusTimeBin = {
start: bucket.start,
end: bucket.end,
ups: 0,
downs: 0,
value: 0,
return timeBuckets.map(({ start, end }) => {
const { ups, downs } = heatmapData
.filter(({ key }) => key >= start && key <= end)
.reduce(
(acc, cur) => ({
ups: acc.ups + cur.up.value,
downs: acc.downs + cur.down.value,
}),
{ ups: 0, downs: 0 }
);

return {
start,
end,
ups,
downs,
value: ups + downs === 0 ? 0 : getStatusEffectiveValue(ups, downs),
};
while (
iPingStatus < pingStatuses.length &&
moment(pingStatuses[iPingStatus].timestamp).valueOf() < bucket.end
) {
currentBin.ups += pingStatuses[iPingStatus]?.summary.up ?? 0;
currentBin.downs += pingStatuses[iPingStatus]?.summary.down ?? 0;
currentBin.value = getStatusEffectiveValue(currentBin.ups, currentBin.downs);
iPingStatus++;
}

return currentBin;
});
}

Expand Down
120 changes: 62 additions & 58 deletions ...public/apps/synthetics/components/monitor_details/monitor_status/monitor_status_panel.tsx
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,11 @@
* 2.0.
*/

import React, { useMemo } from 'react';
import React, { useMemo, useRef } from 'react';

import { EuiPanel, useEuiTheme, EuiResizeObserver, EuiSpacer } from '@elastic/eui';
import { EuiPanel, useEuiTheme, EuiResizeObserver, EuiSpacer, EuiProgress } from '@elastic/eui';
import { Chart, Settings, Heatmap, ScaleType, Tooltip, LEGACY_LIGHT_THEME } from '@elastic/charts';
import { i18n } from '@kbn/i18n';
import { usePingStatusesIsLoading } from '../hooks/use_ping_statuses';
import { MonitorStatusHeader } from './monitor_status_header';
import { MonitorStatusCellTooltip } from './monitor_status_cell_tooltip';
import { MonitorStatusLegend } from './monitor_status_legend';
Expand All @@ -32,9 +31,9 @@ export const MonitorStatusPanel = ({
onBrushed,
}: MonitorStatusPanelProps) => {
const { euiTheme, colorMode } = useEuiTheme();
const { timeBins, handleResize, getTimeBinByXValue, xDomain, intervalByWidth } =
useMonitorStatusData({ from, to });
const isPingStatusesLoading = usePingStatusesIsLoading();
const initialSizeRef = useRef<HTMLDivElement | null>(null);
const { loading, timeBins, handleResize, getTimeBinByXValue, xDomain, minsPerBin } =
useMonitorStatusData({ from, to, initialSizeRef });

const heatmap = useMemo(() => {
return getMonitorStatusChartTheme(euiTheme, brushable);
Expand All @@ -53,61 +52,66 @@ export const MonitorStatusPanel = ({

<EuiSpacer size="m" />

<EuiResizeObserver onResize={handleResize}>
{(resizeRef) => (
<div ref={resizeRef}>
<Chart
size={{
height: 60,
}}
>
<Tooltip
customTooltip={({ values }) => (
<MonitorStatusCellTooltip
timeBin={getTimeBinByXValue(values?.[0]?.datum?.x)}
isLoading={isPingStatusesLoading}
<div ref={initialSizeRef}>
<EuiResizeObserver onResize={(e) => handleResize(e)}>
{(resizeRef) => (
<div ref={resizeRef}>
{minsPerBin && (
<Chart
size={{
height: 60,
}}
>
<Tooltip
customTooltip={({ values }) => (
<MonitorStatusCellTooltip
timeBin={getTimeBinByXValue(values?.[0]?.datum?.x)}
isLoading={loading}
/>
)}
/>
)}
/>
<Settings
showLegend={false}
xDomain={xDomain}
theme={{ heatmap }}
// TODO connect to charts.theme service see src/plugins/charts/public/services/theme/README.md
baseTheme={LEGACY_LIGHT_THEME}
onBrushEnd={(brushArea) => {
onBrushed?.(getBrushData(brushArea));
}}
locale={i18n.getLocale()}
/>
<Heatmap
id="monitor-details-monitor-status-chart"
colorScale={{
type: 'bands',
bands: getColorBands(euiTheme, colorMode),
}}
data={timeBins}
xAccessor={(timeBin) => timeBin.end}
yAccessor={() => 'T'}
valueAccessor={(timeBin) => timeBin.value}
valueFormatter={(d) => d.toFixed(2)}
xAxisLabelFormatter={getXAxisLabelFormatter(intervalByWidth)}
timeZone="UTC"
xScale={{
type: ScaleType.Time,
interval: {
type: 'calendar',
unit: 'm',
value: intervalByWidth,
},
}}
/>
</Chart>
</div>
)}
</EuiResizeObserver>
<Settings
showLegend={false}
xDomain={xDomain}
theme={{ heatmap }}
// TODO connect to charts.theme service see src/plugins/charts/public/services/theme/README.md
baseTheme={LEGACY_LIGHT_THEME}
onBrushEnd={(brushArea) => {
onBrushed?.(getBrushData(brushArea));
}}
locale={i18n.getLocale()}
/>
<Heatmap
id="monitor-details-monitor-status-chart"
colorScale={{
type: 'bands',
bands: getColorBands(euiTheme, colorMode),
}}
data={timeBins}
xAccessor={({ end }) => end}
yAccessor={() => 'T'}
valueAccessor={(timeBin) => timeBin.value}
valueFormatter={(d) => d.toFixed(2)}
xAxisLabelFormatter={getXAxisLabelFormatter(minsPerBin)}
timeZone="UTC"
xScale={{
type: ScaleType.Time,
interval: {
type: 'calendar',
unit: 'm',
value: minsPerBin,
},
}}
/>
</Chart>
)}
</div>
)}
</EuiResizeObserver>
</div>

<MonitorStatusLegend brushable={brushable} />
{loading && <EuiProgress size="xs" color="accent" />}
</EuiPanel>
);
};
Loading

0 comments on commit 8fbf86b

Please sign in to comment.