Skip to content

Commit

Permalink
[SLO] multi group by - replace instanceId runtime type with script pr…
Browse files Browse the repository at this point in the history
…ocessor (#179864)

## Summary

Removes the runtime type for generating the instanceId for multi group
by, in favor of utilizing a script processor to generate the instanceId.

This was done to improve the aggregation speed for the transform
aggregation. Results from search profiler suggest a 3x reduction in
overall aggregation speed.

This PR also introduces the concept of `SLO_RESOURCE_VERSION_MAJOR` in
order for us to bump resource versions for additive changes like the
change present in this PR.

Updating the `instanceId` to utilize a script processor can break search
for existing SLOs. The update is only intended to apply to newly created
or updated SLOs.

To account for this, it's important that this feature is tested against
currently stable SLOs, both with and without a group by

### Testing

1. The most faithful test would be going from 8.13.1 to this branch.
However going from main to this branch is acceptable as well.
2. Checkout tag v8.13.1 (or main). Create an SLO with a group by and
another SLO without a group by. Wait for both to calculate and be
stable.
3. Check out this PR.
4. Ensure your previous SLOs remain stable
5. Create a new SLO with a group by. Wait for the calculations to occur
and ensure everything is working as expected.
6. Create a new SLO without a group by. Wait for the calculations to
occur and ensure everything is working as expected.

---------

Co-authored-by: Chris Cowan <[email protected]>
Co-authored-by: Kibana Machine <[email protected]>
  • Loading branch information
3 people authored Apr 10, 2024
1 parent 19c697d commit 8564b7c
Show file tree
Hide file tree
Showing 17 changed files with 281 additions and 340 deletions.
7 changes: 4 additions & 3 deletions x-pack/plugins/observability_solution/slo/common/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ export const LOW_PRIORITY_ACTION = {
};

export const SLO_MODEL_VERSION = 2;
export const SLO_RESOURCES_VERSION = 3;
export const SLO_RESOURCES_VERSION = 3.1;
export const SLO_RESOURCES_VERSION_MAJOR = 3;

export const SLO_COMPONENT_TEMPLATE_MAPPINGS_NAME = '.slo-observability.sli-mappings';
export const SLO_COMPONENT_TEMPLATE_SETTINGS_NAME = '.slo-observability.sli-settings';
Expand All @@ -51,7 +52,7 @@ export const SLO_INDEX_TEMPLATE_NAME = '.slo-observability.sli';
export const SLO_INDEX_TEMPLATE_PATTERN = `.slo-observability.sli-*`;

export const SLO_DESTINATION_INDEX_NAME = `.slo-observability.sli-v${SLO_RESOURCES_VERSION}`;
export const SLO_DESTINATION_INDEX_PATTERN = `.slo-observability.sli-v${SLO_RESOURCES_VERSION}*`;
export const SLO_DESTINATION_INDEX_PATTERN = `.slo-observability.sli-v${SLO_RESOURCES_VERSION_MAJOR}*`;

export const SLO_INGEST_PIPELINE_NAME = `.slo-observability.sli.pipeline-v${SLO_RESOURCES_VERSION}`;
export const SLO_INGEST_PIPELINE_INDEX_NAME_PREFIX = `.slo-observability.sli-v${SLO_RESOURCES_VERSION}.`;
Expand All @@ -63,7 +64,7 @@ export const SLO_SUMMARY_INDEX_TEMPLATE_PATTERN = `.slo-observability.summary-*`

export const SLO_SUMMARY_DESTINATION_INDEX_NAME = `.slo-observability.summary-v${SLO_RESOURCES_VERSION}`; // store the temporary summary document generated by transform
export const SLO_SUMMARY_TEMP_INDEX_NAME = `.slo-observability.summary-v${SLO_RESOURCES_VERSION}.temp`; // store the temporary summary document
export const SLO_SUMMARY_DESTINATION_INDEX_PATTERN = `.slo-observability.summary-v${SLO_RESOURCES_VERSION}*`; // include temp and non-temp summary indices
export const SLO_SUMMARY_DESTINATION_INDEX_PATTERN = `.slo-observability.summary-v${SLO_RESOURCES_VERSION_MAJOR}*`; // include temp and non-temp summary indices

export const getSLOTransformId = (sloId: string, sloRevision: number) =>
`slo-${sloId}-${sloRevision}`;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,50 @@ export const getSLOPipelineTemplate = (id: string, indexNamePrefix: string) => (
date_formats: ['UNIX_MS', 'ISO8601', "yyyy-MM-dd'T'HH:mm:ss.SSSXX"],
},
},
{
script: {
description: 'Generated the instanceId field for SLO rollup data',
source: `
// This function will recursively collect all the values of a HashMap of HashMaps
Collection collectValues(HashMap subject) {
Collection values = new ArrayList();
// Iterate through the values
for(Object value: subject.values()) {
// If the value is a HashMap, recurse
if (value instanceof HashMap) {
values.addAll(collectValues((HashMap) value));
} else {
values.add(String.valueOf(value));
}
}
return values;
}
// Create the string builder
StringBuilder instanceId = new StringBuilder();
if (ctx["slo"]["groupings"] == null) {
ctx["slo"]["instanceId"] = "*";
} else {
// Get the values as a collection
Collection values = collectValues(ctx["slo"]["groupings"]);
// Convert to a list and sort
List sortedValues = new ArrayList(values);
Collections.sort(sortedValues);
// Create comma delimited string
for(String instanceValue: sortedValues) {
instanceId.append(instanceValue);
instanceId.append(",");
}
// Assign the slo.instanceId
ctx["slo"]["instanceId"] = instanceId.length() > 0 ? instanceId.substring(0, instanceId.length() - 1) : "*";
}
`,
},
},
],
_meta: {
description: 'Ingest pipeline for SLO rollup data',
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ describe('slo transform template', () => {
},
defer_validation: true,
_meta: {
version: 3,
version: 3.1,
managed: true,
managed_by: 'observability',
},
Expand Down Expand Up @@ -247,7 +247,7 @@ describe('slo transform template', () => {
},
defer_validation: true,
_meta: {
version: 3,
version: 3.1,
managed: true,
managed_by: 'observability',
},
Expand Down

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 8564b7c

Please sign in to comment.