[Dataset quality] Support failure store #199806

yngrdyn · 2024-11-12T14:46:59Z

Closes https://github.com/elastic/logs-dev/issues/183 and https://github.com/elastic/logs-dev/issues/184

Summary

This PR aims to show to the user failed docs % in dataset quality table. The following acceptance criteria items were resolved

Dataset quality page

A column for Failed docs is included in the table
A tooltip is placed in the title of the column
A % of documents inside Failure store is calculated for every dataStream
If % is lesser than 0.0001 but greater than 0 we should show ⚠ symbol next to the ~0 value (as we do with degraded docs)
Failed docs percentages greater than 0 should link to discover

Dataset details page

A metric, Failed docs, is included in the Overview panel under Data set quality. This metric includes the number of documents inside the failure store for the specific dataStream.
A tooltip is placed in the title of the Failed docs metric with message: The percentage of docs sent to failure store due to an issue during ingestion.
Degraded docs graph section is transformed to Document trends allowing the users to switch between Degraded docs and Failed docs trends over time.
A new chart for failed documents is created with links to discover/Logs explorer using the right dataView

🎥 Demo

Screen.Recording.2024-11-12.at.11.53.02.mov

Specific cases

Failed documents less than 0.001%

Quality column is shown after degraded docs and failed docs is loaded (Quality is now calculated using both values)

Screen.Recording.2024-11-18.at.15.55.05.mov

Screen.Recording.2024-11-18.at.16.01.03.mov

Users can still see Quality even when they don't have access to stats or something went wrong with that endpoint

Screen.Recording.2024-11-18.at.16.02.47.mov

The selected chart is kept in the URL

Screen.Recording.2024-11-19.at.16.57.54.mov

Missing

The following acceptance criteria are missing

Failed docs percentages greater than 0 should link to discover
A new chart for failed documents is created with links to discover/Logs explorer using the right dataView

because es changes are not finalised

elasticmachine · 2024-11-12T14:47:06Z

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

Click to trigger kibana-pull-request for this PR!
Click to trigger kibana-deploy-project-from-pr for this PR!

yngrdyn · 2024-11-12T14:47:59Z

/ci

yngrdyn · 2024-11-12T15:57:11Z

x-pack/plugins/observability_solution/dataset_quality/common/constants.ts

@@ -26,7 +26,7 @@ export const NONE = 'none';
 export const DEFAULT_TIME_RANGE = { from: 'now-24h', to: 'now' };
 export const DEFAULT_DATEPICKER_REFRESH = { value: 60000, pause: false };

-export const DEFAULT_DEGRADED_DOCS = {
+export const DEFAULT_QUALITY_DOC_STATS = {


Note
This is now reused by degradedDocs and failedDocs.

yngrdyn · 2024-11-12T15:58:08Z

...plugins/observability_solution/dataset_quality/common/data_streams_stats/data_stream_stat.ts

    };

    return new DataStreamStat(dataStreamStatProps);
  }

-  public static fromDegradedDocStat({
+  public static fromQualityStats({


Note
We can extend this method with more qualityStats and construct a Dataset from them

yngrdyn · 2024-11-12T15:59:25Z

x-pack/plugins/observability_solution/dataset_quality/common/utils/dataset_name.ts

@@ -40,7 +40,7 @@ export const indexNameToDataStreamParts = (dataStreamName: string) => {
 };

 export const extractIndexNameFromBackingIndex = (indexString: string): string => {
-  const pattern = /.ds-(.*?)-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-[0-9]{6}/;
+  const pattern = /.(?:ds|fs)-(.*?)-[0-9]{4}\.[0-9]{2}\.[0-9]{2}-[0-9]{6}/;


Important
Failure store is at the moment a backing index that starts with .fs

x-pack/plugins/observability_solution/dataset_quality/common/utils/quality_helpers.ts

yngrdyn · 2024-11-12T16:05:23Z

@mdbirnstiehl I need some help from your side

The tooltip for the quality column should now tell the users that quality is calculated by degradedDocs and failedDocs percentages. I came up with

How can we explain users what is a Failed doc? I came up with

In the summary we should also reference the new calculation of quality (DegradedDocs and failedDocs)

yngrdyn · 2024-11-12T16:08:34Z

x-pack/plugins/observability_solution/dataset_quality/public/hooks/use_summary_panel.ts

-  const datasetsQuality = {
-    percentages: filteredItems.map((item) => item.degradedDocs.percentage),
-  };
+  const datasetsQuality = countBy(filteredItems.map((item) => item.quality)) as Record<


Quality is now part of the dataset, so we don't need this calculation anymore. Instead I just created a map that will hold how many datasets fall into a quality status. e.g.

{ 'poor': 1, 'degraded': 2, 'good': 1, }

yngrdyn · 2024-11-12T18:56:50Z

/ci

mdbirnstiehl · 2024-11-12T19:26:46Z

@mdbirnstiehl I need some help from your side

The tooltip for the quality column should now tell the users that quality is calculated by degradedDocs and failedDocs percentages.

This one looks good.

How can we explain users what is a Failed doc? I came up with

I'm not sure if this is exactly how it works, but is there some way we could be more specific here? Like, "Percentage of docs sent to failure store due to an issue during ingestion."

I'm not sure "failure store" itself will be meaningful for users.

In the summary we should also reference the new calculation of quality (DegradedDocs and failedDocs)

This tool tip can probably match the tool tip for the data set quality column in the table.

A side note, maybe this would need to be a different issue, but this heading:

Should probably be Data Set Quality, not sets. It also looks like the table column headers are in title case, and should be in sentence case to be consistent with the other pages in Stack Management. For example, "Failed docs" instead of "Failed Docs"

yngrdyn · 2024-11-12T19:38:23Z

I'm not sure if this is exactly how it works, but is there some way we could be more specific here? Like, "Percentage of docs sent to failure store due to an issue during ingestion."

That's exactly how it works 👍🏼

Should probably be Data Set Quality, not sets. It also looks like the table column headers are in title case, and should be in sentence case to be consistent with the other pages in Stack Management. For example, "Failed docs" instead of "Failed Docs"

I'll address those in this PR as well. This is how everything looks now

Screen.Recording.2024-11-12.at.16.42.32.mov

yngrdyn · 2024-11-12T19:46:37Z

/ci

yngrdyn · 2024-11-13T10:40:23Z

/ci

mohamedhamed-ahmed · 2024-11-14T11:40:52Z

I started testing quickly and got 2 issues:

When I make the failed_docs request fail from the network tab, I still get the results somehow not sure whats happening there but need to take a further look.

Screen.Recording.2024-11-14.at.12.38.24.mov

When I open any dataset I get an error as the mapPercentageToQuality is expecting an array but gets a single value in this case

Screen.Recording.2024-11-14.at.12.39.03.mov

When I make the failed_docs request fail from the network tab, I still get the results somehow not sure whats happening there but need to take a further look.

This works fine you can ignore it, tried it and seems good. Only the other problem is existing

mohamedhamed-ahmed

Did first round of code review and got some small comments

x-pack/plugins/observability_solution/dataset_quality/common/utils/quality_helpers.ts

...s/observability_solution/dataset_quality/public/components/dataset_quality/table/columns.tsx

...tion/dataset_quality/public/components/dataset_quality/table/failed_docs_percentage_link.tsx

...plugins/observability_solution/dataset_quality/public/hooks/use_dataset_details_telemetry.ts

x-pack/plugins/observability_solution/dataset_quality/common/translations.ts

mohamedhamed-ahmed · 2024-11-19T12:34:43Z

...plugins/observability_solution/dataset_quality/server/routes/data_streams/get_failed_docs.ts

+export async function getFailedDocsPaginated(options: {
+  esClient: ElasticsearchClient;
+  types: DataStreamType[];
+  datasetQuery?: string;


nit: maybe renam datasetQuery to just datasetNames as its a bit confusing since its not really a query

what do you think about renaming it to indexPattern? or maybe just datasetPattern?

Sounds good to me 👍

…seEffect

mohamedhamed-ahmed · 2024-11-21T13:23:07Z

I think the Quality Issues table is meant to show only 5 rows per page

I doubt it since we have a selection component to choose how many rows per page we can view.

patpscal · 2024-11-21T15:48:27Z

I doubt it since we have a selection component to choose how many rows per page we can view.

That's right! I totally forgot, thank you 🤣

yngrdyn · 2024-11-21T18:05:09Z

/ci

yngrdyn · 2024-11-21T18:30:04Z

/ci

yngrdyn · 2024-11-22T16:20:08Z

/ci

yngrdyn · 2024-11-25T13:01:49Z

/ci

yngrdyn · 2024-11-25T18:59:55Z

/ci

yngrdyn · 2024-11-26T14:46:23Z

/ci

yngrdyn · 2024-11-26T16:04:26Z

/ci

yngrdyn · 2024-11-26T16:04:49Z

/ci

yngrdyn · 2024-11-26T18:36:49Z

/ci

yngrdyn · 2024-11-26T19:55:49Z

/ci

…rt-for-failure-store

yngrdyn · 2024-11-27T13:11:12Z

/ci

elasticmachine · 2024-11-27T13:39:20Z

💔 Build Failed

Buildkite Build
Commit: 895110f

Failed CI Steps

Check Types

History

cc @yngrdyn

yngrdyn and others added 2 commits November 12, 2024 11:42

Fetching failed docs and showing them in the table

81ad3fa

Merge branch 'elastic:main' into main

301a53a

yngrdyn self-assigned this Nov 12, 2024

yngrdyn commented Nov 12, 2024

View reviewed changes

x-pack/plugins/observability_solution/dataset_quality/common/utils/quality_helpers.ts Show resolved Hide resolved

yngrdyn commented Nov 12, 2024

View reviewed changes

fixing tests

2ad742f

yngrdyn added 2 commits November 12, 2024 16:44

Addressing copies across table and page

4fe638e

Merge branch 'main' into dataset-quality-support-for-failure-store

56a1793

Column names changed in tests

e11fabf

mohamedhamed-ahmed self-requested a review November 13, 2024 11:52

Merge branch 'main' into dataset-quality-support-for-failure-store

a9f173c

Adding failed info to dataset details

41d249a

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 41e4f61 to 41d249a Compare November 14, 2024 13:17

mohamedhamed-ahmed reviewed Nov 14, 2024

View reviewed changes

Addresing Pr cmments

91e2c3e

mohamedhamed-ahmed reviewed Nov 19, 2024

View reviewed changes

yngrdyn added 2 commits November 19, 2024 12:12

extracting percentage calculator to an util

f753581

merging useDocsChart hooks for quality issues + removing unnecesary u…

74df05c

…seEffect

yngrdyn added 2 commits November 21, 2024 15:04

Added e2e tests to fialure store support

5f1b389

Merge branch 'main' into dataset-quality-support-for-failure-store

2e47a6b

fixing i18n id duplicated

3715579

fixing some e2e tests

dd9816e

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 80f5c53 to dd9816e Compare November 22, 2024 16:19

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 7b32bf0 to 42db254 Compare November 25, 2024 18:59

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 42db254 to 0bb1ac6 Compare November 26, 2024 14:46

fixing e2e tests

076cb6c

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 0bb1ac6 to 076cb6c Compare November 26, 2024 16:04

Merge branch 'main' into dataset-quality-support-for-failure-store

0057ecb

fixing tests

f10abac

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from a33ca8f to f10abac Compare November 26, 2024 19:55

yngrdyn and others added 3 commits November 27, 2024 10:04

fixing some tests

f9ba989

Merge branch 'elastic:main' into main

c6846eb

Merge remote-tracking branch 'origin/main' into dataset-quality-suppo…

895110f

…rt-for-failure-store

yngrdyn force-pushed the dataset-quality-support-for-failure-store branch from 45e0213 to 895110f Compare November 27, 2024 13:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Dataset quality] Support failure store #199806

[Dataset quality] Support failure store #199806

yngrdyn commented Nov 12, 2024 •

edited

Loading

elasticmachine commented Nov 12, 2024

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

yngrdyn commented Nov 12, 2024

mdbirnstiehl commented Nov 12, 2024 •

edited

Loading

yngrdyn commented Nov 12, 2024 •

edited

Loading

yngrdyn commented Nov 12, 2024

yngrdyn commented Nov 13, 2024

mohamedhamed-ahmed commented Nov 14, 2024 •

edited

Loading

mohamedhamed-ahmed left a comment

mohamedhamed-ahmed Nov 19, 2024

yngrdyn Nov 19, 2024 •

edited

Loading

mohamedhamed-ahmed Nov 20, 2024

mohamedhamed-ahmed commented Nov 21, 2024

patpscal commented Nov 21, 2024

yngrdyn commented Nov 21, 2024

yngrdyn commented Nov 21, 2024

yngrdyn commented Nov 22, 2024

yngrdyn commented Nov 25, 2024

yngrdyn commented Nov 25, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

[Dataset quality] Support failure store #199806

Are you sure you want to change the base?

[Dataset quality] Support failure store #199806

Conversation

yngrdyn commented Nov 12, 2024 • edited Loading

Summary

Dataset quality page

Dataset details page

🎥 Demo

Specific cases

Missing

elasticmachine commented Nov 12, 2024

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn commented Nov 12, 2024

yngrdyn Nov 12, 2024

Choose a reason for hiding this comment

yngrdyn commented Nov 12, 2024

mdbirnstiehl commented Nov 12, 2024 • edited Loading

yngrdyn commented Nov 12, 2024 • edited Loading

yngrdyn commented Nov 12, 2024

yngrdyn commented Nov 13, 2024

mohamedhamed-ahmed commented Nov 14, 2024 • edited Loading

mohamedhamed-ahmed left a comment

Choose a reason for hiding this comment

mohamedhamed-ahmed Nov 19, 2024

Choose a reason for hiding this comment

yngrdyn Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

mohamedhamed-ahmed Nov 20, 2024

Choose a reason for hiding this comment

mohamedhamed-ahmed commented Nov 21, 2024

patpscal commented Nov 21, 2024

yngrdyn commented Nov 21, 2024

yngrdyn commented Nov 21, 2024

yngrdyn commented Nov 22, 2024

yngrdyn commented Nov 25, 2024

yngrdyn commented Nov 25, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 26, 2024

yngrdyn commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

💔 Build Failed

Failed CI Steps

History

yngrdyn commented Nov 12, 2024 •

edited

Loading

mdbirnstiehl commented Nov 12, 2024 •

edited

Loading

yngrdyn commented Nov 12, 2024 •

edited

Loading

mohamedhamed-ahmed commented Nov 14, 2024 •

edited

Loading

yngrdyn Nov 19, 2024 •

edited

Loading