Allow data Streams enrich policies #200903

SoniaSanzV · 2024-11-20T12:23:21Z

Summary

This PR introduces the data streams as source indices in the enrich policies. For that, it uses the groups functionality, so the indices are separated from the data streams.

datastreams_enrich_policy.mov

Strings

I've changed some of the existing strings but left other as they were. I'll be happy to also change those if folks feel that should be the case.

I've added a hint text and changed the label for the box in the creation step (marked in yellow)

I haven't change the label for the indices group (green) and I've followed the same structure for the new group category (yellow)

I haven't change the 'Source indices' text in the Summary (green) because in the request tab the correspondan field in the request is indices (blue)

I also haven't change the 'Source indices' text in the table or in the detail flyout (green) since both indices and data streams are treated like indices.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios
If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
Flaky Test Runner was used on any tests changed
The PR description includes the appropriate Release Notes section, and the correct release_node:* label is applied per the guidelines

elasticmachine · 2024-11-20T12:24:40Z

Pinging @elastic/kibana-management (Team:Kibana Management)

florent-leborgne

Copy LGTM 🌆 Just some super small suggestions

florent-leborgne · 2024-11-20T16:13:37Z

...anagement/public/application/sections/enrich_policy_create/steps/fields/indices_selector.tsx

+    noMatches: i18n.translate(
+      'xpack.idxMgmt.enrichPolicyCreate.indicesSelector.dataStream.noMatchingOption',
+      {
+        defaultMessage: 'No data stream match your search criteria.',


Suggested change

defaultMessage: 'No data stream match your search criteria.',

defaultMessage: 'No data streams match your search criteria.',

Making it parallel with plural "indices"

florent-leborgne · 2024-11-20T16:13:56Z

...anagement/public/application/sections/enrich_policy_create/steps/fields/indices_selector.tsx

+  sourcePlaceholder: i18n.translate(
+    'xpack.idxMgmt.enrichPolicyCreate.indicesSelector.placeholder',
+    {
+      defaultMessage: 'Select source indices and/or data streams.',


Suggested change

defaultMessage: 'Select source indices and/or data streams.',

defaultMessage: 'Select source indices and data streams.',

or "or", but I'd prefer to not use "and/or"

mattkime · 2024-11-21T05:41:46Z

A couple of thoughts -

Do we want to separate indices and data streams in the list for selection? I think it might make more sense to show them alphabetized together. The resolve api can get both indices and data streams. It also gets aliases. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-resolve-index-api.html

The Based on your indices text strikes me as odd - its a list label inside the list. It also seems to imply that the field label could do a bit more work.

We should probably be hiding system and hidden indices in this list.

@jovana-andjelkovic is probably the best person to weigh in

SoniaSanzV · 2024-11-21T06:28:50Z

A couple of thoughts -

Do we want to separate indices and data streams in the list for selection? I think it might make more sense to show them alphabetized together. The resolve api can get both indices and data streams. It also gets aliases. https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-resolve-index-api.html

The Based on your indices text strikes me as odd - its a list label inside the list. It also seems to imply that the field label could do a bit more work.

We should probably be hiding system and hidden indices in this list.

@jovana-andjelkovic is probably the best person to weigh in

Thank you for your comment @mattkime! The groups functionality was the proposal that @alisonelizabeth and @jovana-andjelkovic agreed here, but I lack the deeper knowledge to judge whether it makes sense to show them together or separately. I'll do some research to have an opinion.

mattkime · 2024-11-21T06:33:32Z

@SoniaSanzV It looks like there's already been conversation on how the indices and data streams are ordered so I consider the issue resolved and your implementation to be correct.

We should still consider hiding system and hidden indices.

sabarasaba · 2024-11-26T14:40:31Z

@elasticmachine merge upstream

sabarasaba

Nice work @SoniaSanzV! Changes lgtm, tested locally and works great 🚀 Left one nit and I also think we should add an api integration test for the new server route, let me know what you think!

sabarasaba · 2024-11-26T14:54:41Z

...ns/index_management/public/application/sections/enrich_policy_create/steps/configuration.tsx

@@ -95,15 +95,15 @@ export const configurationFormSchema: FormSchema = {

  sourceIndices: {
    label: i18n.translate('xpack.idxMgmt.enrichPolicyCreate.configurationStep.sourceIndicesLabel', {
-      defaultMessage: 'Source indices',


nit: maybe we should rename the key of the i18n.translate to be sourceLabel instead of sourceIndicesLabel (same for the error)

sabarasaba · 2024-11-26T14:59:33Z

x-pack/plugins/index_management/server/routes/api/enrich_policies/register_create_route.ts

@@ -105,6 +111,33 @@ export function registerCreateRoute({ router, lib: { handleEsError } }: RouteDep
      }
    }
  );
+  router.post(


Can we add a unit test for this route here and an api integration test similar to this one ?

sabarasaba

latest changes lgtm, nice work!

jovana-andjelkovic · 2024-11-27T15:27:39Z

@SoniaSanzV I wonder if it's confusing to have "Source" as input field, and its values everywhere else referred to as "Source indices"?

Not sure if .alerts-observability.uptime.alerts-default example is a data stream or an index? Would the Request tab show data streams under indices: or would they be their own property?

…licies_#187438

elasticmachine · 2024-11-27T18:50:52Z

💚 Build Succeeded

Buildkite Build
Commit: e4ea39d

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`indexManagement`	698.2KB	699.2KB	+996.0B

History

💔 Build #255024 failed 21d71a2
💔 Build #254945 failed f167983
💔 Build #254829 failed f167983
💔 Build #252999 failed fdea243
💛 Build #252714 was flaky 0f4325b

cc @SoniaSanzV

kibanamachine · 2024-11-27T18:50:53Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12056556793

Closes [elastic#187438](elastic#187438) ## Summary This PR introduces the data streams as source indices in the enrich policies. For that, it uses the groups functionality, so the indices are separated from the data streams. https://github.com/user-attachments/assets/1716d57b-b8c7-417a-b75c-33378f3269ca ### Strings I've changed some of the existing strings but left other as they were. I'll be happy to also change those if folks feel that should be the case. I've added a hint text and changed the label for the box in the creation step (marked in yellow) <img width="400" alt="Screenshot 2024-11-20 at 12 35 32" src="https://github.com/user-attachments/assets/794a7063-3f75-4f3d-996c-12ca8b9ed9b6"> I haven't change the label for the indices group (green) and I've followed the same structure for the new group category (yellow) <table> <tr> <td style="padding-right: 10px;"> <img alt="Screenshot 2024-11-20 at 12 37 51" src="https://github.com/user-attachments/assets/89a554a6-57e4-4208-bde4-d4868dec11bc"> </td> <td> <img alt="Screenshot 2024-11-20 at 12 38 16" src="https://github.com/user-attachments/assets/d82c7ae3-79c9-432e-aa3c-8ca02502e18f"> </td> </tr> </table> I haven't change the 'Source indices' text in the Summary (green) because in the request tab the correspondan field in the request is indices (blue) <table> <tr> <td style="padding-right: 10px;"> <img alt="Screenshot 2024-11-20 at 12 38 53" src="https://github.com/user-attachments/assets/80623ded-11e3-4661-9fa8-ad84182a72a3"> </td> <td> <img alt="Screenshot 2024-11-20 at 12 39 19" src="https://github.com/user-attachments/assets/f2ad6449-1a3a-448a-8598-c8c9eb41a717"> </td> </tr> </table> I also haven't change the 'Source indices' text in the table or in the detail flyout (green) since both indices and data streams are treated like indices. <table> <tr> <td style="padding-right: 10px;"> <img width="898" alt="Screenshot 2024-11-20 at 12 39 52" src="https://github.com/user-attachments/assets/8165fe3f-0286-4ad1-a40d-44e19e4eef29"> </td> <td> <img width="892" alt="Screenshot 2024-11-20 at 12 40 25" src="https://github.com/user-attachments/assets/bb046aa4-2a6c-40e0-bb32-ecb4a13adea9"> </td> </tr> </table> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) (cherry picked from commit 8629b1d)

kibanamachine · 2024-11-27T18:55:40Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

# Backport This will backport the following commits from `main` to `8.x`: - [Allow data Streams enrich policies (#200903)](#200903)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Sonia Sanz Vivas <[email protected]>

SoniaSanzV · 2024-11-28T07:58:00Z

Sorry @jovana-andjelkovic , I had this PR set to auto-merge and I miss your comment before it was merged. I've taken a few screenshots with much more specific names that I can answer what you are asking. I don't think having "Source" in the first step is bad, since is the only place we separate the concept of index and datastream. From that point, everything is treated as an index. But we can come back to te previous copy if you think is better. Or even do what Matt suggested here.

jovana-andjelkovic · 2024-11-28T12:37:52Z

@SoniaSanzV thank you for these screenshots, super helpful! Yeah I agree with @mattkime that "based on your.." feels odd, I would rather go with just:

Indices
Data streams

I'm a bit confused why we show in response both indices and data streams under indices:, but I would assume data streams are considered in this case a "catch all" (.*) for backing indices that belong to it and hence it belongs in there 🤔

I wouldn't change back the name of the source field, it feels more correct to me. Anyways, I don't think it's a huge deal if we leave it as is, but if we decide to open separate issue to look at the copy, let's consult a tech writer.

Closes [elastic#187438](elastic#187438) ## Summary This PR introduces the data streams as source indices in the enrich policies. For that, it uses the groups functionality, so the indices are separated from the data streams. https://github.com/user-attachments/assets/1716d57b-b8c7-417a-b75c-33378f3269ca ### Strings I've changed some of the existing strings but left other as they were. I'll be happy to also change those if folks feel that should be the case. I've added a hint text and changed the label for the box in the creation step (marked in yellow) <img width="400" alt="Screenshot 2024-11-20 at 12 35 32" src="https://github.com/user-attachments/assets/794a7063-3f75-4f3d-996c-12ca8b9ed9b6"> I haven't change the label for the indices group (green) and I've followed the same structure for the new group category (yellow) <table> <tr> <td style="padding-right: 10px;"> <img alt="Screenshot 2024-11-20 at 12 37 51" src="https://github.com/user-attachments/assets/89a554a6-57e4-4208-bde4-d4868dec11bc"> </td> <td> <img alt="Screenshot 2024-11-20 at 12 38 16" src="https://github.com/user-attachments/assets/d82c7ae3-79c9-432e-aa3c-8ca02502e18f"> </td> </tr> </table> I haven't change the 'Source indices' text in the Summary (green) because in the request tab the correspondan field in the request is indices (blue) <table> <tr> <td style="padding-right: 10px;"> <img alt="Screenshot 2024-11-20 at 12 38 53" src="https://github.com/user-attachments/assets/80623ded-11e3-4661-9fa8-ad84182a72a3"> </td> <td> <img alt="Screenshot 2024-11-20 at 12 39 19" src="https://github.com/user-attachments/assets/f2ad6449-1a3a-448a-8598-c8c9eb41a717"> </td> </tr> </table> I also haven't change the 'Source indices' text in the table or in the detail flyout (green) since both indices and data streams are treated like indices. <table> <tr> <td style="padding-right: 10px;"> <img width="898" alt="Screenshot 2024-11-20 at 12 39 52" src="https://github.com/user-attachments/assets/8165fe3f-0286-4ad1-a40d-44e19e4eef29"> </td> <td> <img width="892" alt="Screenshot 2024-11-20 at 12 40 25" src="https://github.com/user-attachments/assets/bb046aa4-2a6c-40e0-bb32-ecb4a13adea9"> </td> </tr> </table> ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [ ] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [ ] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the [docker list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker) - [ ] This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The `release_note:breaking` label should be applied in these situations. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_node:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

SoniaSanzV requested a review from a team as a code owner November 20, 2024 12:23

SoniaSanzV self-assigned this Nov 20, 2024

SoniaSanzV changed the title ~~Index management/allow datastreams enrich policies #187438~~ Allow data Streams enrich policies Nov 20, 2024

SoniaSanzV requested a review from florent-leborgne November 20, 2024 14:27

florent-leborgne approved these changes Nov 20, 2024

View reviewed changes

SoniaSanzV force-pushed the index_management/allow_datastreams_enrich_policies_#187438 branch from 0f4325b to fdea243 Compare November 21, 2024 06:53

sabarasaba reviewed Nov 26, 2024

View reviewed changes

SoniaSanzV force-pushed the index_management/allow_datastreams_enrich_policies_#187438 branch from 139df16 to f167983 Compare November 27, 2024 10:19

sabarasaba approved these changes Nov 27, 2024

View reviewed changes

SoniaSanzV enabled auto-merge (squash) November 27, 2024 14:25

SoniaSanzV added 5 commits November 27, 2024 16:14

Allow selecting data streams in enrich policies

9529eee

Add hint text and change strings

22568d7

Fix strings

6eacadc

Add API test

f15261e

Update translation file

21d71a2

SoniaSanzV force-pushed the index_management/allow_datastreams_enrich_policies_#187438 branch from f167983 to 21d71a2 Compare November 27, 2024 15:14

Merge branch 'main' into index_management/allow_datastreams_enrich_po…

e4ea39d

…licies_#187438

SoniaSanzV merged commit 8629b1d into elastic:main Nov 27, 2024
8 checks passed

kibanamachine added the v9.0.0 label Nov 27, 2024

kibanamachine mentioned this pull request Nov 27, 2024

[8.x] Allow data Streams enrich policies (#200903) #202055

Merged

kibanamachine added the v8.18.0 label Nov 27, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow data Streams enrich policies #200903

Allow data Streams enrich policies #200903

SoniaSanzV commented Nov 20, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Nov 20, 2024

florent-leborgne left a comment

florent-leborgne Nov 20, 2024

florent-leborgne Nov 20, 2024

mattkime commented Nov 21, 2024 •

edited

Loading

SoniaSanzV commented Nov 21, 2024

mattkime commented Nov 21, 2024

sabarasaba commented Nov 26, 2024

sabarasaba left a comment

sabarasaba Nov 26, 2024

sabarasaba Nov 26, 2024

sabarasaba left a comment

jovana-andjelkovic commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

kibanamachine commented Nov 27, 2024

kibanamachine commented Nov 27, 2024

SoniaSanzV commented Nov 28, 2024

jovana-andjelkovic commented Nov 28, 2024

	defaultMessage: 'No data stream match your search criteria.',
	defaultMessage: 'No data streams match your search criteria.',

	defaultMessage: 'Select source indices and/or data streams.',
	defaultMessage: 'Select source indices and data streams.',

Allow data Streams enrich policies #200903

Allow data Streams enrich policies #200903

Conversation

SoniaSanzV commented Nov 20, 2024 • edited by kibanamachine Loading

Summary

Strings

Checklist

elasticmachine commented Nov 20, 2024

florent-leborgne left a comment

Choose a reason for hiding this comment

florent-leborgne Nov 20, 2024

Choose a reason for hiding this comment

florent-leborgne Nov 20, 2024

Choose a reason for hiding this comment

mattkime commented Nov 21, 2024 • edited Loading

SoniaSanzV commented Nov 21, 2024

mattkime commented Nov 21, 2024

sabarasaba commented Nov 26, 2024

sabarasaba left a comment

Choose a reason for hiding this comment

sabarasaba Nov 26, 2024

Choose a reason for hiding this comment

sabarasaba Nov 26, 2024

Choose a reason for hiding this comment

sabarasaba left a comment

Choose a reason for hiding this comment

jovana-andjelkovic commented Nov 27, 2024

elasticmachine commented Nov 27, 2024

💚 Build Succeeded

Metrics [docs]

Async chunks

History

kibanamachine commented Nov 27, 2024

kibanamachine commented Nov 27, 2024

💚 All backports created successfully

Questions ?

SoniaSanzV commented Nov 28, 2024

jovana-andjelkovic commented Nov 28, 2024

SoniaSanzV commented Nov 20, 2024 •

edited by kibanamachine

Loading

mattkime commented Nov 21, 2024 •

edited

Loading