[Mappings Editor] Add support for synthetic _source #199854

ElenaStoeva · 2024-11-12T18:20:48Z

Closes #198621

Summary

This PR adds support for the synthetic _source field in the mappings Advanced options.

Stored option:

Synthetic option selected:

Disabled option selected:

synthetic_source.mov

How to test:

Start Es with yarn es snapshot --license (we need Enterprise license to see the Synthetic source option) and Kibana with yarn start
Go to Index templates/Component templates and start creating a template
At the Mappings step, go to Advanced options.
Verify that selecting a _source field option translates to the correct Es request.
In Index templates form, verify that the default _source option depends on the index mode selected in the Logistics step. For LogsDB and Time series index mode, the default should be synthetic mode; otherwise, the stored option.
Verify that in Basic license, the synthetic option is not displayed.

Checklist

Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
Unit or functional tests were updated or added to match the most common scenarios
Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
This was checked for cross-browser compatibility

elasticmachine · 2024-11-14T19:29:23Z

Pinging @elastic/kibana-management (Team:Kibana Management)

…-fix'

florent-leborgne

Copy LGTM

florent-leborgne · 2024-11-18T09:35:31Z

.../mappings_editor/components/configuration_form/source_field_section/source_field_section.tsx

@@ -27,7 +77,7 @@ export const SourceFieldSection = () => {
      <p>
        <FormattedMessage
          id="xpack.idxMgmt.mappingsEditor.disabledSourceFieldCallOutDescription1"
-          defaultMessage="Disabling {source} lowers storage overhead within the index, but this comes at a cost. It also disables important features, such as the ability to reindex or debug queries by viewing the original document."
+          defaultMessage="Disabling {source} is not recommended. If storage overhead is a concer, consider using synthetic {source} instead. Disabling {source} will disable important features, such as the ability to reindex or debug queries by viewing the original document."


Suggested change

defaultMessage="Disabling {source} is not recommended. If storage overhead is a concer, consider using synthetic {source} instead. Disabling {source} will disable important features, such as the ability to reindex or debug queries by viewing the original document."

defaultMessage="Disabling {source} is not recommended. If storage overhead is a concern, consider using synthetic {source} instead. Disabling {source} will disable important features, such as the ability to reindex or debug queries by viewing the original document."

Thanks @florent-leborgne! Fixed the typo with aa2b19b.

ElenaStoeva · 2024-11-18T10:13:02Z

Converted back to a draft as I found one issue with setting synthetic source when index mode is logsdb/time series.

…com/ElenaStoeva/kibana into mappings-configuration/source-field

…-fix'

elasticmachine · 2024-11-19T14:06:52Z

💚 Build Succeeded

Buildkite Build
Commit: 12d3b27

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`indexManagement`	698	700	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`aiAssistantManagementSelection`	92.5KB	92.5KB	+78.0B
`indexManagement`	694.7KB	698.1KB	+3.4KB
`lists`	144.9KB	145.0KB	+78.0B
total			+3.5KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`core`	453.9KB	453.9KB	+78.0B

History

💔 Build #252276 failed 1907323
💔 Build #252215 failed 0c70fce
💔 Build #252214 failed 57624e1
💔 Build #252138 failed fbe99d4
💔 Build #252117 failed 772ede9
💔 Build #251561 failed b485c9a

cc @ElenaStoeva

sabarasaba

Nice work @ElenaStoeva! Tested locally and changes lgtm, left two small comments but for sake of speed might be worth to revise in a subsequent PR. What I do would like to see though, is perhaps a test for the license check to make sure the synthetic option isn't show with a basic license. wdyt?

sabarasaba · 2024-11-19T14:25:06Z

...k/plugins/index_management/public/application/components/mappings_editor/mappings_editor.tsx

@@ -120,6 +128,40 @@ export const MappingsEditor = React.memo(
      [dispatch]
    );

+    const [isLicenseCheckComplete, setIsLicenseCheckComplete] = useState(false);
+    useEffect(() => {
+      const subscription = licensing?.license$.subscribe((license: ILicense) => {


what do you think about extracting the license check subcription check into the plugin level and just pass something like canUseSyntheticOption variable into the renderApp method? that way we dont have tightly coupled logic and if the license condition changes its easier to update (I think we did something similar for checking if the user can use the new api key trust model in remote clusters)

That's a great idea! This might actually help with testing if the synthetic option is displayed or not based on the license. I think the issues that I face now might be due to the fact that in the initial render the check is not complete and so the synthetic option is not displayed even when it should.

sabarasaba · 2024-11-19T14:33:16Z

.../application/components/mappings_editor/components/configuration_form/configuration_form.tsx

@@ -37,12 +48,30 @@ const formSerializer = (formData: GenericObject, sourceFieldMode?: string) => {
    ? 'strict'
    : dynamicMapping?.enabled;

+  const _source =


nit: It might just be me, but this seems a little too complex to read at first glance. Perhaps a more explicit structure could make future extensions more straightforward? 🤔

I totally understand! I did my best to make it as clear as possible but it's a bit tricky given that the sourceField.option field actually controls two properties in Es (mode and enabled). We also don't want to add the mode property if the field is undefined (which means it hasn't been set by the user) - otherwise, just opening the "Advanced options" tab would insert a "mode" property which is a behavior that we recently managed to fix in this form. I'm open to any suggestions for shortening/simplifying the code.

ElenaStoeva · 2024-11-19T15:10:12Z

Many thanks for the review @sabarasaba! Yes, I also think we can address these comments in a separate PR so that we can merge this today.

What I do would like to see though, is perhaps a test for the license check to make sure the synthetic option isn't show with a basic license. wdyt?

Haha trust me, I'd also love to see this 😆 I spent quite a bit of time trying to implement these but it turned out trickier than I hoped. I initially added such tests with 32d6672 but then I had to change the implementation and they no longer worked as now the license check is performed in the parent component (the Mappings editor) instead of the Configuration form. I also tried adding these tests in the mappings_editor.test.tsx, but for some reason Jest cannot find any of the options at all (it seems they are not rendered, could be because we use EuiSuperSelect which needs to be mocked). I tried many different approaches but nothing worked so far.

I think if we refactor the license check code as you suggested these tests might be easier to implement.

Edit: Opened an issue for the improvements: #200769

sabarasaba

Approving so we can get it merged today, and we can patch the rest of the feedback in a subsequent PR 🚀

kibanamachine · 2024-11-19T16:50:10Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/11917791552

Closes elastic#198621 ## Summary This PR adds support for the synthetic _source field in the mappings Advanced options. Stored option: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 19" src="https://github.com/user-attachments/assets/086f7a3e-9ca1-42de-9f9c-d3599d839ccf"> Synthetic option selected: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 27" src="https://github.com/user-attachments/assets/3700bced-212a-4378-b51a-ab7a0f4f7b99"> Disabled option selected: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 36" src="https://github.com/user-attachments/assets/c7ddcbae-7c78-4477-824e-99b144a1f750"> https://github.com/user-attachments/assets/399d0f95-a5dd-4874-bb8c-e95d6ed38465 How to test: 1. Start Es with `yarn es snapshot --license` (we need Enterprise license to see the Synthetic source option) and Kibana with `yarn start` 2. Go to Index templates/Component templates and start creating a template 3. At the Mappings step, go to Advanced options. 4. Verify that selecting a _source field option translates to the correct Es request. 5. In Index templates form, verify that the default _source option depends on the index mode selected in the Logistics step. For LogsDB and Time series index mode, the default should be synthetic mode; otherwise, the stored option. 6. Verify that in Basic license, the synthetic option is not displayed. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [x] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [x] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [x] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) --------- Co-authored-by: kibanamachine <[email protected]> (cherry picked from commit 8be679a)

kibanamachine · 2024-11-19T16:55:01Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…200776) # Backport This will backport the following commits from `main` to `8.x`: - [[Mappings Editor] Add support for synthetic _source (#199854)](#199854)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Elena Stoeva <[email protected]>

Closes #200769 ## Summary This PR is a follow-up to #199854 and it adds the following code improvements: - Replaces Mappings-editor-context-level property `hasEnterpriceLicense` with plugin-context-level `canUseSyntheticSource` property - Adds jest tests to check if the synthetic option is correctly displayed based on license - Improves readability of serializer logic for the source field **How to test:** The same test instructions from #199854 can be followed with a focus on checking that the synthetic option is only available in Enterprise license.

) Closes elastic#200769 ## Summary This PR is a follow-up to elastic#199854 and it adds the following code improvements: - Replaces Mappings-editor-context-level property `hasEnterpriceLicense` with plugin-context-level `canUseSyntheticSource` property - Adds jest tests to check if the synthetic option is correctly displayed based on license - Improves readability of serializer logic for the source field **How to test:** The same test instructions from elastic#199854 can be followed with a focus on checking that the synthetic option is only available in Enterprise license. (cherry picked from commit 762bb7f)

…) (#201577) # Backport This will backport the following commits from `main` to `8.x`: - [[Mappings editor] Add code improvements for source field (#201188)](#201188)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Elena Stoeva <[email protected]>

) (#201576) # Backport This will backport the following commits from `main` to `8.17`: - [[Mappings editor] Add code improvements for source field (#201188)](#201188)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Elena Stoeva <[email protected]>

Closes elastic#198621 ## Summary This PR adds support for the synthetic _source field in the mappings Advanced options. Stored option: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 19" src="https://github.com/user-attachments/assets/086f7a3e-9ca1-42de-9f9c-d3599d839ccf"> Synthetic option selected: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 27" src="https://github.com/user-attachments/assets/3700bced-212a-4378-b51a-ab7a0f4f7b99"> Disabled option selected: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 36" src="https://github.com/user-attachments/assets/c7ddcbae-7c78-4477-824e-99b144a1f750"> https://github.com/user-attachments/assets/399d0f95-a5dd-4874-bb8c-e95d6ed38465 How to test: 1. Start Es with `yarn es snapshot --license` (we need Enterprise license to see the Synthetic source option) and Kibana with `yarn start` 2. Go to Index templates/Component templates and start creating a template 3. At the Mappings step, go to Advanced options. 4. Verify that selecting a _source field option translates to the correct Es request. 5. In Index templates form, verify that the default _source option depends on the index mode selected in the Logistics step. For LogsDB and Time series index mode, the default should be synthetic mode; otherwise, the stored option. 6. Verify that in Basic license, the synthetic option is not displayed. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [x] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [x] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [x] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) --------- Co-authored-by: kibanamachine <[email protected]>

) Closes elastic#200769 ## Summary This PR is a follow-up to elastic#199854 and it adds the following code improvements: - Replaces Mappings-editor-context-level property `hasEnterpriceLicense` with plugin-context-level `canUseSyntheticSource` property - Adds jest tests to check if the synthetic option is correctly displayed based on license - Improves readability of serializer logic for the source field **How to test:** The same test instructions from elastic#199854 can be followed with a focus on checking that the synthetic option is only available in Enterprise license.

Closes elastic#198621 ## Summary This PR adds support for the synthetic _source field in the mappings Advanced options. Stored option: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 19" src="https://github.com/user-attachments/assets/086f7a3e-9ca1-42de-9f9c-d3599d839ccf"> Synthetic option selected: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 27" src="https://github.com/user-attachments/assets/3700bced-212a-4378-b51a-ab7a0f4f7b99"> Disabled option selected: <img width="1184" alt="Screenshot 2024-11-14 at 19 19 36" src="https://github.com/user-attachments/assets/c7ddcbae-7c78-4477-824e-99b144a1f750"> https://github.com/user-attachments/assets/399d0f95-a5dd-4874-bb8c-e95d6ed38465 How to test: 1. Start Es with `yarn es snapshot --license` (we need Enterprise license to see the Synthetic source option) and Kibana with `yarn start` 2. Go to Index templates/Component templates and start creating a template 3. At the Mappings step, go to Advanced options. 4. Verify that selecting a _source field option translates to the correct Es request. 5. In Index templates form, verify that the default _source option depends on the index mode selected in the Logistics step. For LogsDB and Time series index mode, the default should be synthetic mode; otherwise, the stored option. 6. Verify that in Basic license, the synthetic option is not displayed. ### Checklist - [x] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md) - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] Any UI touched in this PR is usable by keyboard only (learn more about [keyboard accessibility](https://webaim.org/techniques/keyboard/)) - [x] Any UI touched in this PR does not create any new axe failures (run axe in browser: [FF](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/), [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US)) - [x] This renders correctly on smaller devices using a responsive layout. (You can test this [in your browser](https://www.browserstack.com/guide/responsive-testing-on-local-server)) - [x] This was checked for [cross-browser compatibility](https://www.elastic.co/support/matrix#matrix_browsers) --------- Co-authored-by: kibanamachine <[email protected]>

) Closes elastic#200769 ## Summary This PR is a follow-up to elastic#199854 and it adds the following code improvements: - Replaces Mappings-editor-context-level property `hasEnterpriceLicense` with plugin-context-level `canUseSyntheticSource` property - Adds jest tests to check if the synthetic option is correctly displayed based on license - Improves readability of serializer logic for the source field **How to test:** The same test instructions from elastic#199854 can be followed with a focus on checking that the synthetic option is only available in Enterprise license.

ElenaStoeva self-assigned this Nov 12, 2024

[Mappings Editor] Add support for synthetic _source

96ec5ac

ElenaStoeva force-pushed the mappings-configuration/source-field branch from b234d7a to 96ec5ac Compare November 12, 2024 18:40

ElenaStoeva added 4 commits November 14, 2024 15:07

Fix deserialization

de0be81

Fix error in component template form mappings editor

853f9ff

Fix existing jest tests

92adc29

Add jest tests

32d6672

ElenaStoeva force-pushed the mappings-configuration/source-field branch from 1b18c86 to 32d6672 Compare November 14, 2024 19:15

ElenaStoeva and others added 2 commits November 14, 2024 19:16

Merge branch 'main' into mappings-configuration/source-field

9fedad6

Fix copy

16566e4

ElenaStoeva marked this pull request as ready for review November 14, 2024 19:29

ElenaStoeva requested review from a team as code owners November 14, 2024 19:29

kibanamachine added 2 commits November 14, 2024 19:48

[CI] Auto-commit changed files from 'make api-docs'

991cb4f

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

b485c9a

…-fix'

florent-leborgne approved these changes Nov 18, 2024

View reviewed changes

ElenaStoeva marked this pull request as draft November 18, 2024 10:12

ElenaStoeva added 3 commits November 18, 2024 20:03

Fix _source field value change when index mode is logsdb

a131400

Merge branch 'mappings-configuration/source-field' of https://github.…

fc5c998

…com/ElenaStoeva/kibana into mappings-configuration/source-field

Fix typo in copy

aa2b19b

ElenaStoeva marked this pull request as ready for review November 18, 2024 18:06

Fix i18n errors

772ede9

kibanamachine and others added 5 commits November 19, 2024 04:05

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

0c70fce

…-fix'

Fix type errors

1c8422a

Fix merge conflicts

117ec85

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

1907323

…-fix'

Fix jest tests

4b89cac

sabarasaba self-requested a review November 19, 2024 11:51

Add serialization tests

12d3b27

ElenaStoeva force-pushed the mappings-configuration/source-field branch from cdc8827 to 12d3b27 Compare November 19, 2024 12:18

sabarasaba reviewed Nov 19, 2024

View reviewed changes

ElenaStoeva mentioned this pull request Nov 19, 2024

[Mappings Editor] Follow-up improvement for _source field #200769

Closed

ElenaStoeva enabled auto-merge (squash) November 19, 2024 16:40

sabarasaba approved these changes Nov 19, 2024

View reviewed changes

ElenaStoeva merged commit 8be679a into elastic:main Nov 19, 2024
23 checks passed

kibanamachine mentioned this pull request Nov 19, 2024

[8.x] [Mappings Editor] Add support for synthetic _source (#199854) #200776

Merged

ElenaStoeva mentioned this pull request Nov 21, 2024

[Mappings editor] Add code improvements for source field #201188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Mappings Editor] Add support for synthetic _source #199854

[Mappings Editor] Add support for synthetic _source #199854

ElenaStoeva commented Nov 12, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Nov 14, 2024

florent-leborgne left a comment

florent-leborgne Nov 18, 2024

ElenaStoeva Nov 18, 2024 •

edited

Loading

ElenaStoeva commented Nov 18, 2024

elasticmachine commented Nov 19, 2024

sabarasaba left a comment

sabarasaba Nov 19, 2024

ElenaStoeva Nov 19, 2024

sabarasaba Nov 19, 2024

ElenaStoeva Nov 19, 2024

ElenaStoeva commented Nov 19, 2024 •

edited

Loading

sabarasaba left a comment

kibanamachine commented Nov 19, 2024

kibanamachine commented Nov 19, 2024

	defaultMessage="Disabling {source} is not recommended. If storage overhead is a concer, consider using synthetic {source} instead. Disabling {source} will disable important features, such as the ability to reindex or debug queries by viewing the original document."
	defaultMessage="Disabling {source} is not recommended. If storage overhead is a concern, consider using synthetic {source} instead. Disabling {source} will disable important features, such as the ability to reindex or debug queries by viewing the original document."

[Mappings Editor] Add support for synthetic _source #199854

[Mappings Editor] Add support for synthetic _source #199854

Conversation

ElenaStoeva commented Nov 12, 2024 • edited by kibanamachine Loading

Summary

Checklist

elasticmachine commented Nov 14, 2024

florent-leborgne left a comment

Choose a reason for hiding this comment

florent-leborgne Nov 18, 2024

Choose a reason for hiding this comment

ElenaStoeva Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

ElenaStoeva commented Nov 18, 2024

elasticmachine commented Nov 19, 2024

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

sabarasaba left a comment

Choose a reason for hiding this comment

sabarasaba Nov 19, 2024

Choose a reason for hiding this comment

ElenaStoeva Nov 19, 2024

Choose a reason for hiding this comment

sabarasaba Nov 19, 2024

Choose a reason for hiding this comment

ElenaStoeva Nov 19, 2024

Choose a reason for hiding this comment

ElenaStoeva commented Nov 19, 2024 • edited Loading

sabarasaba left a comment

Choose a reason for hiding this comment

kibanamachine commented Nov 19, 2024

kibanamachine commented Nov 19, 2024

💚 All backports created successfully

Questions ?

ElenaStoeva commented Nov 12, 2024 •

edited by kibanamachine

Loading

ElenaStoeva Nov 18, 2024 •

edited

Loading

ElenaStoeva commented Nov 19, 2024 •

edited

Loading