[UnifiedFieldList] Add new fields ingested in background with valid mappings #172329

kertal · 2023-11-30T21:39:51Z

Summary

This PR improves UnifiedFieldList adding a newly ingested field to be added to the list with already the right type in Discover. On top of that, it triggers a refresh of the selected DataView fields, so the new field will be available by consumers of the DataView, no reload is necessary, and also when we introduce caching, the cache will be refreshed. One more benefit, when switching data views, the field refreshing was removed. It's no longer necessary, making data view switching snappy, while the UnifiedFieldList makes sure, new fields are showing up, and the data view field cache is refreshed behind the scenes, if necessary.

How to test

In Console ingest a new record

PUT field-refresh-test/_doc/1
{
  "message": "Hi"
}

Go to Discover and create a DataView for field-refresh-test

Open Kibana / Console in a second tab/window and ingest

PUT field-refresh-test/_doc/1
{
  "message": "Hi again!",
  "🐣": "🐔"
}

Refresh the search in the first window. Then you should be able to see the new field with the correct mapping in the field list.

Before this change the new field was available, but unmapped. It also works in Lens, when new fields caps are requested due to change of query / filters / time range

Resolves #172342
Resolves #169360
Resolves #170874

Limitation: A type change won't be detected (this could be a follow up task)

Checklist

Unit or functional tests were updated or added to match the most common scenarios

packages/kbn-unified-field-list/src/hooks/use_grouped_fields.ts

kertal · 2023-12-04T20:22:40Z

@elasticmachine merge upstream

kertal · 2023-12-07T09:30:35Z

@elasticmachine merge upstream

…s' into unified-field-list-add-new-fields

…add-new-fields

kertal · 2023-12-13T18:16:06Z

/ci

kertal · 2023-12-13T22:31:32Z

@elasticmachine merge upstream

kertal · 2023-12-13T22:31:52Z

/ci

kertal · 2023-12-14T04:52:13Z

...ages/kbn-unified-field-list/src/containers/unified_field_list_sidebar/field_list_sidebar.tsx

@@ -245,6 +230,27 @@ export const UnifiedFieldListSidebarComponent: React.FC<UnifiedFieldListSidebarP
    onOverrideFieldGroupDetails: stateService.creationOptions.onOverrideFieldGroupDetails,
  });

+  useEffect(() => {


note, this code was just moved to have access to the new modified list of allFields

…/kibana into unified-field-list-add-new-fields

kertal · 2023-12-15T08:47:07Z

/ci

drewdaemon

Seems to work well in Lens.

Related issues #149093, #143877

jughosta · 2024-01-03T13:37:45Z

Reviewed and tested the PR again today. I found some issues and addressed them via kertal#16

@davismcphee Would be great if you could review it too.

…s-updated [Discover] Small fixes for types and memo deps. Add tests.

kertal · 2024-01-08T00:40:28Z

@elasticmachine merge upstream

kertal · 2024-01-08T05:43:01Z

@elasticmachine merge upstream

davismcphee

Code changes look good, and it worked great in testing in both Discover and Lens! I even tried to fool it a bit by deleting and recreating the index, adding different fields, etc., but the behaviour seemed good in each case (old fields after deleting the index got moved to empty fields, etc.).

It's a valuable enhancement with or without caching, but especially useful considering the upcoming changes. I left a couple of minor code suggestions, but nothing I'd consider blocking. Thanks and LGTM!

packages/kbn-unified-field-list/src/hooks/use_existing_fields.ts

x-pack/plugins/lens/public/data_views_service/loader.ts

x-pack/plugins/lens/public/datasources/form_based/datapanel.tsx

…add-new-fields

kibana-ci · 2024-01-09T11:52:47Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: d024027

Failed CI Steps

Jest Tests #7

Test Failures

[job] [logs] Jest Tests #7 / RegisteredAttachmentsPropertyActions does show the property actions with only delete permissions
[job] [logs] Jest Tests #7 / RegisteredAttachmentsPropertyActions remove attachments correctly
[job] [logs] Jest Tests #7 / RegisteredAttachmentsPropertyActions renders correctly registered attachments
[job] [logs] Jest Tests #7 / RegisteredAttachmentsPropertyActions renders the correct number of actions
[job] [logs] Jest Tests #7 / RegisteredAttachmentsPropertyActions renders the modal info correctly

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`discover`	688	689	+1
`eventAnnotationListing`	461	462	+1
`lens`	1191	1193	+2
`logExplorer`	637	638	+1
total			+5

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/unified-field-list`	261	267	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discover`	555.3KB	556.3KB	+1.1KB
`eventAnnotationListing`	196.6KB	196.9KB	+320.0B
`lens`	1.4MB	1.4MB	+1.4KB
`unifiedSearch`	213.8KB	213.7KB	-35.0B
total			+2.7KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`discover`	31.4KB	31.5KB	+139.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/unified-field-list`	285	291	+6

History

💛 Build #186277 was flaky 223f546
💔 Build #186275 failed 8617b3d
💔 Build #185588 failed d62dba7
💚 Build #185358 succeeded 9170300
💚 Build #184951 succeeded 165c3cd
💚 Build #184706 succeeded f747c2e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kertal

davismcphee

Another quick review after the latest round of changes, and everything is still working as intended. LGTM, thanks!

mattkime · 2024-01-09T21:38:34Z

Code looks good, everything seems to be working well. Nice work!

…appings (elastic#172329) Improves UnifiedFieldList by adding a newly ingested field to the list with the right type. On top of that, it triggers a refresh of the selected DataView fields, so the new field be available by consumers of the DataView. Co-authored-by: Julia Rechkunova <[email protected]>

Add new fields to list with correct types

26d3d6d

kertal self-assigned this Nov 30, 2023

kertal added Feature:Discover Discover Application Team:DataDiscovery Discover, search (e.g. data plugin and KQL), data views, saved searches. For ES|QL, use Team:ES|QL. enhancement New value added to drive a business result labels Dec 1, 2023

Improve code

7b999db

kertal commented Dec 1, 2023

View reviewed changes

packages/kbn-unified-field-list/src/hooks/use_grouped_fields.ts Outdated Show resolved Hide resolved

kertal added 2 commits December 1, 2023 09:46

Remove redundant newFieldsByFieldNameMap

3a6c239

Add multi field support

307a16d

Merge branch 'main' into unified-field-list-add-new-fields

54daa2f

kibanamachine and others added 5 commits December 7, 2023 04:30

Merge branch 'main' into unified-field-list-add-new-fields

ca50f4d

Add support for Lens

999a6a9

Merge remote-tracking branch 'origin/unified-field-list-add-new-field…

1a30c70

…s' into unified-field-list-add-new-fields

Merge remote-tracking branch 'upstream/main' into unified-field-list-…

fa237a4

…add-new-fields

Add functional test

15cc582

Merge branch 'main' into unified-field-list-add-new-fields

3cea884

kertal changed the title ~~[UnifiedFieldList] Add new fields to list with correct types~~ [UnifiedFieldList] Add new fields ingested in background with valid mappings Dec 14, 2023

kertal commented Dec 14, 2023

View reviewed changes

kertal added 3 commits December 14, 2023 06:06

Add documentation

065a50c

Improve code

e9fd00e

Merge branch 'unified-field-list-add-new-fields' of github.com:kertal…

63907e4

…/kibana into unified-field-list-add-new-fields

kertal requested a review from mattkime December 14, 2023 21:55

Add Lens test

454592c

Improve Lens code preventing an endless loop of update

75cc235

kertal and others added 4 commits December 22, 2023 15:57

Add useMemo for new fields

c76c87e

Address review feedback

165c3cd

Merge branch 'main' into unified-field-list-add-new-fields

cbee199

[CI] Auto-commit changed files from 'node scripts/notice'

9170300

drewdaemon approved these changes Jan 2, 2024

View reviewed changes

jughosta added 2 commits January 3, 2024 11:46

Merge branch 'main' into unified-field-list-add-new-fields

d62dba7

[Discover] Small fixes for types and memo deps. Add tests.

02a4537

Merge pull request #16 from jughosta/unified-field-list-add-new-field…

621a1df

…s-updated [Discover] Small fixes for types and memo deps. Add tests.

Merge branch 'main' into unified-field-list-add-new-fields

8617b3d

kertal requested a review from davismcphee January 8, 2024 00:40

Merge branch 'main' into unified-field-list-add-new-fields

223f546

davismcphee approved these changes Jan 8, 2024

View reviewed changes

packages/kbn-unified-field-list/src/hooks/use_existing_fields.ts Outdated Show resolved Hide resolved

x-pack/plugins/lens/public/data_views_service/loader.ts Show resolved Hide resolved

x-pack/plugins/lens/public/datasources/form_based/datapanel.tsx Outdated Show resolved Hide resolved

kertal added 3 commits January 9, 2024 09:50

Merge remote-tracking branch 'upstream/main' into unified-field-list-…

8549c3f

…add-new-fields

Address review feedback

f751a4e

Apply useLatest feedback

d024027

davismcphee approved these changes Jan 9, 2024

View reviewed changes

kertal merged commit 1d8eb89 into elastic:main Jan 9, 2024
21 checks passed

kibanamachine added v8.13.0 backport:skip This commit does not require backporting labels Jan 9, 2024

This was referenced Feb 1, 2024

[Discover] Add UI element to refresh data view / field list #170874

Closed

[Logs Explorer] Ensure compatibility with data view caching #171389

Open

davismcphee mentioned this pull request Dec 4, 2024

Discover: Selected field shows with wrong icon when switching from unmapped to mapped #201751

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[UnifiedFieldList] Add new fields ingested in background with valid mappings #172329

[UnifiedFieldList] Add new fields ingested in background with valid mappings #172329

kertal commented Nov 30, 2023 •

edited

Loading

kertal commented Dec 4, 2023

kertal commented Dec 7, 2023

kertal commented Dec 13, 2023

kertal commented Dec 13, 2023

kertal commented Dec 13, 2023

kertal Dec 14, 2023 •

edited

Loading

kertal commented Dec 15, 2023

drewdaemon left a comment

jughosta commented Jan 3, 2024

kertal commented Jan 8, 2024

kertal commented Jan 8, 2024

davismcphee left a comment

kibana-ci commented Jan 9, 2024

API count

davismcphee left a comment

mattkime commented Jan 9, 2024

[UnifiedFieldList] Add new fields ingested in background with valid mappings #172329

[UnifiedFieldList] Add new fields ingested in background with valid mappings #172329

Conversation

kertal commented Nov 30, 2023 • edited Loading

Summary

How to test

Checklist

kertal commented Dec 4, 2023

kertal commented Dec 7, 2023

kertal commented Dec 13, 2023

kertal commented Dec 13, 2023

kertal commented Dec 13, 2023

kertal Dec 14, 2023 • edited Loading

Choose a reason for hiding this comment

kertal commented Dec 15, 2023

drewdaemon left a comment

Choose a reason for hiding this comment

jughosta commented Jan 3, 2024

kertal commented Jan 8, 2024

kertal commented Jan 8, 2024

davismcphee left a comment

Choose a reason for hiding this comment

kibana-ci commented Jan 9, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Public APIs missing comments

Async chunks

Page load bundle

API count

History

davismcphee left a comment

Choose a reason for hiding this comment

mattkime commented Jan 9, 2024

kertal commented Nov 30, 2023 •

edited

Loading

kertal Dec 14, 2023 •

edited

Loading