[Reporting-CSV Export] Re-write CSV Export using SearchSource #88303

tsullivan · 2021-01-14T05:28:19Z

Summary

Resolves #81079

Closes [Reporting] CSV reporting does not work on frozen indices even with Search enabled in Advanced settings #60737
Closes Hide 'Time' column setting doesn't work with export to CSV #34377
Closes Export to CSV from a dashboard is not consistent with Discover CSV behavior #34376
Closes [Reporting] Not able to generate CSV report on object fields; Discover is able #25068
Closes [reporting] formatters are loaded from index pattern saved object, should be loaded via index patterns service #84388
Closes [Reporting] Inconsistent output between Export CSV from Discover, and Download CSV from Dashboard #68718
Closes Failing test: X-Pack Reporting Generate API Integration Tests.x-pack/test/reporting/api/generate/csv_saved_search·ts - CSV Generation from Saved Search ID Saved Search Features With filters and timebased data #37481

This PR adds a new export type keyed by csv_searchsource which runs the query and data formatting through SearchSource methods.

The existing csv export type has already been labeled in the code as deprecated. This export type will have to remain throughout 7.x as it could be referenced in Watcher scripts in customer environments.

Checklist

Delete any items that are not applicable to this PR.

For maintainers

This was checked for breaking API changes and was labeled appropriately

ppisljar · 2021-03-16T06:48:04Z

x-pack/plugins/reporting/server/export_types/csv_searchsource/generate_csv/generate_csv.ts

+
+    const fields = fieldValues[fieldSource];
+    // Check if field name values are string[] and if the fields are user-defined
+    if (isPlainStringArray(fields)) {


why do we try to first parse fields from searchsource instead of always just using the ones in the table ?

if its tabify dropping fields that don't exist in response (but do exist in searchsource) and discover is actually showing those fields, then we should probably update tabify.

ppisljar

code LGTM

kibanamachine · 2021-03-16T18:37:55Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`reporting`	51	50	-1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`discover`	409.1KB	408.4KB	-743.0B
`reporting`	34.6KB	34.6KB	+1.0B
total			-742.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`reporting`	96.5KB	93.9KB	-2.6KB

History

💚 Build #113282 succeeded d786783
💔 Build #113258 failed 78d9a8a
💚 Build #113251 succeeded 249c70f
💔 Build #113235 failed 9d87153
💚 Build #113216 succeeded f183650

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

majagrubic

I had a quick look at this, changes look good to me. I am not very familiar with the reporting code. Discover-related code LGTM 👍

…c#88303) * [Reporting-CSV Export] Re-write CSV Export using SearchSource * replace PIT solution with scan-and-scroll * update tests * cleanup * simplify pr * update docs * update docs * update telemetry schema * use getSearchRequestBody instead of flatten * Revert "update docs" This reverts commit ab9f4d9. * optimize some async calls * cleanup * --wip-- [skip ci] * fix telemetry schema * fix telemetry tests * fix snapshot * api docs * api doc updates * use import type * format the data through chains of maps * add another saved search to reporting/ecommerce_kibana * add a failing test * add error logging to query failures * put clear scroll in a finally so the ES error can be captured * log dat error * set dat fieldsFromSource * --wip-- [skip ci] * Revert "add another saved search to reporting/ecommerce_kibana" This reverts commit 6edf26e. * functional test fixes * clean up ecommerce test archive * add test for new search with fieldsFromSource set * add tests and refactor tests * cleanup redundant conditionals * add GenerateCsv.getFields * fix some tests * fix double-escaping * fix test snapshots and refactoring * fix other tests * fix test * fix default index pattern in functional tests * fix ts and sort fields when they come from API response * --wip-- [skip ci] * fix formatting and increase maxSizeBytes for testing * remove client-side logic for sanitizing fields * do not prepend timefield name if it already is a column * test the logic to prepend timeField * test the logic to sort the fields * fix functional test * preserve the error from data.search * add functional test for ES returning an error * fix snapshot Co-authored-by: Kibana Machine <[email protected]>

…88303) (#94757) * [Reporting-CSV Export] Re-write CSV Export using SearchSource (#88303) * [Reporting-CSV Export] Re-write CSV Export using SearchSource * replace PIT solution with scan-and-scroll * update tests * cleanup * simplify pr * update docs * update docs * update telemetry schema * use getSearchRequestBody instead of flatten * Revert "update docs" This reverts commit ab9f4d9. * optimize some async calls * cleanup * --wip-- [skip ci] * fix telemetry schema * fix telemetry tests * fix snapshot * api docs * api doc updates * use import type * format the data through chains of maps * add another saved search to reporting/ecommerce_kibana * add a failing test * add error logging to query failures * put clear scroll in a finally so the ES error can be captured * log dat error * set dat fieldsFromSource * --wip-- [skip ci] * Revert "add another saved search to reporting/ecommerce_kibana" This reverts commit 6edf26e. * functional test fixes * clean up ecommerce test archive * add test for new search with fieldsFromSource set * add tests and refactor tests * cleanup redundant conditionals * add GenerateCsv.getFields * fix some tests * fix double-escaping * fix test snapshots and refactoring * fix other tests * fix test * fix default index pattern in functional tests * fix ts and sort fields when they come from API response * --wip-- [skip ci] * fix formatting and increase maxSizeBytes for testing * remove client-side logic for sanitizing fields * do not prepend timefield name if it already is a column * test the logic to prepend timeField * test the logic to sort the fields * fix functional test * preserve the error from data.search * add functional test for ES returning an error * fix snapshot Co-authored-by: Kibana Machine <[email protected]> * simplify logging tags * update snapshots * --wip-- [skip ci] * fix types * update jest snapshot * Revert "simplify logging tags" This reverts commit e844dbd. * Revert "update jest snapshot" This reverts commit 8849752. * fix types again Co-authored-by: Kibana Machine <[email protected]>

…-action * 'master' of github.com:elastic/kibana: (44 commits) Migrate the optimizer mixin to core (#94272) Replace EuiCodeBlock with JsonCodeEditor in DiscoverGrid (#92442) [ML] Anomaly Detection: Migrate validation messages links to use docLinks. (#94568) [Lists][Exceptions] - Adding basic linting, i18n and storybook support (#94772) [Fleet] Add test/fix for invalid/missing ids in bulk agent reassign (#94632) [Security Solution] [Cases] Move create page components and dependencies to Cases (#94444) [ML] Data Frame Analytics accessibility tests: fix flaky outlier creation test (#94735) [Security Solutions] Remove commented out old linter rules (#94753) [App Search] Role mappings migration part 2 (#94461) [CI] Update Backport action inputs to match updated ones (#94721) [chore] Remove the infra team from CODEOWNERS (#94740) [Connectors UI] Make UI use new connector APIs (#94180) [ML] Use indices options in anomaly detection job wizards (#91830) Remove `string` as a valid registry var type value (#94174) [Alerts] Replaces legacy es client with the ElasticsearchClient for alerts and triggers_actions_ui plugins. (#93364) [Reporting-CSV Export] Re-write CSV Export using SearchSource (#88303) chore(NA): upgrade bazel rules nodejs to v3.2.2 (#94726) [APM] Settings: Update layout and update/add descriptions (#94398) skip flaky suite (#94666) [XY axis] Integrates legend color picker with the eui palette (#90589) ... # Conflicts: # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/cold_phase/cold_phase.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/components/phases/shared_fields/searchable_snapshot_field/searchable_snapshot_field.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/constants.ts # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/edit_policy.tsx # x-pack/plugins/index_lifecycle_management/public/application/sections/edit_policy/form/configuration_context.tsx # x-pack/plugins/index_lifecycle_management/public/shared_imports.ts

elasticmachine · 2021-05-03T09:13:42Z

Pinging @elastic/kibana-app-services (Team:AppServices)

tsullivan force-pushed the reporting/searchsource-exporttype branch 6 times, most recently from 280d4b9 to 54c23fc Compare January 20, 2021 01:21

tsullivan mentioned this pull request Jan 20, 2021

[AppServices] Allow plugin to access SearchSource SerializedFields th… #88551

Closed

9 tasks

tsullivan force-pushed the reporting/searchsource-exporttype branch 7 times, most recently from 01eebe4 to 21b4cee Compare January 29, 2021 21:31

tsullivan added the (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead label Jan 29, 2021

tsullivan force-pushed the reporting/searchsource-exporttype branch 2 times, most recently from 56f7938 to 91c4372 Compare January 29, 2021 23:29

tsullivan mentioned this pull request Feb 1, 2021

[Reporting] Not able to generate CSV report on object fields; Discover is able #25068

Closed

tsullivan changed the title ~~reporting csv generation from searchSource (new export type)~~ [Reporting-CSV Export] Re-write CSV Export using SearchSource Feb 1, 2021

tsullivan force-pushed the reporting/searchsource-exporttype branch 4 times, most recently from ab59e88 to 5d6550e Compare February 1, 2021 22:53

tsullivan mentioned this pull request Feb 2, 2021

CSV reporting needs to handle empty reports better. #18320

Closed

tsullivan force-pushed the reporting/searchsource-exporttype branch 2 times, most recently from e24a638 to 8777211 Compare February 2, 2021 00:41

tsullivan added release_note:enhancement v7.12.0 v8.0.0 labels Feb 2, 2021

tsullivan added 10 commits March 15, 2021 11:12

fix formatting and increase maxSizeBytes for testing

f183650

remove client-side logic for sanitizing fields

3bbce70

do not prepend timefield name if it already is a column

ae16898

test the logic to prepend timeField

c7694f6

test the logic to sort the fields

9d87153

fix functional test

249c70f

preserve the error from data.search

c0b4f48

Merge branch 'master' into reporting/searchsource-exporttype

2bb267e

add functional test for ES returning an error

78d9a8a

fix snapshot

d786783

ppisljar reviewed Mar 16, 2021

View reviewed changes

ppisljar approved these changes Mar 16, 2021

View reviewed changes

Merge branch 'master' into reporting/searchsource-exporttype

06c7e38

majagrubic approved these changes Mar 16, 2021

View reviewed changes

tsullivan merged commit ec41ae3 into elastic:master Mar 16, 2021

tsullivan deleted the reporting/searchsource-exporttype branch March 16, 2021 18:54

tsullivan mentioned this pull request Mar 16, 2021

[7.x] [Reporting-CSV Export] Re-write CSV Export using SearchSource (#88303) #94757

Merged

spalger mentioned this pull request Apr 1, 2021

Failing ES Promotion: dashboard Reporting Download CSV Field Formatters and Scripted Fields Download CSV export of a saved search panel #96000

Closed

timroes added the Team:AppServices label May 3, 2021

tsullivan mentioned this pull request May 19, 2021

[Reporting] Add guidance to users on the new CSV export type #100363

Closed

tsullivan mentioned this pull request Jun 1, 2021

Kibana 7.12 reporting - blank csv #100470

Closed

tsullivan mentioned this pull request Jul 27, 2021

[Reporting Management] Consolidate report job warnings and add warning for deprecated types #106184

Merged

6 tasks

tsullivan mentioned this pull request Sep 20, 2021

[Reporting] Fix incorrect number of hits being exported from CSV #112406

Closed

1 task

tsullivan mentioned this pull request Oct 27, 2022

[Reporting] Improve deep pagination method for CSV export #143144

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Reporting-CSV Export] Re-write CSV Export using SearchSource #88303

[Reporting-CSV Export] Re-write CSV Export using SearchSource #88303

tsullivan commented Jan 14, 2021 •

edited

Loading

ppisljar Mar 16, 2021

ppisljar Mar 16, 2021

ppisljar left a comment

kibanamachine commented Mar 16, 2021

majagrubic left a comment

elasticmachine commented May 3, 2021

[Reporting-CSV Export] Re-write CSV Export using SearchSource #88303

[Reporting-CSV Export] Re-write CSV Export using SearchSource #88303

Conversation

tsullivan commented Jan 14, 2021 • edited Loading

Summary

Checklist

For maintainers

ppisljar Mar 16, 2021

Choose a reason for hiding this comment

ppisljar Mar 16, 2021

Choose a reason for hiding this comment

ppisljar left a comment

Choose a reason for hiding this comment

kibanamachine commented Mar 16, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

majagrubic left a comment

Choose a reason for hiding this comment

elasticmachine commented May 3, 2021

tsullivan commented Jan 14, 2021 •

edited

Loading