Exporting search CSV with deep data fails

[dyst5422]

Kibana version:
6.4
Elasticsearch version:
6.4
Describe the bug:
When exporting deep data to CSV, like

{
  "id": 123,
  "names": [{
    "first": "frank",
    "last": "underwood"
  },
  {
    "first": "joe",
    "last": "dimaggio"
  }]
}

where the columns id and names are selected, the exported CSV turns out empty with just headers.

Expected behavior:
I would expect it to just put the json string into that cell

[elasticmachine]

Pinging @elastic/kibana-app

[timroes]

Hi, could you please clarify on a view things: Where do you export that CSV? From Discovery? Also could you perhaps attach a CSV (one row would be enough) that shows that issue? Last but not least, it would be nice to see the mapping for that index (by querying /index_name/_mapping in the dev tools console), so check if that field is a regular array field or a nested field.

Cheers,
Tim

[dyst5422]

I'm exporting the CSV by doing generating a report from discover and downloading it from the management -> reporting page

Mapping

{"dictionaries:command:cruise":{"mappings":{"dictionaries:command:cruise":{"properties":{"arguments":{"properties":{"$enumValue":{"properties":{"comment":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"include_file":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"typedef":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"values":{"properties":{"dict_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsw_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"numeric_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"$stringValue":{"properties":{"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"pattern":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"$varArrayArguments":{"properties":{"$enumValue":{"properties":{"comment":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"include_file":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"typedef":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"values":{"properties":{"dict_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsw_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"numeric_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"$stringValue":{"properties":{"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"pattern":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"arg_order":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comment":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"count":{"properties":{"max":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"dict_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"io_format":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"length":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"length_type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"range":{"properties":{"max_val":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"min_val":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"units":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"arg_order":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"comment":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"count":{"properties":{"default_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"max":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"min":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"default_value":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"dict_name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"io_format":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"length":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"length_type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"range":{"properties":{"max_val":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"min_val":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"type":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"units":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"domain":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsw_implementation":{"properties":{"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"developer":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsw_custom_validation_method":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"fsw_enqueue_method":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"include_file":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"module":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"header":{"properties":{"project":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"version":{"properties":{"FSW_release_version_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"FSW_version_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"customer":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"dictionary_version_id":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"timestamp":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}},"schema_version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"spacecraft_restricted_phases":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"specification_maturity":{"properties":{"definition":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"implementation":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"stcode":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"stem":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sys_specification":{"properties":{"$operationCategory":{"properties":{"abbv":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"desc":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"name":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"description":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"operational_category":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"soe_desc":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}},"sys_engineer":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}},"version":{"type":"text","fields":{"keyword":{"type":"keyword","ignore_above":256}}}}}}}}

Commands_reporting.txt

[dyst5422]

Figured out the reason for this is that the request being fulfilled for generating the csv has the stored_fields option on it listing all fields currently available in the table. This is not a valid option with non-leaf fields. If that is removed, it works like a charm.

Per the note at the top of https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-stored-fields.html#search-request-stored-fields it looks like the stored_fields option is no longer best practice and as the request already uses the _source: { includes: [] }, it can safely be removed.

I will look to submit a PR with the fix to remove the stored_fields

[tsullivan]

duplicate of #25068