[data.plugin] Use CBOR instead of JSON as transfer format

[thomasneirynck]

Describe the feature:

Use cbor instead of JSON as transfer format of the data-plugin (e.g. ese and esql search-strategies)

Describe a specific use case for the feature:

Earlier investigation has shown that using cbor provides a marginal benefit over using JSON (5% size reduction, faster decoding on client).

Changing the transfer format is now possible due to #193060. In that PR, data-service now uses metadata about async-searches from the http-headers rather than decoding it from the body. It would allow leaving the cbor returned from ES intact and only decode it on client.

Overall, this would be a small performance benefit in isolation for a single search. It would have broad overall impact on all applications using the data-plugin (Discover, Dashboard) since that benefit would apply to all searches issued by those apps (DSL/ES|QL in Lens and Discover).

[elasticmachine]

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

[kertal]

One question, would't CBOR instead of JSON introduce another barrier in debugging problems? Since CBOR is encoded, I guess it's harder do debug when we get HAR reporting proplems. We then would need decoders to have a look at the responses raw data? Does a marginal benefit in terms of speed and performance a higher cost in maintenance of dev & support side?

[thomasneirynck]

@kertal

One question, would't CBOR instead of JSON introduce another barrier in debugging problems?

Yes, 100% agreed. That would be a drawback.

It would come down to make a trade-off between the gain (if any) and a decrease of debugability. We do not have a good idea of the gain at this point.