Enable stream stats in serverless

[SoniaSanzV]

Closes #192584

Summary

This PR sets the value of xpack.index_management.enableDataStreamStats to true in Serverless. The objective is to display the “Last updated” column in Index Management > Data Streams Table (propety is maximun_timestamp). It can be enabled now since the maximum_timestamp property was added to the Get data stream API (see elastic/elasticsearch#112303).

On the @elastic/kibana-management side, I've been looking at it carefully and I think only this change in config/serverless.yml. The rest of the necessary changes were already incorporated in this PR, testing included.
With this change the column is now displayed as expected:

I'm adding kibana-management as a reviewer just in case I'm missing something

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 43b8e7d

Failed CI Steps

• FTR Configs #9
• FTR Configs #9
• FTR Configs #62
• FTR Configs #62
• FTR Configs #64
• FTR Configs #64
• FTR Configs #68
• FTR Configs #68

Test Failures

• [job] [logs] FTR Configs #9 / Serverless Common UI - Management Index Management Data Streams allows to update data retention
• [job] [logs] FTR Configs #9 / Serverless Common UI - Management Index Management Data Streams allows to update data retention
• [job] [logs] FTR Configs #62 / Serverless Index Management APIs Data streams Get returns a single data stream by ID
• [job] [logs] FTR Configs #64 / Serverless Index Management APIs Data streams Get returns a single data stream by ID
• [job] [logs] FTR Configs #68 / Serverless Index Management APIs Data streams Get returns a single data stream by ID
• [job] [logs] FTR Configs #64 / Serverless Index Management APIs Data streams Get returns a single data stream by ID
• [job] [logs] FTR Configs #62 / Serverless Index Management APIs Data streams Get returns a single data stream by ID
• [job] [logs] FTR Configs #68 / Serverless Index Management APIs Data streams Get returns a single data stream by ID

Metrics [docs]

✅ unchanged

History

cc @SoniaSanzV

[elasticmachine]

🤖 Jobs for this PR can be triggered through checkboxes. 🚧

ℹ️ To trigger the CI, please tick the checkbox below 👇

• Click to trigger kibana-pull-request for this PR!
• Click to trigger kibana-deploy-project-from-pr for this PR!

[ElenaStoeva]

Hey @SoniaSanzV, thanks for picking this up! I think additional changes would be needed for this enhancement. If we set enableDataStreamStats to true in serverless, we would use the Get datastreams stats API (see here) but we don't want to display results from this API in serverless. We should instead use the Get data stream API which should now return a maximum_timestamp property which has the value that we want to display in the UI.

[SoniaSanzV]

Hey @SoniaSanzV, thanks for picking this up! I think additional changes would be needed for this enhancement. If we set enableDataStreamStats to true in serverless, we would use the Get datastreams stats API (see here) but we don't want to display results from this API in serverless. We should instead use the Get data stream API which should now return a maximum_timestamp property which has the value that we want to display in the UI.

I completely misunderstood, I thought the API was the same (the name is similar and I wasn't familiar with any of them). I'll continue to work on it.

[jloleysens]

Just wanted to FYI in case this hasn't been tested in QA, you can use the following label ci:project-deploy-elasticsearch and buildkite will provision a project based on this PR for manually testing in serverless.

[SoniaSanzV]

Just wanted to FYI in case this hasn't been tested in QA, you can use the following label ci:project-deploy-elasticsearch and buildkite will provision a project based on this PR for manually testing in serverless.

Thank you for the info. I changed the status of the PR to Draft because it needs more work.

[SoniaSanzV]

As this task has been stopped for quite some time, I think I should give an update on the status. After talking with Elena we saw that maybe you can use this same endpoint in stateful instead of using GET /_data_stream/{name}/_stats in one place and GET /_data_stream/{name}/verbose in another. I have asked Lee Hinman about this.
Second, when I tried to use the endpoint with the wildcard and the verbose flag in the console it returns a permissions error. This does not happen if you enter the name of a data stream. I need to investigate this further as the field is wanted to be displayed in the table.

Example request:

GET /_data_stream/*?expand_wildcards=all&verbose

Response:

{
  "error": {
    "root_cause": [
      {
        "type": "security_exception",
        "reason": "action [indices:monitor/data_stream/stats] is unauthorized for user [_xpack] with effective roles [_xpack] on restricted indices [.security-7,.security-profile-8], this action is granted by the index privileges [monitor,manage,all]"
      }
    ],
    "type": "security_exception",
    "reason": "action [indices:monitor/data_stream/stats] is unauthorized for user [_xpack] with effective roles [_xpack] on restricted indices [.security-7,.security-profile-8], this action is granted by the index privileges [monitor,manage,all]"
  },
  "status": 403
}

[azasypkin]

Changes in config/serverless.yml and test/plugin_functional/test_suites/core_plugins/rendering.ts LGTM