[Kubernetes]K8s Dashboard Visualisation is not showing anyn information

[gizas]

Kibana version:
8.14.0

Describe the bug:
Only Pods visualisation of Kubernetes Cluster Overview Dashboard is not showing any information!

The specific Pods Visualisation uses runtime fields in order to create a range of specific metrics based on the value of a specific metric: the kubernetes.pod.status.phase.

For eg it creates the Running metric with value 1 when:

if (doc['kubernetes.pod.status.phase'].value == 'running') {emit(1) }

Same fields called Pending, Succeeded, Failed are being created per doc['kubernetes.pod.status.phase'].value

Steps to reproduce:

1. Install k8s cluster
2. Install Elastic Agent with default K8s Integration policy
3. Open [Metrics Kubernetes] Cluster Overview The Pods Visualisation does not show anything

Expected behavior:
Pods Visualisation to show the values of the Pods in cluster

Screenshots (if relevant):

I have confirmed that data is there and the Explore In Discover of relevant Visualisation returns metrics:

Provide logs and/or server output (if relevant):
I have created a copy from another Visualisation and edit it with the same configuration options like the non-working one. As said before, the specific visualisation uses some runtime fields to populate some new field. The weird thing is that the new duplicated Visualisation is working.

• This is the Visualisation that is not working
  export_not_working_pod.ndjson.txt

• This is a copy from another visaulisation, where same code works
  export_working_pod.ndjson.txt

See the comparison below:

Any additional context:

• The relevant K8s Dashboard: https://github.com/elastic/integrations/blob/main/packages/kubernetes/kibana/dashboard/kubernetes-f4dc26db-1b53-4ea2-a78b-1bfab8ea267c.json

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[gizas]

@elastic/obs-ux-infra_services-team please let me know if this is the correct team that handles Lens/ visualisation issues or I need to forward this to another team

[mlunadia]

@teresaalvarezsoler not sure if this falls into one of the teams you work with. This dashboard is our presentation to K8s users after onboarding and crucial that it works flawlessly. Any help you can provide would be great!
cc: @akhileshpok

[akhileshpok]

cc @flash1293. Is it possible to somehow link this issue to the Onboarding board, in order to help with tracking? Thanks

[flash1293]

@akhileshpok I don't think this should be on the onboarding initiative, it looks like an integration issue - maybe the field names changed or something like this and the visualization configuration needs to be fixed? Via the package manifest, it is owned by @elastic/obs-cloudnative-monitoring

So what I would like suggest:

• Someone from @elastic/obs-cloudnative-monitoring checks whether the panel is misconfigured or something else isn't right with the data ingestion/processing
• If the data and the config is right but the panel doesn't render right for some reason, we can reach out to the AppEx visualizations team

[gizas]

If the data and the config is right but the panel doesn't render right for some reason, we can reach out to the AppEx visualizations team

@flash1293 the second is the case here . Indeed (as I am part of @elastic/obs-cloudnative-monitoring ) I see that all data is there. Also nothing changed on our side on this dashboard and also I have made a duplication of the visualisation and The weird thing is that the new duplicated Visualisation is working. So same config on a new visualisation is working.

So we need some help from AppEx visualizations team. Which team is this to tag?

[gizas]

FYI I have a cluster running and if someone is assigned I can join a zoom for live troubleshooting

[flash1293]

@elastic/kibana-visualizations could you take a look at this together with @gizas ?

[gizas]

@dej611 Testing in version 8.13.1 :

Same visualisation is working:

The filter with label Status includes the '*'

FYI the filter by box has not double quotes (it is runing:* and not "running":*)

[dej611]

The issue seems to come from a changed behaviour in Elasticsearch.
The same query was working on 8.13.1 but returning 0 results in 8.14 .
I think this issue should be moved to the Elasticsearch repository.

[dej611]

Opened an issue on the Elasticsearch side: elastic/elasticsearch#110133

[dej611]

After some investigation on the ES we figured what's going on here.
Here's a quick recap:

• in 8.13 this PR removed the logic from Fleet to workout the default_field logic
  • this was done thinking about the ES behaviour of applying a ["*"] as default field
• What actually happens is that ES applies to the metrics-* indexes a default template that contains a default_field value of ["message"]
• Due to this final value in the default_field the search scope is reduced and producing a different result in 8.14

So the problem resides in the default_field value set for the metrics-* indexes.

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[flash1293]

Great sleuthing @dej611 - this is an important bug to fix.

@kpollich it seems like the simplest approach is to explicitly set the default_field to ["*"] in the index template, what do you think about that?

@felixbarny maybe you know more about the context of why the default metrics template is only setting message here?

[kpollich]

it seems like the simplest approach is to explicitly set the default_field to ["*"] in the index template, what do you think about that?

You mean setting this in the fleet-managed index template for metrics, correct?

[flash1293]

@kpollich That was what I meant, but after discussing with @felixbarny it makes sense to fix it in Elasticsearch, so no change needed on Kibana level. We should probably keep this issue open until it's fixed, validate the problem is actually gone and then close.

@felixbarny could you link the PR here once you have it up?

[felixbarny]

This was an oversight from when we removed the default field setting from logs@settings in elastic/elasticsearch#102456.

I've created ES PRs to remove the default field from metrics settings as well:

• Remove default_field: message from metrics index templates elasticsearch#110651
• https://github.com/elastic/elasticsearch-serverless/pull/2331