feat(example,metrics): kube-state-metrics to monitor custom resource …

[sebastiangaiser]

…state

In order to monitor the state of custom resources (CR) inside of the Kubernetes cluster, kube-state-metrics can be deployed. This describes the deployment using the prometheus-community Helm chart.

Issue: #10276

Type of change

Select the type of your PR

• Enhancement / new feature

Description

In order to monitor the state of custom resources (CR) inside of the Kubernetes cluster, kube-state-metrics can be deployed. This describes the deployment using the prometheus-community Helm chart.

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[scholzj]

Thanks for the PR. I think this is a good idea. I think we should consider some additional things here:

• Do we want to document this? (I guess we have some notes on what is int he examples, so we should mention it?)
• Do we want to have some no-Helm variant of this in the examples? As we in general have a pure YAML based installation as the primary source and as many users do not use Helm, so this useful only for some users.
• Should we provide this for all custom resources and deprecated (and later remove) the state metrics provided by Strimzi CO? (this might not be a task for this PR, but it should be considered when adopting this)
• Do we want to have some System Test coverage?

CC @strimzi/maintainers

[sebastiangaiser]

Should we provide this for all custom resources

I think this would be a good idea. E.g. reconcile for a Kafka resource fails, a KafkaRebalance has problems, ... But I didn't had any problems, yet. Neither I'm not sure which fields are relevant for all resources.

Do we want to have some no-Helm variant of this in the examples?

I think this depends on how you would like to deploy the monitoring. The Flux example uses the kube-prometheus-stack and injects the values there (into the Helm release). Personally I'm not a big fan of mixing my monitoring stack with some Flux/Strimzi/... specific monitoring. So I ended up deploying a second ksm for Flux and Strimzi (yes I use Flux).

[scholzj]

I think this depends on how you would like to deploy the monitoring. The Flux example uses the kube-prometheus-stack and injects the values there (into the Helm release). Personally I'm not a big fan of mixing my monitoring stack with some Flux/Strimzi/... specific monitoring. So I ended up deploying a second ksm for Flux and Strimzi (yes I use Flux).

Well, the Prometheus part seems to be just a custom resource that can have its own YAMl as well. I also assume that the confguration will end up in some ConfigMap or some environment variables to configure the Kube State Metrics? So that can be described or stored in separate YAMLs.

I think this would be a good idea. E.g. reconcile for a Kafka resource fails, a KafkaRebalance has problems, ... But I didn't had any problems, yet. Neither I'm not sure which fields are relevant for all resources.

I guess a start might be to replicate what the Cluster operator does -> a metric to indicate if the resource is ready or not. That would allow us to drop it from the Cluster operator. We can improve things later as some ideas pop-out. But we can wait for this for some discussion and have others chime in with their thoughts.

[ppatierno]

Following my thoughts ... and I am not saying I am against it but just thinking aloud:

• using kube-state-metrics add one more component to the overall deployment. Something we don't need today.
• why is it better deploying one more component to expose metrics when they are already exposed by the operator? Because we see kube-state-metrics more flexible when it comes to add new metrics (change its configuration, no change in the operator code).
• for sure we would need a way to deploy it without Helm Charts.
• for backward compatibility the provided configuration should export the same metrics that we have today in the operator. No less. We could break users' systems using the removed ones. We can add more metrics in the future.
• we should think about how the upgrade works, when you move from a Strimzi version exposing these metrics out of box to a newer version which needs kube-state-metrics to be deployed.

[scholzj]

@ppatierno Keep in mind that this is not removing anything at this point. It just adds metrics similar to those we never implemented in UO and TO.

[scholzj]

Discussed on the community call on 5.9.2024: This PR should be closed and the discussion should continue on the #10276 issue and the related proposal. Thanks for opening this topic @sebastiangaiser