OSSM-3374 Add servicemesh_members metric

[bmangoen]

We want to add the Service Mesh Control Plane Namespaces counter metric that would be exposed by Telemetry

[jewertow]

@bmangoen could you rebase this PR?

[dgn]

how best to test this? it looks like it's integrated with cluster-monitoring, but I don't know how to verify that

[simonpasquier]

I'd suggest to implement a Collector interface which is less error-prone from my experience than setting/deleting the metric on the fly.
Here is an example of such implementation.

[bmangoen]

how best to test this? it looks like it's integrated with cluster-monitoring, but I don't know how to verify that

@dgn You can check these metrics from the OpenShift web console Observe > Metrics but you first need to label the operator namespace oc label ns openshift-operators openshift.io/cluster-monitoring=true (it will be done later at the operator install via OLM)

[bmangoen]

/retest

[luksa]

@bmangoen is this now ready to be merged or does it need more work? is the metric name now correct?

[bmangoen]

@luksa It is ready to be merged IMO. I renamed the metric service_member_count as recommended servicemesh_members.

[rcernich]

@bmangoen , it looks like this needs to be rebased

[jewertow]

I would like to have metrics, which will answer to the following questions:

1. How many SMCPs of a particular version are deployed.
2. How many tenants our customers have per cluster.
3. Average, maximum and total number of namespaces included in a mesh.

Current metrics do not provide enough information and we can only conclude which SMCP versions are still used, but not how many of a particular version and how many tenants users have per cluster, because metrics are summed up. Additionally, SMCP version is irrelevant in the context of namespaces included in the mesh, so this will create unnecessary time series.

I suggest to collect the following metrics:

1. servicemesh_controlplanes{version,mode,namespace} - we will not export the namespace, but it's necessary to create unique metrics for each SMCP controller.
2. servicemesh_members{mode,namespace} - namespace will not be exported but it's necessary to create a unique metrics for each tenant.

And then we have to create the following PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: service-mesh-2
  namespace: openshift-operators
spec:
  groups:
  - name: service-mesh-2-control-planes
    rules:
    - expr: |-
        sum by (version,mode) (servicemesh_controlplanes)
      record: 'cluster:servicemesh_controlplanes:sum'
  - name: service-mesh-2-members
    rules:
    - expr: |-
        avg by (mode) (servicemesh_members)
      record: 'cluster:servicemesh_members:avg'
     - expr: |-
        max by (mode) (servicemesh_members)
      record: 'cluster:servicemesh_members:max'
     - expr: |-
        sum by (mode) (servicemesh_members)
      record: 'cluster:servicemesh_members:sum'

[openshift-ci]

@bmangoen: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/unit	1a85505	link	true	/test unit
ci/prow/lint	1a85505	link	true	/test lint
ci/prow/gencheck	1a85505	link	true	/test gencheck

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[bmangoen]

I would like to have metrics, which will answer to the following questions:

1. How many SMCPs of a particular version are deployed.
2. How many tenants our customers have per cluster.
3. Average, maximum and total number of namespaces included in a mesh.

Current metrics do not provide enough information and we can only conclude which SMCP versions are still used, but not how many of a particular version and how many tenants users have per cluster, because metrics are summed up. Additionally, SMCP version is irrelevant in the context of namespaces included in the mesh, so this will create unnecessary time series.

I suggest to collect the following metrics:

1. servicemesh_controlplanes{version,mode,namespace} - we will not export the namespace, but it's necessary to create unique metrics for each SMCP controller.
2. servicemesh_members{mode,namespace} - namespace will not be exported but it's necessary to create a unique metrics for each tenant.

And then we have to create the following PrometheusRule:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: service-mesh-2
  namespace: openshift-operators
spec:
  groups:
  - name: service-mesh-2-control-planes
    rules:
    - expr: |-
        sum by (version,mode) (servicemesh_controlplanes)
      record: 'cluster:servicemesh_controlplanes:sum'
  - name: service-mesh-2-members
    rules:
    - expr: |-
        avg by (mode) (servicemesh_members)
      record: 'cluster:servicemesh_members:avg'
     - expr: |-
        max by (mode) (servicemesh_members)
      record: 'cluster:servicemesh_members:max'
     - expr: |-
        sum by (mode) (servicemesh_members)
      record: 'cluster:servicemesh_members:sum'

@jewertow Does this still make sense to implement?