Error if using in filters metadata.user_labels.

[vladadm]

Hello,
I want to collect GKE metrics from GCP Monitoring.

if i run:

stackdriver_exporter 
    --google.project-id=qa 
    --monitoring.metrics-type-prefixes "kubernetes.io/node/memory/used_bytes"  
    --monitoring.filters 'kubernetes.io/node/memory/used_bytes:resource.label."cluster_name"="cluster00"' 
    --log.level=debug

metrics collecting and all right

ts=2023-11-28T14:38:27.542Z caller=monitoring_collector.go:372 level=debug msg="listing Google Stackdriver Monitoring metric descriptors starting with" prefix=kubernetes.io/node/memory/used_bytes ts=2023-11-28T14:38:28.314Z caller=monitoring_collector.go:274 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=kubernetes.io/node/memory/used_bytes ts=2023-11-28T14:38:28.315Z caller=monitoring_collector.go:304 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"kubernetes.io/node/memory/used_bytes\" AND (resource.label.\"cluster_name\"=\"cluster_name\")" ts=2023-11-28T14:38:28.465Z caller=monitoring_collector.go:387 level=debug msg="Done reporting monitoring metrics"

but if add the additional filter from metadata.user_labels collection and run it:

stackdriver_exporter 
    --google.project-id=qa 
    --monitoring.metrics-type-prefixes "kubernetes.io/node/memory/used_bytes" 
    --monitoring.filters 'kubernetes.io/node/memory/used_bytes:resource.label."cluster_name"="cluster00" AND metadata.user_labels."beta.kubernetes.io/os"="windows"' 
    --log.level=debug

i receive the error:
ts=2023-11-28T14:44:19.693Z caller=monitoring_collector.go:274 level=debug msg="retrieving Google Stackdriver Monitoring metrics for descriptor" descriptor=kubernetes.io/node/memory/used_bytes ts=2023-11-28T14:44:19.694Z caller=monitoring_collector.go:304 level=debug msg="retrieving Google Stackdriver Monitoring metrics with filter" filter="metric.type=\"kubernetes.io/node/memory/used_bytes\" AND (resource.label.\"cluster_name\"=\"cluster_name\" AND metadata.user_labels.\"beta.kubernetes.io/os\"=\"windows\")" ts=2023-11-28T14:44:19.857Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=kubernetes.io/node/memory/used_bytes err="googleapi: Error 400: Field filter had an invalid value of \"metric.type=\"kubernetes.io/node/memory/used_bytes\" AND (resource.label.\"cluster_name\"=\"cluster_name\" AND metadata.user_labels.\"beta.kubernetes.io/os\"=\"windows\")\": Resource metadata filters are only allowed on aligned metrics., badRequest" ts=2023-11-28T14:44:19.857Z caller=monitoring_collector.go:387 level=debug msg="Done reporting monitoring metrics"

i try to use the command:

stackdriver_exporter 
      --google.project-id=qa 
      --monitoring.metrics-type-prefixes "kubernetes.io/node/memory/used_bytes" 
      --monitoring.filters 'kubernetes.io/node/memory/used_bytes:resource.label."cluster_name"="cluster00"' 
      --monitoring.filters 'kubernetes.io/node/memory/used_bytes:metadata.user_labels."beta.kubernetes.io/os"="windows"'
      --log.level=debug

error:
ts=2023-11-28T14:47:13.104Z caller=monitoring_collector.go:315 level=error msg="error retrieving Time Series metrics for descriptor" descriptor=kubernetes.io/node/memory/used_bytes err="googleapi: Error 400: Field filter had an invalid value of \"metric.type=\"kubernetes.io/node/memory/used_bytes\" AND (resource.label.\"cluster_name\"=\"cluster_name\") AND (metadata.user_labels.\"beta.kubernetes.io/os\"=\"windows\")\": Resource metadata filters are only allowed on aligned metrics., badRequest" ts=2023-11-28T14:47:13.104Z caller=monitoring_collector.go:228 level=error msg="Error while getting Google Stackdriver Monitoring metrics" err="googleapi: Error 400: Field filter had an invalid value of \"metric.type=\"kubernetes.io/node/memory/used_bytes\" AND (resource.label.\"cluster_name\"=\"cluster_name\") AND (metadata.user_labels.\"beta.kubernetes.io/os\"=\"windows\")\": Resource metadata filters are only allowed on aligned metrics., badRequest"

how i can select metrics with using metadata.user_labels. ?

[michel-numan]

+1