cannot override "__name__" in google_monitoring_alert_policy, condition_prometheus_query_language block using the labels option.

[nick-lawrence]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version & Provider Version(s)

Terraform v1.9.8
on linux_amd64

• provider registry.terraform.io/hashicorp/google v6.16.0
• provider registry.terraform.io/hashicorp/google-beta v6.16.0
• provider registry.terraform.io/hashicorp/random v3.6.3
• provider registry.terraform.io/hashicorp/time v0.12.1
• provider registry.terraform.io/hashicorp/tls v4.0.6

Affected Resource(s)

google_monitoring_alert_policy

within that resource condition_prometheus_query_language

Terraform Configuration

This a cut and paste of my partial configuration. The labels doesn't let me set "__name__", the documentation implies that I should be able to set it. I want to be able to set it so that google incident notifications include a metric name for complex queries where 'name' is removed from the query.

resource "google_monitoring_alert_policy" "promql_policies" {
  project      = var.project_id
  for_each     = { for i, policy in local.policies["promql_policies"] : i => policy }
  display_name = each.value["display_name"]
  combiner     = each.value["combiner"]

  dynamic "conditions" {
    for_each = { for i, condition in each.value["conditions"] : i => condition }
    content {
      display_name = conditions.value["display_name"]
      condition_prometheus_query_language {
        query                     = conditions.value["query"]
        duration                  = conditions.value["duration"]
        disable_metric_validation = true

        # >>>>>>> This is the important part
        # >>>>>>> Overriding "__name__" causes the apply to fail
        labels = { "__name__" = conditions.value["metric_name"] }
      }
    }
  }

  severity = each.value["severity"]
  enabled  = each.value["enabled"]
  documentation {
    subject   = "${each.value["documentation"]["subject"]} (${var.project_id})"
    mime_type = "text/markdown"
    content   = each.value["documentation"]["content"]

  }

Debug Output

No response

Expected Behavior

Intending to override/set the "__name__" label for my query.

The reason is when goggle sends me a notification email, it includes a
metric : missing

label.

I believe that 'metric' is the "__name__" from the query, because my query is complex, the label is not in the result, and thus it's reported as missing.

I'd like to set that to avoid confusion when reading the notification.

Documentation implies that I should be able to do this:

labels - (Optional) Labels to add to or overwrite in the PromQL query result. Label names must be valid. Label values can be templatized by using variables. The only available variable names are the names of the labels in the PromQL result, including "name" and "value". "labels" may be empty. This field is intended to be used for organizing and identifying the AlertPolicy

Actual Behavior

Apply fails with:

"__name__": PromQL names beginning with __ are reserved for internal use.

Steps to reproduce

1. terraform apply

Important Factoids

I tried using "name", but this just created another label. Same thing with "metric" which is how google displays this.

It seems like "name" should be allowed, but is being blocked.

References

No response

[ggtisc]

Hi @nick-lawrence

As the error message indicates the usage of label names beginning with __ (two "_") are reserved for internal use. And you can confirm this statement in the following link https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels

Error: Error creating AlertPolicy: googleapi: Error 400: Field alert_policy.conditions[0].condition_prometheus_query_language.labels had an invalid value of "__name__": PromQL names beginning with __ are reserved for internal use.

You could use any other different labels for your purposes or if you want to parameterize the name of the label you could use a variable like this:

labels = { 
        var.label_name = var.label_name_value
}

I leave the following OFFICIAL DOCUMENTATION links for more information about this topic:

• https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/monitoring_alert_policy#labels-2
• https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies#resource:-alertpolicy
• https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies#condition
• https://cloud.google.com/monitoring/api/ref_v3/rest/v3/projects.alertPolicies#PrometheusQueryLanguageCondition
• https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels
• https://prometheus.io/docs/practices/naming/#labels

[nick-lawrence]

Appreciate the reply,

I understand that....and it seems plausable to me that this would be an unsupported operation.

Here's the details the rational and source of my confusion.

The notification emails that are sent by google include the concept of a metric name. I believe when a PromQL query is used, google get's this name from the __name__ label. In my query, __name__ is (correctly) removed, and the notification has __missing__ for the metric label. If I create my own "metric" label, it does show up in the message, but it is an extra label, there are two "metric" label values in the email. The __missing__ is not really a big deal, but it is confusing.

So I went looking for a way to get rid of that, or change it to something meaningful. I came across this section of the documentation that you linked to (https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/monitoring_alert_policy#labels-1)

The only available variable names are the names of the labels in the PromQL result, including "name" and "value". "labels" may be empty. This field is intended to be used for organizing and identifying the AlertPolicy

OK so in the actual text, name is bolded, which makes me think that this is a markdown typo for __name__. And maybe that makes sense, I can change the __name__ and if I wanted to the value...and that might avoid my issue. Maybe that's why that note is in there.

Except it doesn't work.

Totally fine if you want to close this or switch it to enhancment or doc update. But it seems like something is missing here to me.

Thanks!

[ggtisc]

Sure @nick-lawrence, I'm forwarding this issue to evaluate if it is possible this kind of enhancement