Cloud Run env variables reordering causing resource to update in-place #7467

skreinberg97 · 2020-10-08T17:39:54Z

Our cloud run resource needs to be updated every run as the environment variables are reordering. This seems to be completely on the Terraform side as our actual Cloud Run instance does not seem to be affected.

Terraform Version

v0.13.2

Affected Resource(s)

google_cloud_run_service

Terraform Configuration Files

 containers {
        image = "XXXXX"

        env {
          name = "ENV_1"
          value = data.google_secret_manager_secret_version.secrets["XXXX"].secret_data
        }

        env {
          name = "ENV_2"
          value = data.google_secret_manager_secret_version.secrets["XXXX"].secret_data
        }
}

Expected Behavior

For the environment variables to be in the same order, leading to no change in the resource.

Actual Behavior

~ env {
      ~ name  = "ENV_1" -> "ENV_2"
      ~ value = "XXX" -> "XXXX"
}
~ env {
      ~ name  = "ENV_2" -> "ENV_1"
      ~ value = "XXXX" -> "XXX"
}

Steps to Reproduce

terraform apply and/or terraform plan

Important Factoids

environment variables are coming from google_secret_manager_secret_version data object

References

b/272365080

The text was updated successfully, but these errors were encountered:

venkykuberan · 2020-10-08T20:25:50Z

I don't see that happening for the sample config below

  name     = "cloudrun-srv"
  location = "us-central1"

  template {
    spec {
      containers {
        image = "gcr.io/cloudrun/hello"
        env {
          name = "SOURCE"
          value = "remote"
        }
        env {
          name = "TARGET"
          value = "home"
        }
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }
  autogenerate_revision_name = true
}

Can you please attach the plan output and debug log for your apply

skreinberg97 · 2020-10-08T21:27:34Z

Hey! Thanks for your help, hopefully this is useful. I believe the issue is caused by the google_secret_manager_secret_version data object -- if the environment variables are hard-coded in plain text then we do not see the same behavior

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # google_cloud_run_service.default will be updated in-place
  ~ resource "google_cloud_run_service" "default" {
        id       = "REDACTED"
        location = "us-central1"
        name     = "terraform"
        project  = "REDACTED"
        status   = [
            {
                conditions                   = [
                    {
                        message = ""
                        reason  = ""
                        status  = "True"
                        type    = "Ready"
                    },
                    {
                        message = ""
                        reason  = ""
                        status  = "True"
                        type    = "ConfigurationsReady"
                    },
                    {
                        message = ""
                        reason  = ""
                        status  = "True"
                        type    = "RoutesReady"
                    },
                ]
                latest_created_revision_name = "terraform-bdff6"
                latest_ready_revision_name   = "terraform-bdff6"
                observed_generation          = 4
                url                          = "REDACTED"
            },
        ]

        metadata {
            annotations      = {
                "serving.knative.dev/creator"      = "REDACTED"
                "serving.knative.dev/lastModifier" = "REDACTED"
            }
            generation       = 4
            labels           = {
                "cloud.googleapis.com/location" = "us-central1"
            }
            namespace        = "REDACTED"
            resource_version = "AAWw4zi+fg0"
            self_link        = "REDACTED"
            uid              = "77fe171e-b695-427c-9eb0-833643c1523e"
        }

      ~ template {
            metadata {
                annotations = {
                    "autoscaling.knative.dev/maxScale"        = "1000"
                    "run.googleapis.com/vpc-access-connector" = "REDACTED"
                }
                generation  = 0
                labels      = {}
            }

          ~ spec {
                container_concurrency = 20
                service_account_name  = "REDACTED"

              ~ containers {
                    args    = []
                    command = []
                    image   = "REDACTED"

                  ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }
                 
                  ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                   ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                    ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                   ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }

                 ~ env {
                      ~ name  = "ENV" -> "EXISTING_ENV"
                      ~ value = "EXISTING_ENV_VALUE" -> "ENV_VALUE"
                    }
                    resources {
                        limits   = {
                            "cpu"    = "1000m"
                            "memory" = "256Mi"
                        }
                        requests = {}
                    }
                }
            }
        }

        traffic {
            latest_revision = true
            percent         = 100
        }
    }

Debug Output

2020/10/08 17:12:42 [DEBUG] command: asking for input: "\nDo you want to perform these actions in workspace \"REDACTED"?"

venkykuberan · 2020-10-10T10:24:35Z

Please attach the debug output, I want to see API request/response.

blueyed · 2021-10-05T12:30:10Z

I assume that it is using the data source, isn't it? (https://registry.terraform.io/providers/hashicorp/google/latest/docs/data-sources/secret_manager_secret_version)

If so, I could not verify this to be an issue, using:

Terraform v1.0.8
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v3.58.0
+ provider registry.terraform.io/hashicorp/google-beta v3.86.0
+ provider registry.terraform.io/hashicorp/random v3.1.0

@skreinberg97 is it still an issue for you? If so, can you provide more information, please?

markesha · 2021-10-19T11:32:08Z

Hi @blueyed,
I am experiencing the same with:

Terraform v1.0.9
on darwin_amd64
+ provider registry.terraform.io/hashicorp/google v3.89.0
+ provider registry.terraform.io/hashicorp/google-beta v3.89.0
+ provider registry.terraform.io/hashicorp/vault v2.24.1

In my case, it is google_secret_manager_secret_version resource, not data source.

Terraform will perform the following actions:
# google_cloud_run_service.service will be updated in-place
  ~ resource "google_cloud_run_service" "service" {
        id                         = "locations/region/namespaces/project/services/service"
        name                       = "service"
        # (4 unchanged attributes hidden)
      ~ template {

          ~ spec {
                # (3 unchanged attributes hidden)

              ~ containers {
                    # (2 unchanged attributes hidden)

                  - env {
                      - name = "ENV_VAR" -> null

                      - value_from {
                          - secret_key_ref {
                              - key  = "1" -> null
                              - name = "secret_manager_secret_name" -> null
                            }
                        }
                    }
                  + env {
                      + name = "ENV_VAR"

                      + value_from {
                          + secret_key_ref {
                              + key  = "1"
                              + name = "secret_manager_secret_name"
                            }
                        }
                    }

2021-10-19T12:59:44.812+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Retry Transport: Stopping retries, last request was successful: timestamp=2021-10-19T12:59:44.812+0200
2021-10-19T12:59:44.812+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-10-19T12:59:44.812+0200
2021-10-19T12:59:44.816+0200 [WARN]  Provider "registry.terraform.io/hashicorp/google-beta" produced an invalid plan for google_secret_manager_secret_version.vault_secrets["ENV_VAR"], but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .enabled: planned value cty.True for a non-computed attribute
2021-10-19T12:59:44.818+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:44 [DEBUG] Google API Response Details:
---[ RESPONSE ]--------------------------------------
HTTP/2.0 200 OK

2021-10-19T12:59:46.016+0200 [INFO]  provider.terraform-provider-google-beta_v3.89.0_x5: 2021/10/19 12:59:46 [DEBUG] Retry Transport: Returning after 1 attempts: timestamp=2021-10-19T12:59:46.016+0200
2021-10-19T12:59:46.046+0200 [WARN]  Provider "registry.terraform.io/hashicorp/google-beta" produced an unexpected new value for google_cloud_run_service.service during refresh.
      - .template[0].spec[0].containers[0].env: planned set element cty.ObjectVal(map[string]cty.Value{"name":cty.StringVal("ENV_VAR"), "value":cty.NullVal(cty.String), "value_from":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"secret_key_ref":cty.ListVal([]cty.Value{cty.ObjectVal(map[string]cty.Value{"key":cty.StringVal("2"), "name":cty.StringVal("secret_manager_secret_name")})})})})}) does not correlate with any element in actual

2021-10-19T12:59:46.128+0200 [WARN]  Provider "registry.terraform.io/hashicorp/google-beta" produced an invalid plan for google_cloud_run_service.service, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .traffic: block count in plan (1) disagrees with count in config (0)
      - .template[0].spec[0].containers[0].args: planned value cty.ListValEmpty(cty.String) for a non-computed attribute
      - .template[0].spec[0].containers[0].working_dir: planned value cty.StringVal("") for a non-computed attribute
      - .template[0].spec[0].containers[0].ports[0].protocol: planned value cty.StringVal("") for a non-computed attribute
2021-10-19T12:59:46.195+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-10-19T12:59:46.198+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google-beta/3.89.0/darwin_amd64/terraform-provider-google-beta_v3.89.0_x5 pid=43591
2021-10-19T12:59:46.198+0200 [DEBUG] provider: plugin exited
2021-10-19T12:59:46.199+0200 [INFO]  backend/local: plan operation completed

quulah · 2022-03-01T11:24:08Z

Encountering the same thing. We also have some Secret versions being managed directly, not via a data source.

For those I have generated an UUID as the name prefixed with secret-, since that's the way the UI / cloud side of Cloud Run seems to handle things. Not sure if it has an effect on this.

In any case, even the non-secret environment variables are shuffled around.

I wonder if we could just sort the environment variables a certain way to keep Terraform state happy and have the order be the same during all runs.

jeremad · 2022-12-13T10:48:25Z

It looks it is fixed by using https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/cloud_run_v2_service

FabianFrank · 2023-02-03T20:00:45Z

This also reproduces with google_cloud_run_v2_service if you have a value_source of secret_key_ref.

TheXienator · 2023-02-14T00:52:52Z

I am also having this issue still

alina-bylkova · 2023-07-13T13:10:08Z

Still having same issue using google_cloud_run_v2_service resource

FredoPhxlabs · 2023-08-31T20:03:49Z

Same issue here with google_cloud_run_v2_service

dominik1001 · 2023-10-16T12:52:22Z

A temporary workaround is to delete all env variables in the UI and then terraform apply again. But would really be nice to see this fixed.

heidi-manish · 2024-02-24T14:06:56Z

facing same problem with google_cloud_run_v2_service

XGManuelJager · 2024-03-01T11:40:17Z

The same is happening to me, but with secrets volumes

xiujuan-li · 2024-03-04T09:56:24Z

This my terraform version. I have the same problem. Is there any solution for help?

xiujuan-li · 2024-03-05T07:49:45Z

I found this env change happend after import cloud_run_v2_service resourve,looks like import changed the env sort in terraform state ***.tfstate .

~ env {
      ~ name  = "ENV_1" -> "ENV_2"
      ~ value = "XXX" -> "XXXX"
}
~ env {
      ~ name  = "ENV_2" -> "ENV_1"
      ~ value = "XXXX" -> "XXX"
}

meer-online · 2024-03-31T22:36:42Z

facing same problem with google_cloud_run_v2_service, was anyone able to figure out a solution?

scuba-st3v3 · 2024-05-01T13:48:48Z

Same issue here

duksis · 2024-05-09T06:58:19Z

Same here and also for regular variables - no secrets

ajoy39 · 2024-08-03T17:51:02Z

Throwing my voice into the void here, same issue, spot checking ENV VAR changes gets really hard when you have more than a handful of them.

galah92 · 2024-09-16T18:30:16Z

Same here.

manuelbernhardt · 2024-10-16T08:18:36Z

Same issue here on the latest version of the provider. We have about 20 services deployed and the output of terraform apply is constantly polluted by changes in env variables order (each env var references a secret from secret manager).

It is worth noting that I'm using google_secret_manager_secrets to load the secrets for a service and use a dynamic block in order to add them to the service template. From all that I've read so far in terms of documentation, I got the impression that order should be preserved at the terraform level during iteration, so I'm unsure where this re-ordering is caused by:

the implementation of the google terraform provider using a data structure that doesn't preserve order
the API response used by google_secret_manager_secrets not returning secrets in a consistent order

goody44 · 2024-11-01T02:37:07Z

Should this have been fixed by this change? It looks like it got included in v6.0.0 of the provider but the issue still persists for me. I'm adding the envs using dynamic blocks, if that makes a difference.

dvoetsGSOI · 2025-01-22T13:32:30Z

Hello, I'm still having the problem. Adding env vars via dynamic block always create a lot of env vars shuffling when doing a terraform plan

venkykuberan self-assigned this Oct 8, 2020

venkykuberan added bug waiting-response labels Oct 8, 2020

ghost removed the waiting-response label Oct 8, 2020

venkykuberan added the waiting-response label Oct 10, 2020

rileykarson unassigned venkykuberan Jul 19, 2021

github-actions bot removed the waiting-response label Oct 5, 2021

markesha mentioned this issue Dec 17, 2021

Cloud Run it's recreating all the secret environment variables when adding/removing a secret env #10634

Open

melinath added forward/linked service/run labels Mar 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cloud Run env variables reordering causing resource to update in-place #7467

Cloud Run env variables reordering causing resource to update in-place #7467

skreinberg97 commented Oct 8, 2020 •

edited by melinath

Loading

venkykuberan commented Oct 8, 2020

skreinberg97 commented Oct 8, 2020

venkykuberan commented Oct 10, 2020

blueyed commented Oct 5, 2021

markesha commented Oct 19, 2021 •

edited

Loading

quulah commented Mar 1, 2022

jeremad commented Dec 13, 2022

FabianFrank commented Feb 3, 2023 •

edited

Loading

TheXienator commented Feb 14, 2023

alina-bylkova commented Jul 13, 2023

FredoPhxlabs commented Aug 31, 2023

dominik1001 commented Oct 16, 2023

heidi-manish commented Feb 24, 2024

XGManuelJager commented Mar 1, 2024

xiujuan-li commented Mar 4, 2024

xiujuan-li commented Mar 5, 2024

meer-online commented Mar 31, 2024

scuba-st3v3 commented May 1, 2024

duksis commented May 9, 2024

ajoy39 commented Aug 3, 2024

galah92 commented Sep 16, 2024

manuelbernhardt commented Oct 16, 2024

goody44 commented Nov 1, 2024

dvoetsGSOI commented Jan 22, 2025

Cloud Run env variables reordering causing resource to update in-place #7467

Cloud Run env variables reordering causing resource to update in-place #7467

Comments

skreinberg97 commented Oct 8, 2020 • edited by melinath Loading

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

References

venkykuberan commented Oct 8, 2020

skreinberg97 commented Oct 8, 2020

venkykuberan commented Oct 10, 2020

blueyed commented Oct 5, 2021

markesha commented Oct 19, 2021 • edited Loading

quulah commented Mar 1, 2022

jeremad commented Dec 13, 2022

FabianFrank commented Feb 3, 2023 • edited Loading

TheXienator commented Feb 14, 2023

alina-bylkova commented Jul 13, 2023

FredoPhxlabs commented Aug 31, 2023

dominik1001 commented Oct 16, 2023

heidi-manish commented Feb 24, 2024

XGManuelJager commented Mar 1, 2024

xiujuan-li commented Mar 4, 2024

xiujuan-li commented Mar 5, 2024

meer-online commented Mar 31, 2024

scuba-st3v3 commented May 1, 2024

duksis commented May 9, 2024

ajoy39 commented Aug 3, 2024

galah92 commented Sep 16, 2024

manuelbernhardt commented Oct 16, 2024

goody44 commented Nov 1, 2024

dvoetsGSOI commented Jan 22, 2025

skreinberg97 commented Oct 8, 2020 •

edited by melinath

Loading

markesha commented Oct 19, 2021 •

edited

Loading

FabianFrank commented Feb 3, 2023 •

edited

Loading