Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"billing_project" in provider configuration ignored if GOOGLE_CLOUD_QUOTA_PROJECT environment variable is set #17882

Open
mkielar opened this issue Apr 17, 2024 · 10 comments · May be fixed by GoogleCloudPlatform/magic-modules#12411
Open

"billing_project" in provider configuration ignored if GOOGLE_CLOUD_QUOTA_PROJECT environment variable is set #17882

mkielar opened this issue Apr 17, 2024 · 10 comments · May be fixed by GoogleCloudPlatform/magic-modules#12411
Labels
bug service/terraform

Comments

@mkielar
Copy link

mkielar commented Apr 17, 2024

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
  • Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment.
  • If an issue is assigned to a user, that user is claiming responsibility for the issue.
  • Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

Terraform v1.7.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v5.25.0

Affected Resource(s)

google_storage_bucket_iam_member (and probably others)

Terraform Configuration

# In "example-project-1", create a Service Account.
provider "google" {

  # Each project manages its own billing.
  user_project_override = true
  billing_project       = "[MASKED-PROJECT-A]" # Project A
  project               = "[MASKED-PROJECT-A]" # Project A
  region                = "europe-west3"

  # Note I'm impersonating different SAs here and below. In my scenario, each GCP Project 
  # has its own, dedicated deployer SA.
  impersonate_service_account = "infrastructure-deployer@[MASKED-PROJECT-A].iam.gserviceaccount.com"
}

resource "google_service_account" "example_sa" {
  account_id   = "example-sa"
}

# In "example-project-2", bind that Service Account as Viever for some existing bucket.
provider "google" {
  alias = "p2"

  # Each project manages its own billing.
  user_project_override = true
  billing_project       = "[MASKED-PROJECT-B]" # Project B
  project               = "[MASKED-PROJECT-B]" # Project B
  region                = "europe-west3"

  # Note I'm impersonating different SAs here and above. In my scenario, each GCP Project 
  # has its own, dedicated deployer SA.
  impersonate_service_account = "infrastructure-deployer@[MASKED-PROJECT-B].iam.gserviceaccount.com"
}

resource "google_storage_bucket_iam_member" "sa_viewer_permission" {
  provider = google.p2

  bucket = "[MASKED-PROJECT-B]-data-source-[MASKED]" # Some existing bucket
  member = "serviceAccount:${google_service_account.example_sa.email}"
  role   = "roles/storage.objectViewer"
}

Debug Output

2024-04-17T14:04:51.027+0200 [INFO] backend/local: apply calling Apply
2024-04-17T14:04:51.027+0200 [DEBUG] Building and walking apply graph for NormalMode plan
2024-04-17T14:04:51.027+0200 [DEBUG] Resource state not found for node "google_storage_bucket_iam_member.sa_viewer_permission", instance google_storage_bucket_iam_member.sa_viewer_permission
2024-04-17T14:04:51.027+0200 [DEBUG] ProviderTransformer: "google_service_account.example_sa (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"]
2024-04-17T14:04:51.028+0200 [DEBUG] ProviderTransformer: "google_storage_bucket_iam_member.sa_viewer_permission (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/google"].p2
2024-04-17T14:04:51.028+0200 [DEBUG] ProviderTransformer: "google_storage_bucket_iam_member.sa_viewer_permission" (*terraform.NodeApplyableResourceInstance) needs provider["registry.terraform.io/hashicorp/google"].p2
2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "google_service_account.example_sa (expand)" references: []
2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "google_storage_bucket_iam_member.sa_viewer_permission (expand)" references: [google_service_account.example_sa (expand)]
2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "google_storage_bucket_iam_member.sa_viewer_permission" references: [google_service_account.example_sa (expand)]
2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "provider["registry.terraform.io/hashicorp/google"]" references: []
2024-04-17T14:04:51.028+0200 [DEBUG] ReferenceTransformer: "provider["registry.terraform.io/hashicorp/google"].p2" references: []
2024-04-17T14:04:51.028+0200 [DEBUG] Starting graph walk: walkApply
2024-04-17T14:04:51.029+0200 [DEBUG] created provider logger: level=debug
2024-04-17T14:04:51.029+0200 [INFO] provider: configuring client automatic mTLS
2024-04-17T14:04:51.031+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5"]
2024-04-17T14:04:51.036+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11096
2024-04-17T14:04:51.036+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5
2024-04-17T14:04:51.072+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: configuring server automatic mTLS: timestamp="2024-04-17T14:04:51.072+0200"
2024-04-17T14:04:51.077+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: plugin address: address=/tmp/plugin402184316 network=unix timestamp="2024-04-17T14:04:51.077+0200"
2024-04-17T14:04:51.077+0200 [DEBUG] provider: using plugin: version=5
2024-04-17T14:04:51.082+0200 [DEBUG] created provider logger: level=debug
2024-04-17T14:04:51.082+0200 [INFO] provider: configuring client automatic mTLS
2024-04-17T14:04:51.085+0200 [DEBUG] provider: starting plugin: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 args=[".terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5"]
2024-04-17T14:04:51.089+0200 [DEBUG] provider: plugin started: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11109
2024-04-17T14:04:51.089+0200 [DEBUG] provider: waiting for RPC address: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5
2024-04-17T14:04:51.124+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: configuring server automatic mTLS: timestamp="2024-04-17T14:04:51.124+0200"
2024-04-17T14:04:51.129+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: plugin address: address=/tmp/plugin3613715731 network=unix timestamp="2024-04-17T14:04:51.129+0200"
2024-04-17T14:04:51.129+0200 [DEBUG] provider: using plugin: version=5
2024-04-17T14:04:51.136+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Authenticating using DefaultClient...: tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_rpc=ConfigureProvider @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1777 @module=google tf_req_id=f33b28ba-688d-fce3-e6a8-e8c7a8b272b7 timestamp="2024-04-17T14:04:51.136+0200"
2024-04-17T14:04:51.137+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=f33b28ba-688d-fce3-e6a8-e8c7a8b272b7 @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1778 tf_rpc=ConfigureProvider timestamp="2024-04-17T14:04:51.136+0200"
2024-04-17T14:04:51.137+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success]
2024-04-17T14:04:51.159+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Authenticating using DefaultClient...: @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1777 tf_rpc=ConfigureProvider @module=google tf_mux_provider="*proto5server.Server" tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=fc8319ed-5ec3-acdb-8d07-4dc9cd7a0a4b timestamp="2024-04-17T14:04:51.159+0200"
2024-04-17T14:04:51.159+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]: tf_mux_provider="*proto5server.Server" @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1778 tf_req_id=fc8319ed-5ec3-acdb-8d07-4dc9cd7a0a4b tf_rpc=ConfigureProvider @module=google tf_provider_addr=registry.terraform.io/hashicorp/google timestamp="2024-04-17T14:04:51.159+0200"
2024-04-17T14:04:51.159+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success]
2024-04-17T14:04:51.431+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com: tf_mux_provider="*proto5server.Server" tf_rpc=ConfigureProvider @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1667 tf_req_id=fc8319ed-5ec3-acdb-8d07-4dc9cd7a0a4b @module=google tf_provider_addr=registry.terraform.io/hashicorp/google timestamp="2024-04-17T14:04:51.431+0200"
2024-04-17T14:04:51.433+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Authenticating using DefaultClient...
2024-04-17T14:04:51.433+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]
2024-04-17T14:04:51.434+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success]
2024-04-17T14:04:51.502+0200 [INFO] provider.terraform-provider-google_v5.25.0_x5: Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_A].iam.gserviceaccount.com: @module=google tf_provider_addr=registry.terraform.io/hashicorp/google @caller=github.com/hashicorp/terraform-provider-google/google/fwtransport/framework_config.go:1667 tf_mux_provider="*proto5server.Server" tf_req_id=f33b28ba-688d-fce3-e6a8-e8c7a8b272b7 tf_rpc=ConfigureProvider timestamp="2024-04-17T14:04:51.502+0200"
2024-04-17T14:04:51.504+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Authenticating using DefaultClient...
2024-04-17T14:04:51.504+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]
2024-04-17T14:04:51.505+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success]
2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com
2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches
2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "Service Usage"
2024-04-17T14:04:51.583+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches
2024-04-17T14:04:51.584+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "IAM"
2024-04-17T14:04:51.654+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [INFO] Terraform is configured with service account impersonation, original identity: marcin.kielar.ext@[MASKED], impersonated identity: infrastructure-deployer@[MASKED_PROJECT_A].iam.gserviceaccount.com
2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches
2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "Service Usage"
2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] parent context canceled, cleaning up batcher batches
2024-04-17T14:04:51.655+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Stopping batcher "IAM"
2024-04-17T14:04:51.657+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] [transport] [server-transport 0xc001000b60] Closing: Server.Stop called
2024-04-17T14:04:51.657+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] [transport] [server-transport 0xc001000b60] loopyWriter exiting with error: transport closed by client
2024-04-17T14:04:51.657+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-04-17T14:04:51.659+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11096
2024-04-17T14:04:51.659+0200 [DEBUG] provider: plugin exited
google_storage_bucket_iam_member.sa_viewer_permission: Creating...
2024-04-17T14:04:51.661+0200 [INFO] Starting apply for google_storage_bucket_iam_member.sa_viewer_permission
2024-04-17T14:04:51.661+0200 [DEBUG] google_storage_bucket_iam_member.sa_viewer_permission: applying the planned Create change
2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] matching ID [MASKED_PROJECT_B]-data-source-[MASKED] to regex (?P[^/]+).
2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Locking "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]"
2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Locked "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]"
2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG]: Retrieving policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]"
2024-04-17T14:04:51.662+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Waiting for state to become: [success]
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Retry Transport: starting RoundTrip retry loop
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Retry Transport: request attempt 0
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:51 [DEBUG] Google API Request Details:
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ---[ REQUEST ]---------------------------------------
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: GET /storage/v1/b/[MASKED_PROJECT_B]-data-source-[MASKED]/iam?alt=json&optionsRequestedPolicyVersion=3 HTTP/1.1
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Host: storage.googleapis.com
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: User-Agent: Terraform/1.7.0 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/5.25.0
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Goog-User-Project: [MASKED_PROJECT_B]
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Accept-Encoding: gzip
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-04-17T14:04:51.663+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: -----------------------------------------------------
2024-04-17T14:04:52.344+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Google API Response Details:
2024-04-17T14:04:52.344+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ---[ RESPONSE ]--------------------------------------
2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: HTTP/2.0 403 Forbidden
2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Length: 626
2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-04-17T14:04:52.345+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 17 Apr 2024 12:04:52 GMT
2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Pragma: no-cache
2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: UploadServer
2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-04-17T14:04:52.346+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Guploader-Uploadid: ABPtcPq7mokd--I0jt3pGsme3vdcsCDuC2C1HDbwdsJJGBlxpKsdQUk-t-yTOb-lzI4l05T5Dgddn5HHuQ
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "error": {
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "code": 403,
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).",
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "errors": [
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).",
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "domain": "global",
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "reason": "forbidden"
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ]
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-04-17T14:04:52.347+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: -----------------------------------------------------
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Retry Transport: Stopping retries, last request failed with non-retryable error: googleapi: got HTTP response code 403 with body: HTTP/2.0 403 Forbidden
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Length: 626
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Cache-Control: no-cache, no-store, max-age=0, must-revalidate
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Content-Type: application/json; charset=UTF-8
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Date: Wed, 17 Apr 2024 12:04:52 GMT
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Expires: Mon, 01 Jan 1990 00:00:00 GMT
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Pragma: no-cache
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Server: UploadServer
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: Origin
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: Vary: X-Origin
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: X-Guploader-Uploadid: ABPtcPq7mokd--I0jt3pGsme3vdcsCDuC2C1HDbwdsJJGBlxpKsdQUk-t-yTOb-lzI4l05T5Dgddn5HHuQ
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "error": {
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "code": 403,
2024-04-17T14:04:52.348+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).",
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "errors": [
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: {
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "message": "infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist).",
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "domain": "global",
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: "reason": "forbidden"
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: ]
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: }
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Retry Transport: Returning after 1 attempts
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Unlocking "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]"
2024-04-17T14:04:52.349+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] Unlocked "iam-storage-bucket-b/[MASKED_PROJECT_B]-data-source-[MASKED]"
2024-04-17T14:04:52.349+0200 [ERROR] provider.terraform-provider-google_v5.25.0_x5: Response contains error diagnostic: @module=sdk.proto tf_req_id=b0c0c6be-5ba1-b287-368e-9557d3135e3d diagnostic_summary="Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden" tf_proto_version=5.4 tf_rpc=ApplyResourceChange tf_provider_addr=registry.terraform.io/hashicorp/google tf_resource_type=google_storage_bucket_iam_member @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" diagnostic_severity=ERROR timestamp="2024-04-17T14:04:52.344+0200"
2024-04-17T14:04:52.356+0200 [DEBUG] State storage *statemgr.Filesystem declined to persist a state snapshot
2024-04-17T14:04:52.356+0200 [ERROR] vertex "google_storage_bucket_iam_member.sa_viewer_permission" error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden
╷
│ Error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden
│
│ with google_storage_bucket_iam_member.sa_viewer_permission,
│ on main.tf line 34, in resource "google_storage_bucket_iam_member" "sa_viewer_permission":
│ 34: resource "google_storage_bucket_iam_member" "sa_viewer_permission" {
│
╵
2024-04-17T14:04:52.360+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] [transport] [server-transport 0xc000e1da00] Closing: Server.Stop called
2024-04-17T14:04:52.360+0200 [DEBUG] provider.terraform-provider-google_v5.25.0_x5: 2024/04/17 14:04:52 [DEBUG] [transport] [server-transport 0xc000e1da00] loopyWriter exiting with error: transport closed by client
2024-04-17T14:04:52.360+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-04-17T14:04:52.364+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/google/5.25.0/linux_amd64/terraform-provider-google_v5.25.0_x5 pid=11109
2024-04-17T14:04:52.364+0200 [DEBUG] provider: plugin exited

Expected Behavior

I would expect the terraform apply to be able to create both, the Service Account in Project A, and it's binding in IAM Policy of the Bucket in Project B.

Actual Behavior

Terraform creates the SA in Project A, but fails to provision the binding in Project 2. The error message is:

│ Error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden
│
│ with google_storage_bucket_iam_member.sa_viewer_permission,
│ on main.tf line 34, in resource "google_storage_bucket_iam_member" "sa_viewer_permission":
│ 34: resource "google_storage_bucket_iam_member" "sa_viewer_permission" {

I believe the issue is a result of terraform provider implementation ignoring the billing_project setting on the provider whenever the GOOGLE_CLOUD_QUOTA_PROJECT environment variable is set.

Steps to reproduce

First, a bit of explanation: I'm working on multiple GCP Projects (and credentials) at the same time, and it's easier for me to configure gcloud command line tool, as well as all tools using GCloud SDK (like terraform), using Environment Variables, rather than through gcloud auth login and gcloud auth application-default login. This way I can have multiple terminal sessions opened an operate with different credentials / gcp projects in each terminal.

Now to the steps:

  1. Following environment variables are set when running terraform apply 
    GOOGLE_CLOUD_PROJECT:                        [MASKED-PROJECT-A]
GOOGLE_APPLICATION_CREDENTIALS:              /home/mkielar/.config/gcloud/configurations/config_[MASKED-PROJECT-A].adc.json
GOOGLE_CLOUD_CPP_USER_PROJECT:               [MASKED-PROJECT-A]
GOOGLE_CLOUD_QUOTA_PROJECT:                  [MASKED-PROJECT-A]
  2. terraform apply (fails)

This causes the error as logged in the "Debug output":

│ Error: Error retrieving IAM policy for storage bucket "b/[MASKED_PROJECT_B]-data-source-[MASKED]": googleapi: Error 403: infrastructure-deployer@[MASKED_PROJECT_B].iam.gserviceaccount.com does not have serviceusage.services.use access to the Google Cloud project. Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden
│
│ with google_storage_bucket_iam_member.sa_viewer_permission,
│ on main.tf line 34, in resource "google_storage_bucket_iam_member" "sa_viewer_permission":
│ 34: resource "google_storage_bucket_iam_member" "sa_viewer_permission" {

However, when I unset the GOOGLE_CLOUD_QUOTA_PROJECT Environment Variable, everything works:

  1. Updated environment: 
    GOOGLE_CLOUD_PROJECT:                        [MASKED-PROJECT-A]
GOOGLE_APPLICATION_CREDENTIALS:              /home/mkielar/.config/gcloud/configurations/config_[MASKED-PROJECT-A].adc.json
GOOGLE_CLOUD_CPP_USER_PROJECT:               [MASKED-PROJECT-A]
  2. terraform apply (succeeds)

Important Factoids

  1. See "Actual behavior" and "Steps to reproduce".
  2. According to https://github.com/googleapis/google-cloud-php/blob/main/AUTHENTICATION.md#project-and-credential-lookup, the order in which GCloud PHP SDK (I know, Terraform uses Go, but they should be consistent), is that the priority is given to values defined in code, and only when they're missing in code, Environment is being considered. What terraform-provider-google is doing when GOOGLE_CLOUD_QUOTA_PROJECT suggests a reversed order - first, the Environment is considered, and if it's set, the value of billing_project in provider configuration is ignored.

References

No response
@mkielar mkielar added the bug label Apr 17, 2024
@github-actions github-actions bot added forward/review In review; remove label to forward service/storage labels Apr 17, 2024
@mkielar mkielar changed the title is:issue is:open "billing_project" in provider configuration ignored if GOOGLE_CLOUD_QUOTA_PROJECT environment variable is set "billing_project" in provider configuration ignored if GOOGLE_CLOUD_QUOTA_PROJECT environment variable is set Apr 17, 2024
@mkielar
Copy link
Author

mkielar commented Apr 17, 2024

Few more words about the "real-world" scenario:

In this case, I work predominantly on Project A, which is why I have it set up in Environment Variables. This setup allowed me to use gcloud cli, Python SDK and Terraform, while using dedicated Service Accounts for Terraform on each GCP Project (least-priviledge principle) while running it both locally and on our CICD Platform, without any changes in code.

The Terraform template in this case provisions around 60 other resources, and it does so predominantly in PROJECT A. The resources in PROJECT B (and C, and D, and so on) are mostly - as shown - IAM Policy Bindings, to allow Compute Resources in Project A, to access Storage Resources (GCS Buckets, BigQuery Tables) in Projects B, C, D...

Having things configured via Environment Variables allows me to use tools other than terraform (and these just operate on Project A, because that's like 99% of work), and get them billed correctly to PROJECT A billing.

@ggtisc
Copy link
Collaborator

ggtisc commented Apr 19, 2024

Hi @mkielar!

Please confirm if both projects have the next same attributes with yes or no:

  • organization (1234567890): Y/N
  • domain (example-abc.net): Y/N
  • billing account (1234AB-5678CD-9012EF): Y/N
  • region (us-central1): Y/N

@ggtisc ggtisc self-assigned this Apr 19, 2024
@mkielar
Copy link
Author

mkielar commented Apr 23, 2024

Hi, @ggtisc , sorry for late reply. The thing is, I don't have access to entire GCP Organization. In fact, I can only see the two projects, and so far I used to set billing project to be the same as the project I'm operating on (so no shared billing project that I'd be aware of). all of that worked when I used gcloud CLI or when I used Terraform with just a single GCP Provider instance.

So, here are the answers, to my best knowledge:

  • organization: Yes, both projects are part of the same GCP Organization.
  • domain: Not sure.
    I don't understand what "domain" is in context GCP Projects.
  • billing account: No.
    My understanding is that each of the project bills itself. That's why I'm trying to set billing projects separately for each Terraform Provider instance.
  • region: Yes. (europe-west3)

@rileykarson rileykarson added service/terraform and removed service/storage forward/review In review; remove label to forward labels Apr 23, 2024
@SarahFrench
Copy link
Member

👋 Hi @mkielar , I noticed this issue by chance and my spidey senses are making me wonder if this is a result of some changes to the provider configuration code that we introduced in later v4.x.x versions. Your issue description mentions you're using a v5.x.x version, but could you please try reproducing the scenario using:

  • v4.52.0 - the version of the provider before those changes were added
  • v4.60.0 - a stable version of the provider following those changes being added

If the problem doesn't occur with v4.52.0 and does occur with v4.60.0 then that'll confirm the code changes I mentions are the origin of this problem. Also, your config from this issue will be a nice starting point for an acceptance test we can add to avoid a regression in future. Thanks!

@mkielar
Copy link
Author

mkielar commented May 20, 2024 •
edited
Loading

Hi @SarahFrench, I finally had some time over the weekend to test this, and here are the results:

  • v4.52.0 - works, both resources are created.
  • v4.60.0 - fails the same as v5.25.0 (and also v5.29.1), i.e. with the "Permission 'serviceusage.services.use' denied on resource (or it may not exist)., forbidden" error message.

All of the failing ones work when I unset the GOOGLE_CLOUD_QUOTA_PROJECT environment variable.

Hope that helps.

@SarahFrench
Copy link
Member

Thank you for checking, given what you found I think it is related to the changes I mentioned.

@SarahFrench SarahFrench assigned SarahFrench and unassigned ggtisc Aug 21, 2024
@SarahFrench SarahFrench mentioned this issue Aug 21, 2024
Closed
@SarahFrench
Copy link
Member

I've been looking into this a bit more and some stuff I found:

  • GOOGLE_CLOUD_QUOTA_PROJECT isn't explicitly in our codebase, so it's not a simple case of the ENV being mishandled in our code
  • GOOGLE_CLOUD_QUOTA_PROJECT is used in google-api-go-client, which is used in the Google provider and is probably the route 'in' for the ENV.

I'll have a go using the reproduction info in the description to reproduce the problem and test the different versions mentioned here, as there isn't a clear reason why the resources being provisioned in the reproduction above should be impacted by the muxing (as the reproduction uses no plugin-framework implemented resources).

It'll be useful to create an acceptance test defining the expected behaviour, but I think a fix for the changed behaviour might come as a side effect of GoogleCloudPlatform/magic-modules#11577

@SarahFrench
Copy link
Member

👋 Hi, I'm back 😁

I've created some Terraform config to reproduce the scenario above and I saw, depending on the provider version I used, a terraform apply could be disrupted if GOOGLE_CLOUD_QUOTA_PROJECT=foobar was set but otherwise would be successful.

I found that the problem first appears in v4.59.0, and in v4.59.0 we bumped google.golang.org/api from v0.111.0 to v0.114.0. That updated version includes this PR that adds the env GOOGLE_CLOUD_QUOTA_PROJECT to that client library.

I'll raise this internally but currently I can't think of a way we can make a provider-side change to fix the impact of that change in the client library.

@mkielar
Copy link
Author

mkielar commented Sep 30, 2024

Welcome back! :)

The PR you mention points to googleapis/google-api-go-client#1891, which, at least in its desired scope, says:

Precedence should be:

Quota Project Client options
Environment Variable
ADC Credential

And I can see that the GetQuotaProject function is designed this way (and used too, if you grep through references to it - but I may be missing something, I'm terrible with Go).

Anyway, assuming they made no mistakes with the way they handle GOOGLE_CLOUD_QUOTA_PROJECT in the Go Client, you should be able to configure the client using explicit options and that should override whatever is set in the env variable, isn't it?

@SarahFrench
Copy link
Member

That's a good callout, and hopefully there is a solution we can find! I haven't dived into the problem fully but there are a few places where we impact the X-Goog-User-Project header's value so a solution needs further investigation (and some more eyes on it, as this would impact most or all HTTP traffic from the provider).

Making a note for others on the team for the future:

SarahFrench added a commit to SarahFrench/magic-modules that referenced this issue Oct 1, 2024
@SarahFrench
Add a test case showing the impact of GOOGLE_CLOUD_QUOTA_PROJECT
c520d71 
Related to hashicorp/terraform-provider-google#17882
@ScottSuarez ScottSuarez linked a pull request Nov 22, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug service/terraform
Projects
None yet
Milestone
No milestone
4 participants
@rileykarson @mkielar @SarahFrench @ggtisc