add verify prod and refactor terraform check/deploy

[cpanato]

Fixes #54

Need #196

[cpanato]

there is a couple of permission denied for the pull request service account

Error: Error when reading or editing Resource "pubsub topic \"projects/octo-sts/topics/octo-sts-broker-us-central1\"" with IAM Binding (Role "roles/pubsub.publisher"): Error retrieving IAM policy for pubsub topic "projects/octo-sts/topics/octo-sts-broker-us-central1": googleapi: Error 403: User not authorized to perform this action.


Error: Error when reading or editing Resource "storage bucket \"b/octo-sts-recorder-us-central1-92d6\"" with IAM Binding (Role "roles/storage.admin"): Error retrieving IAM policy for storage bucket "b/octo-sts-recorder-us-central1-92d6": googleapi: Error 403: [email protected] does not have storage.buckets.getIamPolicy access to the Google Cloud Storage bucket. Permission 'storage.buckets.getIamPolicy' denied on resource (or it may not exist)., forbidden

Error: [id=gcr.io/octo-sts/cmd/otel-collector@sha256:687f1295df7d63c6992bd779c649a3ec4ac59afa847cf3d71c0170103683cf30] read doBuild: publish: writing sbom: PATCH https://gcr.io/v2/octo-sts/cmd/otel-collector/blobs/uploads/ALXj4xrU_2UMDmfx7hyWY71i37vaWqK7RyK-xY0BSklGLKmhCf5WqhG5gTBQsjUJi3hsTp6i8ETLOoSHP2kUDNM: DENIED: Access denied.

https://github.com/octo-sts/app/actions/runs/8469299780/job/23204323113?pr=197

[mattmoor]

I think that what we want is probably just to exclude readonly operations that happen as part of terraform plan from auditing because cases like the recorder are going to be annoying to plumb new identities into everywhere, and this is a common chainguardian tripping hazard as well in mono (and this will bite mono as we segment CI identities as well).

[octo-sts]

Terraform checks for "./iac"

Terraform Format and Style 🖌 success

Terraform Initialization ⚙️ success

Terraform Validation 🤖 success

Validation Output

Success! The configuration is valid.

Terraform Plan 📖 success

Show Plan

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # module.sts-service.module.otel-collector.ko_build.otel-image has been deleted
  - resource "ko_build" "otel-image" {
        id          = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76"
      - image_ref   = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76" -> null
        # (4 unchanged attributes hidden)
    }

  # module.cloudevent-recorder.module.this[0].module.otel-collector.ko_build.otel-image has been deleted
  - resource "ko_build" "otel-image" {
        id          = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76"
      - image_ref   = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76" -> null
        # (4 unchanged attributes hidden)
    }

  # module.negative_prober.module.this.module.otel-collector.ko_build.otel-image has been deleted
  - resource "ko_build" "otel-image" {
        id          = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76"
      - image_ref   = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76" -> null
        # (4 unchanged attributes hidden)
    }

  # module.prober.module.this.module.otel-collector.ko_build.otel-image has been deleted
  - resource "ko_build" "otel-image" {
        id          = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76"
      - image_ref   = "gcr.io/octo-sts/cmd/otel-collector@sha256:88ab4ae035258bbc722886400f04712b25c1ca82f3cd0136aacba6b5059ded76" -> null
        # (4 unchanged attributes hidden)
    }


Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.

─────────────────────────────────────────────────────────────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
-/+ destroy and then create replacement
+/- create replacement and then destroy
 <= read (data resources)

Terraform will perform the following actions:

  # google_monitoring_alert_policy.anomalous-kms-access will be updated in-place
  ~ resource "google_monitoring_alert_policy" "anomalous-kms-access" {
        id                    = "projects/octo-sts/alertPolicies/3908022799698009840"
        name                  = "projects/octo-sts/alertPolicies/3908022799698009840"
        # (7 unchanged attributes hidden)

      ~ conditions {
            name         = "projects/octo-sts/alertPolicies/3908022799698009840/conditions/3908022799698007353"
            # (1 unchanged attribute hidden)

          ~ condition_matched_log {
              ~ filter           = <<-EOT
                    -- KMS operations
                          protoPayload.serviceName="cloudkms.googleapis.com"
                    
                          -- Against our Github App's keyring
                          protoPayload.resourceName: "projects/octo-sts/locations/global/keyRings/octo-sts/"
                    
                          -- Skip operations that are a part of terraform plan
                          -protoPayload.methodName=("GetCryptoKey")
                    
                          -- The application itself should only perform signing operations.
                          -(
                            protoPayload.authenticationInfo.principalEmail="[email protected]" AND
                            protoPayload.methodName=("AsymmetricSign")
                          )
                    
                          -- Github IaC should only reconcile the keyring and keys.
                          -(
                  -         protoPayload.authenticationInfo.principalEmail="[email protected]" AND
                  +         protoPayload.authenticationInfo.principalEmail="[email protected]" AND
                            protoPayload.methodName=("CreateKeyRing" OR "CreateCryptoKey" OR "SetIamPolicy")
                          )
                    
                          -- If we were to filter out import events they would look like
                          -- this, but instead I am opting to explicitly have these alert,
                          -- to raise awareness of the rotation, since it means that a human
                          -- has interacted with an App key locally.
                          -- -(
                          --   protoPayload.authenticationInfo.principalEmail="[email protected]" AND
                          --   protoPayload.methodName=("CreateImportJob" OR "ImportCryptoKeyVersion")
                          -- )
                EOT
                # (1 unchanged attribute hidden)
            }
        }

        # (1 unchanged block hidden)
    }

  # module.cloudevent-broker.google_monitoring_dashboard.dashboard will be updated in-place
  ~ resource "google_monitoring_dashboard" "dashboard" {
      ~ dashboard_json = jsonencode(
          ~ {
              - etag             = "09069d0236af4fc68406f03dfac2832c"
              ~ mosaicLayout     = {
                  ~ tiles   = [
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                          + yPos   = 0
                            # (2 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                          + yPos   = 0
                            # (2 unchanged attributes hidden)
                        },
                      ~ {
                          + yPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + yPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                        {
                            height = 4
                            widget = {
                                title   = "Push latency"
                                xyChart = {
                                    chartOptions      = {
                                        mode = "COLOR"
                                    }
                                    dataSets          = [
                                        {
                                            minAlignmentPeriod = "60s"
                                            plotType           = "LINE"
                                            targetAxis         = "Y1"
                                            timeSeriesQuery    = {
                                                timeSeriesFilter = {
                                                    aggregation = {
                                                        alignmentPeriod    = "60s"
                                                        crossSeriesReducer = "REDUCE_PERCENTILE_99"
                                                        groupByFields      = [
                                                            "resource.label.\"subscription_id\"",
                                                        ]
                                                        perSeriesAligner   = "ALIGN_DELTA"
                                                    }
                                                    filter      = <<-EOT
                                                        resource.type="pubsub_subscription"
                                                        metric.type="pubsub.googleapis.com/subscription/push_request_latencies"
                                                        metadata.system_labels."topic_id"=monitoring.regex.full_match("octo-sts-broker-.*")
                                                    EOT
                                                }
                                            }
                                        },
                                    ]
                                    timeshiftDuration = "0s"
                                    yAxis             = {
                                        label = "y1Axis"
                                        scale = "LINEAR"
                                    }
                                }
                            }
                            width  = 4
                            xPos   = 4
                            yPos   = 4
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets     = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilterRatio = {
                                                  ~ denominator = {
                                                      ~ aggregation = {
                                                          + groupByFields      = []
                                                            # (3 unchanged attributes hidden)
                                                        }
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                  ~ numerator   = {
                                                      ~ aggregation = {
                                                          + groupByFields      = []
                                                            # (3 unchanged attributes hidden)
                                                        }
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                }
                                            }
                                            # (4 unchanged attributes hidden)
                                        },
                                    ]
                                  + thresholds   = []
                                    # (2 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                    # (2 unchanged attributes hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                    ]
                    # (1 unchanged attribute hidden)
                }
              - name             = "projects/96355665038/dashboards/4fcfc437-e961-41bb-9636-cbae85d9d303"
                # (3 unchanged attributes hidden)
            }
        )
        id             = "projects/96355665038/dashboards/4fcfc437-e961-41bb-9636-cbae85d9d303"
        # (1 unchanged attribute hidden)
    }

  # module.cloudevent-recorder.google_monitoring_alert_policy.bucket-access will be updated in-place
  ~ resource "google_monitoring_alert_policy" "bucket-access" {
        id                    = "projects/octo-sts/alertPolicies/2273930906264969635"
        name                  = "projects/octo-sts/alertPolicies/2273930906264969635"
        # (7 unchanged attributes hidden)

      ~ conditions {
            name         = "projects/octo-sts/alertPolicies/2273930906264969635/conditions/2273930906264972132"
            # (1 unchanged attribute hidden)

          ~ condition_matched_log {
              ~ filter           = <<-EOT
                    logName="projects/octo-sts/logs/cloudaudit.googleapis.com%2Fdata_access"
                          protoPayload.serviceName="storage.googleapis.com"
                          protoPayload.resourceName=~"projects/_/buckets/octo-sts-recorder-(us-central1)-92d6"
                    
                          -- Exclude things that happen during terraform plan.
                          -protoPayload.methodName=("storage.buckets.get")
                    
                          -- The recorder service write objects into the bucket.
                          -(
                            protoPayload.authenticationInfo.principalEmail="[email protected]"
                            protoPayload.methodName="storage.objects.create"
                          )
                    
                          -- The importer identity (used by DTS) enumerates and reads objects.
                          -(
                            protoPayload.authenticationInfo.principalEmail="[email protected]"
                            protoPayload.methodName=("storage.objects.get" OR "storage.objects.list")
                          )
                    
                          -- Our CI identity reconciles the bucket.
                          -(
                  -         protoPayload.authenticationInfo.principalEmail="[email protected]"
                  +         protoPayload.authenticationInfo.principalEmail="[email protected]"
                            protoPayload.methodName=("storage.getIamPermissions")
                          )
                    
                          -- Security scanners frequently probe for public buckets via listing buckets
                          -- and then getting permissions, so we ignore these even though they pierce
                          -- the abstraction.
                          -protoPayload.methodName="storage.getIamPermissions"
                EOT
                # (1 unchanged attribute hidden)
            }
        }

        # (1 unchanged block hidden)
    }

  # module.cloudevent-recorder.google_service_account_iam_binding.provisioner-acts-as-import-identity will be updated in-place
  ~ resource "google_service_account_iam_binding" "provisioner-acts-as-import-identity" {
        id                 = "projects/octo-sts/serviceAccounts/[email protected]/roles/iam.serviceAccountUser"
      ~ members            = [
          - "serviceAccount:[email protected]",
          + "serviceAccount:[email protected]",
        ]
        # (3 unchanged attributes hidden)
    }

  # module.dashboard.google_monitoring_dashboard.dashboard will be updated in-place
  ~ resource "google_monitoring_dashboard" "dashboard" {
      ~ dashboard_json = jsonencode(
          ~ {
              - etag             = "48010e563043221994493e3f8ca3c5fd"
              ~ mosaicLayout     = {
                  ~ tiles   = [
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                          + yPos   = 0
                            # (2 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                          + yPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets     = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilterRatio = {
                                                  ~ denominator = {
                                                      ~ aggregation = {
                                                          + groupByFields      = []
                                                            # (3 unchanged attributes hidden)
                                                        }
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                  ~ numerator   = {
                                                      ~ aggregation = {
                                                          + groupByFields      = []
                                                            # (3 unchanged attributes hidden)
                                                        }
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                }
                                            }
                                            # (4 unchanged attributes hidden)
                                        },
                                    ]
                                  + thresholds   = []
                                    # (2 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                        {
                            height = 6
                            widget = {
                                title   = "Incoming request latency"
                                xyChart = {
                                    chartOptions      = {
                                        mode = "COLOR"
                                    }
                                    dataSets          = [
                                        {
                                            minAlignmentPeriod = "60s"
                                            plotType           = "LINE"
                                            targetAxis         = "Y1"
                                            timeSeriesQuery    = {
                                                timeSeriesFilter = {
                                                    aggregation = {
                                                        alignmentPeriod    = "60s"
                                                        crossSeriesReducer = "REDUCE_PERCENTILE_99"
                                                        groupByFields      = [
                                                            "metric.label.\"grpc_service\"",
                                                            "metric.label.\"grpc_method\"",
                                                        ]
                                                        perSeriesAligner   = "ALIGN_DELTA"
                                                    }
                                                    filter      = <<-EOT
                                                        metric.type="prometheus.googleapis.com/grpc_server_handling_seconds/histogram"
                                                        resource.label."job"="octo-sts"
                                                    EOT
                                                }
                                            }
                                        },
                                    ]
                                    timeshiftDuration = "0s"
                                    yAxis             = {
                                        label = "y1Axis"
                                        scale = "LINEAR"
                                    }
                                }
                            }
                            width  = 6
                            xPos   = 6
                            yPos   = 39
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                        {
                            height = 6
                            widget = {
                                title   = "Outbound request latency"
                                xyChart = {
                                    chartOptions      = {
                                        mode = "COLOR"
                                    }
                                    dataSets          = [
                                        {
                                            minAlignmentPeriod = "60s"
                                            plotType           = "LINE"
                                            targetAxis         = "Y1"
                                            timeSeriesQuery    = {
                                                timeSeriesFilter = {
                                                    aggregation = {
                                                        alignmentPeriod    = "60s"
                                                        crossSeriesReducer = "REDUCE_PERCENTILE_99"
                                                        groupByFields      = [
                                                            "metric.label.\"grpc_service\"",
                                                            "metric.label.\"grpc_method\"",
                                                        ]
                                                        perSeriesAligner   = "ALIGN_DELTA"
                                                    }
                                                    filter      = <<-EOT
                                                        metric.type="prometheus.googleapis.com/grpc_client_handling_seconds/histogram"
                                                        resource.label."job"="octo-sts"
                                                    EOT
                                                }
                                            }
                                        },
                                    ]
                                    timeshiftDuration = "0s"
                                    yAxis             = {
                                        label = "y1Axis"
                                        scale = "LINEAR"
                                    }
                                }
                            }
                            width  = 6
                            xPos   = 6
                            yPos   = 45
                        },
                      ~ {
                          ~ widget = {
                              ~ collapsibleGroup = {
                                  + collapsed = false
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          + xPos   = 0
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                          + xPos   = 0
                            # (3 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ aggregation          = {
                                                      + groupByFields      = []
                                                        # (3 unchanged attributes hidden)
                                                    }
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + groupByFields      = []
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (1 unchanged attribute hidden)
                                                    }
                                                    # (1 unchanged attribute hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 unchanged attributes hidden)
                                }
                                # (1 unchanged attribute hidden)
                            }
                            # (4 unchanged attributes hidden)
                        },
                      ~ {
                          ~ widget = {
                              ~ xyChart = {
                                  ~ dataSets          = [
                                      ~ {
                                          ~ timeSeriesQuery    = {
                                              ~ timeSeriesFilter = {
                                                  ~ secondaryAggregation = {
                                                      + crossSeriesReducer = "REDUCE_NONE"
                                                      + perSeriesAligner   = "ALIGN_NONE"
                                                        # (2 unchanged attributes hidden)
                                                    }
                                                    # (2 unchanged attributes hidden)
                                                }
                                            }
                                            # (3 unchanged attributes hidden)
                                        },
                                    ]
                                    # (3 u ...

Output is too long and was truncated. You can read full Plan in Actions.

Pusher: @cpanato, Action: pull_request, Working Directory: ./iac, Workflow: Verify prod Octo-sts

[mattmoor]

This needs: chainguard-dev/terraform-infra-common#241 to avoid alerting.

[cpanato]

@mattmoor ready to go

[mattmoor]

Do you know of anything that we can run over this to check for injection vulnerabilities?

Skimming, the only one that stood out is inputs.working_directory, and that's a reach, but would be good to sanity check things

[mattmoor]

I'm going to merge this, so that we can start using it (probably would have avoided mild outage yesterday). I'd love an answer for the shell check issue.