TeamCity: Enable automated testing of provider-defined functions

[SarahFrench]

Background

Provider-defined functions are being prepared on a feature branch but only work with TF 1.8.0+, which our automated testing systems don't use. We need to enable testing of the feature branch for provider functions in the lead up to the 1.8.0 release, and then later update all testing to use TF 1.8.0+

What kind of contribution is this issue about?

• TeamCity
• Feature branch testing
• Automated testing of PRs

Related PR(s), if any:

• Add project_from_id provider-defined function GoogleCloudPlatform/magic-modules#10021
• Add region_from_id and zone_from_id provider-defined functions GoogleCloudPlatform/magic-modules#10060
• and others in drafts

Details

1) Enable feature branch testing in TeamCity

✅ I'm going to add a new project in TeamCity for testing the branch FEATURE-BRANCH-provider-functions that uses the alpha release of TF 1.8.0. This will allow us to test the new functions without causing all the provider testing to rely on an alpha release.

Completed in : GoogleCloudPlatform/magic-modules#10088

2) Enable testing in TeamCity once the feature branch is merged to main

✅ We'll need to update TeamCity to use Terraform 1.8.0 then.

TeamCity was updated to use 1.8.0-rc1 in the past, then 1.8.0. Recently it was bumped to 1.8.3 : GoogleCloudPlatform/magic-modules#10719

3) Update automation on PRs to use TF 1.8.0+

• Update build-environment image to use Terraform 1.8.3 GoogleCloudPlatform/magic-modules#10293

Until this is done, all acceptance tests for provider-defined functions will need to be skipped in VCR

[SarahFrench]

Part 1 is complete from the list in the issue description.

[SarahFrench]

2. Enable testing in TeamCity once the feature branch is merged to main

TeamCity is currently using TF 1.8.0 but there are some tests showing funny behaviour:

• TestAccBigQueryJob_bigqueryJobCopyExample
• TestAccBigQueryJob_bigqueryJobCopyTableReferenceExample

------- Stdout: -------
=== RUN   TestAccBigQueryJob_bigqueryJobCopyTableReferenceExample
=== PAUSE TestAccBigQueryJob_bigqueryJobCopyTableReferenceExample
=== CONT  TestAccBigQueryJob_bigqueryJobCopyTableReferenceExample
    vcr_utils.go:152: Step 2/2 error running import: exit status 1
        Error: Invalid count argument
          on .../omitted/terraform_plugin_test.tf line 4, in resource "google_bigquery_table" "source":
           4:   count = length(google_bigquery_dataset.source)
        The "count" value depends on resource attributes that cannot be determined
        until apply, so Terraform cannot predict how many instances will be created.
        To work around this, use the -target argument to first apply only the
        resources that the count depends on.
--- FAIL: TestAccBigQueryJob_bigqueryJobCopyTableReferenceExample (34.76s)
FAIL 

• TestAccCloudfunctions2functionIamPolicyGenerated
• TestAccCloudfunctions2functionIamMemberGenerated
• TestAccCloudfunctions2functionIamBindingGenerated

------- Stdout: -------
=== RUN   TestAccCloudfunctions2functionIamBindingGenerated
=== PAUSE TestAccCloudfunctions2functionIamBindingGenerated
=== CONT  TestAccCloudfunctions2functionIamBindingGenerated
    testing_new_import_state.go:125: unexpected output type: <nil>
--- FAIL: TestAccCloudfunctions2functionIamBindingGenerated (90.61s)
FAIL

[melinath]

@SarahFrench I thought we rolled back the 1.8.0 usage but I still see the Cloudfunctions2 failures in the nightlys - is that expected?

[SarahFrench]

We rolled back use of 1.8.0-rc1 and later the failures we noticed weren't present in 1.8.0. There were some failures still but the discussion in the google space suggested they're due to how the tests are configured.

If you'd like to revert to 1.2.5 again we can do that and investigate the lingering errors separately

[melinath]

I am down for us to investigate without rolling back to 1.2.5 if that's what we want to do - it just might make sense to split out into a separate ticket that we can address.

[SarahFrench]

I've created that new issue here: #18109

[SarahFrench]

The PR that closed this issue didn't make VCR tests run using TF 1.8.3 - this PR should do that: GoogleCloudPlatform/magic-modules#10790

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.