Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skipped failing test and added race condition check to CI #1217

Closed
wants to merge 3 commits into from
+26 −2
Closed

Skipped failing test and added race condition check to CI #1217

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
300b919
[ignore] Added race condition check to tests
samiib May 15, 2024
0190fa0
[ignore] Split TestAccAciRestManaged_import into seperate tests
samiib May 16, 2024
f118801
[ignore] Skip troublesome aci_rest_managed test in CI until it gets f…
samiib May 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions .github/workflows/checks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,9 @@ jobs:
terraform_version: '1.7.*'
terraform_wrapper: false
- name: Terraform Acceptance Test (APIC ${{ matrix.apic_host.name }})
run: go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -timeout 300m -coverprofile=coverage.out -covermode=atomic
# TODO: Fix TestAccAciRestManaged_importWithIpv6 to resolve data race failures occurring in the CI
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be an interesting fix.

Authenticate() has c.skipLoggingPayload whose value is being changed and reset in the end.
Now Authenticate() is called from multiple go-routines, one being MakeRestRequestRaw() via DoRestRequestEscapeHtml and the other go-routine handling MakeRestRequestRaw() within the Authenticate() function itself. This might intermittently be creating a race condition due to c.skipLoggingPayload state being read by one and written by the other at the same time.
I think this is what's causing it.

It would be great if this PR is updated with the fix.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @shrsr, yeah this sounds like what could be happening based on the stack trace error of the data race warnings. Its odd that its only happening consistently in this one test. It appears to be quite a complex fix that will effect every other module, hence I created another issue for it. #1218
For now this PR will turn on the race condition check in the CI and skip this one test to stop PR's failing randomly.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's only happening for this test because of the way the config. is specified in the test. As seen below, two BDs are being created in parallel. This is only showing up during the CI because more than one cpu is utilized by the go test environment in the CI workflow enabling parallelism. The utilization of more than one CPU also causes parallel refreshes in TF acceptance tests further adding to race conditions.

resource "aci_rest_managed" "tenant_%[2]s" {
		dn = "uni/tn-%[1]s"
		class_name = "fvTenant"
		content = {
			name = "%[1]s"
		}
		child {
			rn         = "ctx-VRF1"
			class_name = "fvCtx"
			content = {
				name = "VRF1"
			}
		}
		child {
			rn         = "ctx-VRF2"
			class_name = "fvCtx"
			content = {
				name = "VRF2"
			}
		}
	}

	resource "aci_rest_managed" "bd_%[2]s" {
		dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s"
		class_name = "fvBD"
		content = {
			name  = "%[1]s"
		}
		child {
			rn         = "rsctx"
			class_name = "fvRsCtx"
			content = {
				tnFvCtxName = "VRF1"
			}
		}
	}

	resource "aci_rest_managed" "subnet_%[2]s" {
		dn         = "${aci_rest_managed.bd_%[2]s.id}/subnet-[%[3]s]"
		class_name = "fvSubnet"
		content = {
		  ip           = "%[3]s"
		  scope        = "private"
		  ipDPLearning = "enabled"
		  ctrl         = "nd"
		}
	}

	resource "aci_rest_managed" "bd_%[2]s_2" {
		dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s_2"
		class_name = "fvBD"
		content = {
			name  = "%[1]s_2"
		}
		child {
			rn         = "rsctx"
			class_name = "fvRsCtx"
			content = {
				tnFvCtxName = "VRF2"
			}
		}
		child {
			rn         = "subnet-[%[4]s]"
			class_name = "fvSubnet"
			content = {
				ip = "%[4]s"
				scope        = "private"
				ipDPLearning = "enabled"
				ctrl         = "nd"
			}
		}
	}

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of skipping the test, we could enforce the CI to use 1 cpu for now while the issue is being fixed.
What do you think?

go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -race -cpu 1 -timeout 300m -coverprofile=coverage.out -covermode=atomic

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it! I'll try it out now. 😸

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shrsr Unfortunately with -cpu 1 the data race is still detected. I'll revert that commit and skip the test for now. 😞

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah looks like there are other factors that come into play in containers and simply setting the core to 1 is not enough to mask the race.

# https://github.com/CiscoDevNet/terraform-provider-aci/issues/1218
run: go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -race -timeout 300m -coverprofile=coverage.out -covermode=atomic -skip TestAccAciRestManaged_importWithIpv6
env:
TF_ACC: '1'
TF_ACC_STATE_LINEAGE: '1'
Expand All @@ -87,4 +89,4 @@ jobs:
if: ${{ matrix.apic_host.name == 'v6.0'}}
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
token: ${{ secrets.CODECOV_TOKEN }}
22 changes: 22 additions & 0 deletions internal/provider/resource_aci_rest_managed_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,6 +311,17 @@ func TestAccAciRestManaged_import(t *testing.T) {
resource.TestCheckResourceAttr("aci_rest_managed.import", "child.#", "2"),
),
},
},
})
}

func TestAccAciRestManaged_importWithIpv6(t *testing.T) {
name := acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)

resource.Test(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
Steps: []resource.TestStep{
{
Config: testAccAciRestManagedConfig_importWithIpv6(name, "import", "2001:1:2::5/28", "2001:1:2::5/28"),
Check: resource.ComposeTestCheckFunc(
Expand Down Expand Up @@ -388,6 +399,17 @@ func TestAccAciRestManaged_import(t *testing.T) {
resource.TestCheckResourceAttr("aci_rest_managed.bd_import_2", "child.1.content.tnFvCtxName", "VRF2"),
),
},
},
})
}

func TestAccAciRestManaged_importWithBracket(t *testing.T) {
name := acctest.RandStringFromCharSet(10, acctest.CharSetAlphaNum)

resource.Test(t, resource.TestCase{
PreCheck: func() { testAccPreCheck(t) },
ProtoV6ProviderFactories: testAccProtoV6ProviderFactories,
Steps: []resource.TestStep{
{
Config: testAccAciRestManagedConfig_importWithBracket(name),
Check: resource.ComposeTestCheckFunc(
Expand Down