Skipped failing test and added race condition check to CI

[samiib]

Skipped the aci_rest_managed importWithIpv6 test from CI to stop random failures in the acceptance test.
Issue #1218 is created to fix this single test case.

• The race check should be enabled in CI to detect any future issues.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Please upload report for BASE (master@0b33a73). Learn more about missing BASE report.

Additional details and impacted files

@@            Coverage Diff            @@
##             master    #1217   +/-   ##
=========================================
  Coverage          ?   86.94%           
=========================================
  Files             ?       43           
  Lines             ?    11258           
  Branches          ?        0           
=========================================
  Hits              ?     9788           
  Misses            ?     1005           
  Partials          ?      465           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[shrsr]

This should be an interesting fix.

Authenticate() has c.skipLoggingPayload whose value is being changed and reset in the end.
Now Authenticate() is called from multiple go-routines, one being MakeRestRequestRaw() via DoRestRequestEscapeHtml and the other go-routine handling MakeRestRequestRaw() within the Authenticate() function itself. This might intermittently be creating a race condition due to c.skipLoggingPayload state being read by one and written by the other at the same time.
I think this is what's causing it.

It would be great if this PR is updated with the fix.

[samiib]

Thanks @shrsr, yeah this sounds like what could be happening based on the stack trace error of the data race warnings. Its odd that its only happening consistently in this one test. It appears to be quite a complex fix that will effect every other module, hence I created another issue for it. #1218
For now this PR will turn on the race condition check in the CI and skip this one test to stop PR's failing randomly.

[shrsr]

It's only happening for this test because of the way the config. is specified in the test. As seen below, two BDs are being created in parallel. This is only showing up during the CI because more than one cpu is utilized by the go test environment in the CI workflow enabling parallelism. The utilization of more than one CPU also causes parallel refreshes in TF acceptance tests further adding to race conditions.

resource "aci_rest_managed" "tenant_%[2]s" {
		dn = "uni/tn-%[1]s"
		class_name = "fvTenant"
		content = {
			name = "%[1]s"
		}
		child {
			rn         = "ctx-VRF1"
			class_name = "fvCtx"
			content = {
				name = "VRF1"
			}
		}
		child {
			rn         = "ctx-VRF2"
			class_name = "fvCtx"
			content = {
				name = "VRF2"
			}
		}
	}

	resource "aci_rest_managed" "bd_%[2]s" {
		dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s"
		class_name = "fvBD"
		content = {
			name  = "%[1]s"
		}
		child {
			rn         = "rsctx"
			class_name = "fvRsCtx"
			content = {
				tnFvCtxName = "VRF1"
			}
		}
	}

	resource "aci_rest_managed" "subnet_%[2]s" {
		dn         = "${aci_rest_managed.bd_%[2]s.id}/subnet-[%[3]s]"
		class_name = "fvSubnet"
		content = {
		  ip           = "%[3]s"
		  scope        = "private"
		  ipDPLearning = "enabled"
		  ctrl         = "nd"
		}
	}

	resource "aci_rest_managed" "bd_%[2]s_2" {
		dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s_2"
		class_name = "fvBD"
		content = {
			name  = "%[1]s_2"
		}
		child {
			rn         = "rsctx"
			class_name = "fvRsCtx"
			content = {
				tnFvCtxName = "VRF2"
			}
		}
		child {
			rn         = "subnet-[%[4]s]"
			class_name = "fvSubnet"
			content = {
				ip = "%[4]s"
				scope        = "private"
				ipDPLearning = "enabled"
				ctrl         = "nd"
			}
		}
	}

[shrsr]

Instead of skipping the test, we could enforce the CI to use 1 cpu for now while the issue is being fixed.
What do you think?

go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -race -cpu 1 -timeout 300m -coverprofile=coverage.out -covermode=atomic

[samiib]

I like it! I'll try it out now. 😸

[samiib]

@shrsr Unfortunately with -cpu 1 the data race is still detected. I'll revert that commit and skip the test for now. 😞

[shrsr]

Yeah looks like there are other factors that come into play in containers and simply setting the core to 1 is not enough to mask the race.

[shrsr]

It's only happening for this test because of the way the config. is specified in the test. As seen below, two BDs are being created in parallel. This is only showing up during the CI because more than one cpu is utilized by the go test environment in the CI workflow enabling parallelism. The utilization of more than one CPU also causes parallel refreshes in TF acceptance tests further adding to race conditions.

resource "aci_rest_managed" "tenant_%[2]s" {
		dn = "uni/tn-%[1]s"
		class_name = "fvTenant"
		content = {
			name = "%[1]s"
		}
		child {
			rn         = "ctx-VRF1"
			class_name = "fvCtx"
			content = {
				name = "VRF1"
			}
		}
		child {
			rn         = "ctx-VRF2"
			class_name = "fvCtx"
			content = {
				name = "VRF2"
			}
		}
	}

	resource "aci_rest_managed" "bd_%[2]s" {
		dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s"
		class_name = "fvBD"
		content = {
			name  = "%[1]s"
		}
		child {
			rn         = "rsctx"
			class_name = "fvRsCtx"
			content = {
				tnFvCtxName = "VRF1"
			}
		}
	}

	resource "aci_rest_managed" "subnet_%[2]s" {
		dn         = "${aci_rest_managed.bd_%[2]s.id}/subnet-[%[3]s]"
		class_name = "fvSubnet"
		content = {
		  ip           = "%[3]s"
		  scope        = "private"
		  ipDPLearning = "enabled"
		  ctrl         = "nd"
		}
	}

	resource "aci_rest_managed" "bd_%[2]s_2" {
		dn = "${aci_rest_managed.tenant_%[2]s.id}/BD-%[1]s_2"
		class_name = "fvBD"
		content = {
			name  = "%[1]s_2"
		}
		child {
			rn         = "rsctx"
			class_name = "fvRsCtx"
			content = {
				tnFvCtxName = "VRF2"
			}
		}
		child {
			rn         = "subnet-[%[4]s]"
			class_name = "fvSubnet"
			content = {
				ip = "%[4]s"
				scope        = "private"
				ipDPLearning = "enabled"
				ctrl         = "nd"
			}
		}
	}

[shrsr]

Instead of skipping the test, we could enforce the CI to use 1 cpu for now while the issue is being fixed.
What do you think?

go test github.com/CiscoDevNet/terraform-provider-aci/v2/internal/provider -v -race -cpu 1 -timeout 300m -coverprofile=coverage.out -covermode=atomic

[shrsr]

LGTM

[anvitha-jain]

LGTM

[gmicol]

LGTM

[samiib]

Long term fix is ready here #1236. Cancelling this PR.