Add antrea-interworking package #4510

wenqiq · 2023-03-22T06:49:05Z

Signed-off-by: Wenqi Qiu [email protected]

What this PR does / why we need it

This pr extends antrea package with interworking, when antreaNsx is enabled in antreaConfig, interworking will also be reconciled by kapp-controller

Which issue(s) this PR fixes

Fixes #

Describe testing done for PR

Release note

Additional information

Special notes for your reviewer

github-actions · 2023-03-22T06:57:50Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230322065748/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

codecov · 2023-03-22T08:03:01Z

Codecov Report

Merging #4510 (1e97f4f) into main (bb07e03) will decrease coverage by 1.32%.
The diff coverage is 23.25%.

@@            Coverage Diff             @@
##             main    #4510      +/-   ##
==========================================
- Coverage   49.77%   48.46%   -1.32%     
==========================================
  Files         453      485      +32     
  Lines       45424    47981    +2557     
==========================================
+ Hits        22612    23252     +640     
- Misses      20652    22518    +1866     
- Partials     2160     2211      +51

Impacted Files	Coverage Δ
...dons/controllers/antrea/antreaconfig_controller.go	`0.00% <0.00%> (ø)`
addons/controllers/antrea/antreaconfig_util.go	`6.28% <27.77%> (ø)`

... and 36 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

github-actions · 2023-03-23T06:33:43Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230323063342/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-03-23T08:49:32Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230323084931/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-03-24T03:48:51Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230324034849/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

addons/controllers/antrea/antreaconfig_util.go

hangyan · 2023-03-27T02:27:30Z

seems the build CI faild.

wjun · 2023-03-27T06:02:00Z

addons/controllers/antrea/antreaconfig_util.go

+			configSpec.AntreaNsx.BootstrapFrom.ProviderRef.ApiVersion = config.Spec.AntreaNsx.BootstrapFrom.ProviderRef.ApiGroup
+			configSpec.AntreaNsx.BootstrapFrom.ProviderRef.Kind = config.Spec.AntreaNsx.BootstrapFrom.ProviderRef.Kind
+			configSpec.AntreaNsx.BootstrapFrom.ProviderRef.Name = config.Spec.AntreaNsx.BootstrapFrom.ProviderRef.Name
+		}


Is there any else branch required here?

if this pr #4219 can be merged into v0.28, this should also be included in v0.28

...vendir/cni/_ytt_lib/addons/packages/antrea/1.9.0-p1/bundle/config/upstream/interworking.yaml

.gitignore

...cni/_ytt_lib/addons/packages/antrea/1.9.0-p1/bundle/config/overlay/interworking-overlay.yaml

providers/ytt/vendir/cni/_ytt_lib/addons/packages/antrea/1.9.0-p1/bundle/config/schema.yaml

providers/ytt/vendir/cni/_ytt_lib/addons/packages/antrea/1.9.0-p1/bundle/config/values.yaml

addons/controllers/antrea/antreaconfig_util.go

...ib/addons/packages/antrea/1.9.0-p1/bundle/config/overlay/interworking-bootstrap-overlay.yaml

github-actions · 2023-03-29T01:04:47Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230329010445/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-03-29T06:44:59Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230329064458/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

joshuatcasey

Hi @wenqiq I'm not actually sure why this PR requires an approval from @vmware-tanzu/tkg-iam-owners , but I'll drop an approve here as this does not appear to impact any TKG IAM code.

wenqiq · 2023-03-30T01:35:23Z

Hi @wenqiq I'm not actually sure why this PR requires an approval from @vmware-tanzu/tkg-iam-owners , but I'll drop an approve here as this does not appear to impact any TKG IAM code.

Thanks for reviewing. Not quite sure how it works, it seems to be triggered automatically.

github-actions · 2023-03-30T19:35:48Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230330193546/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-03-30T19:46:37Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230330194635/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-advanced-security · 2023-03-31T05:11:43Z

You have successfully added a new Trivy configuration .github/workflows/cve_scan.yaml:trivy_scan. As part of the setup process, we have scanned this repository and found no existing alerts. In the future, you will see all code scanning alerts on the repository Security tab.

github-actions · 2023-03-31T05:11:47Z

CVE Scan results for this PR can be viewed from
https://github.com/vmware-tanzu/tanzu-framework/security/code-scanning?query=pr%3A4510

github-actions · 2023-03-31T05:19:21Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230331051919/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-04-03T09:00:54Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230403090052/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-04-03T09:27:22Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230403092720/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-04-04T07:48:55Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230404074853/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-04-05T01:08:22Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230405010820/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

edwardbadboy · 2023-04-07T07:39:48Z

apis/addonconfigs/cni/v1alpha2/antreaconfig_types.go

 	// +kubebuilder:validation:Optional
-	Inline *AntreaNsxInline `json:"inline,omitempty"`
-}
+	BootstrapSupervisorResourceName string `json:"bootstrapSupervisorResourceName,omitempty"`


I think we don't need to expose BootstrapSupervisorResourceName to CRD. BootstrapSupervisorResourceName is created by the antrea addon controller itself.

edwardbadboy · 2023-04-07T07:42:33Z

addons/controllers/antrea/antreaconfig_util.go

+			configSpec.AntreaNsx.AntreaNsxConfig.NSXCert = string(secret.Data["tls.crt"])
+			configSpec.AntreaNsx.AntreaNsxConfig.NSXKey = string(secret.Data["tls.key"])
+			configSpec.AntreaNsx.AntreaNsxConfig.VPCPath = config.Spec.AntreaNsx.AntreaNsxConfig.VPCPath
+		case bootstrapFromSupervisorCluster:


If config.Spec.AntreaNsx.AntreaNsxConfig.BootstrapFrom is empty or == bootstrapFromSupervisorCluster

edwardbadboy · 2023-04-07T07:43:28Z

addons/controllers/antrea/antreaconfig_util.go

+			configSpec.AntreaNsx.AntreaNsxConfig.NSXKey = string(secret.Data["tls.key"])
+			configSpec.AntreaNsx.AntreaNsxConfig.VPCPath = config.Spec.AntreaNsx.AntreaNsxConfig.VPCPath
+		case bootstrapFromSupervisorCluster:
+			configSpec.AntreaNsx.AntreaNsxConfig.BootstrapSupervisorResourceName = config.Spec.AntreaNsx.AntreaNsxConfig.BootstrapSupervisorResourceName


configSpec.AntreaNsx.AntreaNsxConfig.BootstrapSupervisorResourceName = NSXServiceAccount CR name. The NSXServiceAccount is created by Antrea addon controller, too. It's better find a way to derive the name automatically, rather than having user fill in the name manually.

edwardbadboy · 2023-04-07T07:56:45Z

apis/addonconfigs/cni/v1alpha2/antreaconfig_types.go

@@ -288,51 +288,169 @@ type AntreaNsx struct {
 	// +kubebuilder:validation:Optional
 	// +kubebuilder:default:=false
 	Enable bool `json:"enable,omitempty"`
-	// BootstrapFrom either providerRef or inline configs
-	// +kubebuilder:validation:Optional
-	BootstrapFrom AntreaNsxBootstrapFrom `json:"bootstrapFrom,omitempty"`


We changed the CRD, so need to generate manifest and commit the manifest changes.

github-actions · 2023-04-07T08:44:35Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230407084433/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

edwardbadboy · 2023-04-07T08:57:44Z

addons/controllers/antrea/antreaconfig_util.go

+			configSpec.AntreaNsx.AntreaNsxConfig.BootstrapFrom = bootstrapFromSupervisorCluster
+			configSpec.AntreaNsx.AntreaNsxConfig.BootstrapSupervisorResourceName = getNSXServiceAccountName(cluster.Name)
+		}
+		configSpec.AntreaNsx.AntreaNsxConfig.ProxyEndpoints.NSXRpcFwdProxy = config.Spec.AntreaNsx.AntreaNsxConfig.ProxyEndpoints.NSXRpcFwdProxy


Line 346 and 347 should be moved to inside if config.Spec.AntreaNsx.AntreaNsxConfig.BootstrapFrom == bootstrapFromInline {. This is when bootstrap from SupervisorCluster, the Antrea-NSX adapters load proxy endpoints from NSXServiceAccount from Supervisor, it will ignore the proxy endpoints set initially in the config file.

edwardbadboy · 2023-04-07T09:06:07Z

addons/controllers/antrea/antreaconfig_controller.go

@@ -232,14 +225,14 @@ func (r *AntreaConfigReconciler) getProviderServiceAccountName(clusterName strin
 	return fmt.Sprintf("%s-antrea", clusterName)


There is also a method func (r *AntreaConfigReconciler) getProviderServiceAccountName(clusterName string) in this file. I think we can also remove (r *AntreaConfigReconciler) from that method to make it a pure function, and then we move both getProviderServiceAccountName and getNSXServiceAccountName functions to antreaconfig_util.go.

edwardbadboy · 2023-04-07T09:15:04Z

apis/addonconfigs/cni/v1alpha2/antreaconfig_types.go

 	// +kubebuilder:validation:Optional
-	Name string `json:"name,omitempty"`
-}
+	NSXUser string `json:"nsxUser,omitempty"`


I think we don't need NSXUser and NSXPassword, maybe you forgot to remove them? They are useful for bootstrap Job, but we remove the bootstrap Job from the Package. Bootstrap Job is only used in interworking CI.

edwardbadboy · 2023-04-07T09:22:20Z

apis/addonconfigs/cni/v1alpha2/antreaconfig_types.go

 	// +kubebuilder:validation:Optional
-	ApiGroup string `json:"apigroup,omitempty"`
-	// Kind is the kind for crd, here its value is NsxServiceAccount
+	NSXCert string `json:"nsxCert,omitempty"`


I see that antreaconfig_util.go uses a getNSXCert function to get tls.crt and tls.key content from a Secret. I think that we can remove the NSXCert and NSXKey from the AntreaConfig CRD. Instead, we can introduce a NSXSecretName string field. It points to a Secret resource by name in the same Namespace. Then antreaconfig_util.go can refers to the NSXSecretName and pass it to getNSXCert function.

edwardbadboy · 2023-04-07T10:10:21Z

addons/controllers/antrea/antreaconfig_util.go

+	bootstrapFromInline            = "Inline"
+	bootstrapFromSupervisorCluster = "SupervisorCluster"
+)
+
 // AntreaConfigSpec defines the desired state of AntreaConfig
 type AntreaConfigSpec struct {


AntreaConfigSpec should be aligned with Antrea package schema.yaml.

The AntreaNSX member in this struct needs to be renamed as antrea_nsx, and yaml:"antreaNsx,omitempty" needs to be changed to yaml:"antrea_nsx,omitempty".

edwardbadboy · 2023-04-07T10:11:44Z

addons/controllers/antrea/antreaconfig_util.go

-	BootstrapFrom   antreaNsxBootstrapFrom `yaml:"bootstrapFrom,omitempty"`
-	AntreaNsxConfig antreaNsxConfig        `yaml:"config,omitempty"`
+	Enable          bool            `yaml:"enable,omitempty"`
+	AntreaNsxConfig antreaNsxConfig `yaml:"config,omitempty"`


Looks like AntreaNsxConfig should be moved to AntreaConfigSpec, and yaml:"config,omitempty" should be changed to yaml:"antrea_interworking,omitempty". This is to be aligned with Antrea package schema.yaml.

liu4480 · 2023-04-07T10:22:00Z

addons/controllers/antrea/antreaconfig_util.go


 	return configSpec, nil
 }
+
+func copyStructAtoB(a interface{}, b interface{}) error {
+	va := reflect.ValueOf(a)


reflect can make it more feasible, but at same time it can introduce performance cost, maybe consider if the implementation is OK.

github-actions · 2023-04-07T12:04:33Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230407120432/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-04-07T15:31:59Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230407153157/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

github-actions · 2023-04-07T18:53:14Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230407185312/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

Signed-off-by: Wenqi Qiu <[email protected]> Fix unit-test Signed-off-by: Wenqi Qiu <[email protected]>

Signed-off-by: Wenqi Qiu <[email protected]>

github-actions · 2023-04-10T15:53:10Z

Cluster Generation A/B Results:
https://storage.googleapis.com/tkg-clustergen/4510/20230410155308/clustergen.diff.txt
Author/reviewers:
Please review to verify that the effects on the generated cluster configurations are exactly what the PR intended, and give a thumbs-up if so.

wenqiq requested review from a team as code owners March 22, 2023 06:49

vmwclabot added the cla-not-required label Mar 22, 2023

wenqiq mentioned this pull request Mar 22, 2023

[WIP]Add antrea-interworking package #4098

Closed

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from c4cabd5 to 9966d51 Compare March 23, 2023 08:41

wjun requested review from hangyan, edwardbadboy and knabben March 24, 2023 08:07

hangyan reviewed Mar 27, 2023

View reviewed changes

addons/controllers/antrea/antreaconfig_util.go Outdated Show resolved Hide resolved

wjun reviewed Mar 27, 2023

View reviewed changes

...vendir/cni/_ytt_lib/addons/packages/antrea/1.9.0-p1/bundle/config/upstream/interworking.yaml Outdated Show resolved Hide resolved

knabben reviewed Mar 27, 2023

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

edwardbadboy reviewed Mar 28, 2023

View reviewed changes

kmova added the do-not-merge/hold Some fixes necessary, hold for merging label Mar 28, 2023

wenqiq requested a review from a team as a code owner March 29, 2023 00:49

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from 3be91aa to 06e90be Compare March 29, 2023 06:33

joshuatcasey approved these changes Mar 29, 2023

View reviewed changes

kmova removed the do-not-merge/hold Some fixes necessary, hold for merging label Mar 30, 2023

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from 2bfaeeb to 5c348fd Compare March 30, 2023 19:30

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from 5c348fd to f163330 Compare March 30, 2023 19:34

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from f163330 to c990216 Compare March 31, 2023 05:10

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from c990216 to 27c0d76 Compare April 3, 2023 08:53

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from 27c0d76 to fd444fd Compare April 3, 2023 09:20

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from 31d8665 to 1b6ac9b Compare April 5, 2023 00:58

edwardbadboy reviewed Apr 7, 2023

View reviewed changes

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from 1b6ac9b to e1aa0ba Compare April 7, 2023 08:36

edwardbadboy reviewed Apr 7, 2023

View reviewed changes

liu4480 reviewed Apr 7, 2023

View reviewed changes

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from e1aa0ba to a8d0063 Compare April 7, 2023 11:57

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from 2317ac9 to dcfc598 Compare April 7, 2023 18:44

wenqiq added 2 commits April 10, 2023 23:44

Add antrea-interworking 0.9.0

f4d1379

Signed-off-by: Wenqi Qiu <[email protected]> Fix unit-test Signed-off-by: Wenqi Qiu <[email protected]>

Update images.yaml

1e97f4f

Signed-off-by: Wenqi Qiu <[email protected]>

wenqiq force-pushed the topic/wenqi/antrea-interworking branch from dcfc598 to 1e97f4f Compare April 10, 2023 15:45

		@@ -232,14 +225,14 @@ func (r *AntreaConfigReconciler) getProviderServiceAccountName(clusterName strin
		return fmt.Sprintf("%s-antrea", clusterName)

Add antrea-interworking package #4510

Are you sure you want to change the base?

Add antrea-interworking package #4510

Conversation

wenqiq commented Mar 22, 2023

What this PR does / why we need it

Which issue(s) this PR fixes

Describe testing done for PR

Release note

Additional information

Special notes for your reviewer

github-actions bot commented Mar 22, 2023

codecov bot commented Mar 22, 2023 • edited Loading

Codecov Report

github-actions bot commented Mar 23, 2023

github-actions bot commented Mar 23, 2023

github-actions bot commented Mar 24, 2023

hangyan commented Mar 27, 2023

wjun Mar 27, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 29, 2023

github-actions bot commented Mar 29, 2023

joshuatcasey left a comment

Choose a reason for hiding this comment

wenqiq commented Mar 30, 2023

github-actions bot commented Mar 30, 2023

github-actions bot commented Mar 30, 2023

github-advanced-security bot commented Mar 31, 2023

github-actions bot commented Mar 31, 2023

github-actions bot commented Mar 31, 2023

github-actions bot commented Apr 3, 2023

github-actions bot commented Apr 3, 2023

github-actions bot commented Apr 4, 2023

github-actions bot commented Apr 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Apr 7, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Apr 7, 2023

github-actions bot commented Apr 7, 2023

github-actions bot commented Apr 7, 2023

github-actions bot commented Apr 10, 2023

codecov bot commented Mar 22, 2023 •

edited

Loading

wjun Mar 27, 2023 •

edited

Loading