Update `vpc.max.networks` setting & settings to limit the number of NICs for each hypervisor #8654

hsato03 · 2024-02-14T18:27:38Z

Description

Each hypervisor can support a different number of network adapters. Comparing KVM and VMWare, VMWare defines a limited number of NICs for each ESXi machine (https://configmax.esp.vmware.com/guest?vmwareproduct=vSphere&release=vSphere%207.0&categories=1-0), while the number of tiers that can be allocated using KVM depends on the number of PCI slots availabe. For example, KVM provides 32 PCI slots, which are used to connect several devices, e.g. CD-ROM, keyboard, etc. Every ACS VR already consumes 9 slots of the 32 available; thus, in KVM we can have 23 slots for new tiers to be added.

This PR updates the vpc.max.networks setting to ConfigKey and changes its scope to the cluster level, as the maximum number of networks per VPC depends on the hypervisor where the VR is deployed.

The setting value is defined based on the cluster in which the VPC's VR is running, using the lowest value found if the VPC has more than one VR and they are in different clusters. If the VPC does not have a VR, the value defined in the global setting is used.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

I changed the vpc.max.networks value in the following resources:

Global setting: 3;
cluster-test: 7;
cluster-test2: 5.

I created 3 VPCs:

VPC1: without VRs;
VPC2: VR within cluster-test;
VPC3: VRs within cluster-test and cluster-test2.

Then, I verfiied that the vpc.max.networks setting value was:

3 in VPC1;
7 in VPC2;
5 in VPC3.

codecov · 2024-02-14T19:19:16Z

Codecov Report

Attention: Patch coverage is 88.73239% with 8 lines in your changes are missing coverage. Please review.

Project coverage is 30.83%. Comparing base (12f65fb) to head (63ab94a).
Report is 5 commits behind head on main.

Files	Patch %	Lines
...ain/java/com/cloud/network/vpc/VpcManagerImpl.java	72.72%	4 Missing and 2 partials ⚠️
...tack/engine/orchestration/NetworkOrchestrator.java	94.44%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8654      +/-   ##
============================================
- Coverage     31.04%   30.83%   -0.21%     
+ Complexity    33902    33647     -255     
============================================
  Files          5404     5405       +1     
  Lines        380305   380371      +66     
  Branches      55506    55519      +13     
============================================
- Hits         118056   117298     -758     
- Misses       246496   247468     +972     
+ Partials      15753    15605     -148

Flag	Coverage Δ
simulator-marvin-tests	`24.42% <60.56%> (-0.35%)`	⬇️
uitests	`4.34% <ø> (ø)`
unit-tests	`16.89% <76.05%> (+0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

weizhouapache · 2024-02-14T19:23:00Z

@hsato03

I agree this setting should by dynamic, and it is better to be a zone-level setting. But, vpc does not belong to a cluster and vms in vpc might run with different hypervisors, in different clusters. IMHO, It does not make sense to be a cluster-level setting.

hsato03 · 2024-02-14T21:09:16Z

@weizhouapache

I agree this setting should by dynamic, and it is better to be a zone-level setting.

Since the maximum number of networks per VPC depends on the hypervisor where the VR is deployed (as pointed out in the description), I don't think it makes sense to change it to the zone level as the cluster is the highest possible structure to define a hypervisor.

But, vpc does not belong to a cluster and vms in vpc might run with different hypervisors, in different clusters.

The setting value is defined by the cluster in which the VPC's VR is running, using the lowest value found if the VPC has more than one VR and they are in different clusters (https://github.com/apache/cloudstack/pull/8654/files#diff-07bd71d9f58832d4429d7743f4887188a96aacc913dc48e0101470147ce42032R1893-R1922). Regarding and vms in vpc might run with different hypervisors, VMs running in different hypervisors do not impact in the VPC creation; therefore, does not make sense to consider it in the algorithm.

weizhouapache · 2024-02-14T21:22:42Z

@weizhouapache

I agree this setting should by dynamic, and it is better to be a zone-level setting.

Since the maximum number of networks per VPC depends on the hypervisor where the VR is deployed (as pointed out in the description), I don't think it makes sense to change it to the zone level as the cluster is the highest possible structure to define a hypervisor.

But, vpc does not belong to a cluster and vms in vpc might run with different hypervisors, in different clusters.

The setting value is defined by the cluster in which the VPC's VR is running, using the lowest value found if the VPC has more than one VR and they are in different clusters (https://github.com/apache/cloudstack/pull/8654/files#diff-07bd71d9f58832d4429d7743f4887188a96aacc913dc48e0101470147ce42032R1893-R1922). Regarding and vms in vpc might run with different hypervisors, VMs running in different hypervisors do not impact in the VPC creation; therefore, does not make sense to consider it in the algorithm.

I can understand your code.
Your requirement is not clear to me. Can you explain more about the infrastructure? e.g. the hypervisor types, why the max vpc networks are different, what are the issues? Can it be a domain or account setting?

hsato03 · 2024-02-15T21:19:25Z

I can understand your code. Your requirement is not clear to me. Can you explain more about the infrastructure? e.g. the hypervisor types, why the max vpc networks are different, what are the issues? Can it be a domain or account setting?

@weizhouapache Each hypervisor can support a different number of network adapters. Comparing KVM and VMWare, VMWare defines a limited number of NICs for each ESXi machine (https://configmax.esp.vmware.com/guest?vmwareproduct=vSphere&release=vSphere%207.0&categories=1-0), while the number of tiers that can be allocated using KVM depends on the number of PCI slots availabe. For example, KVM provides 32 PCI slots, which are used to connect several devices, e.g. CD-ROM, keyboard, etc. Every ACS VR already consumes 9 slots of the 32 available; thus, in KVM we can have 23 slots for new tiers to be added.

Therefore, in an environment with KVM and VMware clusters under the same zone, applying the VMware limit to KVM is not interesting, as a VPC in KVM supports way more tiers than in VMware.

I will update the PR's description to make it clearer.

weizhouapache · 2024-02-16T10:42:08Z

I can understand your code. Your requirement is not clear to me. Can you explain more about the infrastructure? e.g. the hypervisor types, why the max vpc networks are different, what are the issues? Can it be a domain or account setting?

@weizhouapache Each hypervisor can support a different number of network adapters. Comparing KVM and VMWare, VMWare defines a limited number of NICs for each ESXi machine (https://configmax.esp.vmware.com/guest?vmwareproduct=vSphere&release=vSphere%207.0&categories=1-0), while the number of tiers that can be allocated using KVM depends on the number of PCI slots availabe. For example, KVM provides 32 PCI slots, which are used to connect several devices, e.g. CD-ROM, keyboard, etc. Every ACS VR already consumes 9 slots of the 32 available; thus, in KVM we can have 23 slots for new tiers to be added.

Therefore, in an environment with KVM and VMware clusters under the same zone, applying the VMware limit to KVM is not interesting, as a VPC in KVM supports way more tiers than in VMware.

I will update the PR's description to make it clearer.

@hsato03
makes perfect sense.

However ...
the number of nics of VPC VRs contains

1 control NIC
multiple guest NICs
multiple public NICs

I would suggest you to create a setting for each hypervisor, e.g. virtual.machine.max.nics.kvm and virtual.machine.max.nics.vmware which consider all type of NICs, and check the value when

add guest NIC to VR (of VPC and isolated networks, and even Shared networks)
add public NICs to VR (of VPC and isolated networks, and even Shared networks)
add NICs to user VMs

weizhouapache · 2024-02-17T08:38:04Z

@hsato03
What do you think?
The limitations apply on all vms, including user vms and VRs.

@hsato03
makes perfect sense.

However ...
the number of nics of VPC VRs contains

1 control NIC

multiple guest NICs

multiple public NICs

I would suggest you to create a setting for each hypervisor, e.g. virtual.machine.max.nics.kvm and virtual.machine.max.nics.vmware which consider all type of NICs, and check the value when

add guest NIC to VR (of VPC and isolated networks, and even Shared networks)

add public NICs to VR (of VPC and isolated networks, and even Shared networks)

add NICs to user VMs

hsato03 · 2024-02-21T20:26:21Z

@weizhouapache Thanks for your suggestion.

I agree that this situation should include VMs and VRs but the vpc.max.networks setting should still exist if the user wants to limit the number of tiers in a VPC regardless of the hypervisor. What do you think about delegating the limit checking of a hypervisor based on the settings you suggested and changing the scope of the vpc.max.networks to the account level in this case?

weizhouapache · 2024-02-21T21:30:09Z

@weizhouapache Thanks for your suggestion.

I agree that this situation should include VMs and VRs but the vpc.max.networks setting should still exist if the user wants to limit the number of tiers in a VPC regardless of the hypervisor.

agree.

What do you think about delegating the limit checking of a hypervisor based on the settings you suggested and changing the scope of the vpc.max.networks to the account level in this case?

It looks good. Looking forward to your changes

github-actions · 2024-03-08T10:41:00Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java

DaanHoogland · 2024-03-29T15:43:27Z

@blueorangutan package

blueorangutan · 2024-03-29T15:44:03Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-05-07T22:06:43Z

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9570

hsato03 · 2024-05-31T18:50:01Z

@blueorangutan package

blueorangutan · 2024-05-31T18:52:03Z

@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-05-31T19:09:02Z

Packaging result [SF]: ✖️ el7 ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 9751

github-actions · 2024-06-20T06:39:29Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

DaanHoogland · 2024-06-20T07:22:16Z

@hsato03

21:06:02 [INFO] -------------------------------------------------------------
21:06:07 [ERROR] COMPILATION ERROR : 
21:06:07 [INFO] -------------------------------------------------------------
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1143,15] error: incompatible types: HypervisorType cannot be converted to int
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1144,17] error: cannot find symbol
21:06:07   symbol:   variable KVM
21:06:07   location: class NetworkOrchestrator
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1147,17] error: cannot find symbol
21:06:07   symbol:   variable VMware
21:06:07   location: class NetworkOrchestrator
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1150,17] error: cannot find symbol
21:06:07   symbol:   variable XenServer
21:06:07   location: class NetworkOrchestrator
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1167,15] error: incompatible types: HypervisorType cannot be converted to int
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1168,17] error: cannot find symbol
21:06:07   symbol:   variable KVM
21:06:07   location: class NetworkOrchestrator
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1171,17] error: cannot find symbol
21:06:07   symbol:   variable VMware
21:06:07   location: class NetworkOrchestrator
21:06:07 [ERROR] /jenkins/workspace/acs-centos7-pkg-builder/dist/rpmbuild/BUILD/cloudstack-4.20.0.0-SNAPSHOT/engine/orchestration/src/main/java/org/apache/cloudstack/engine/orchestration/NetworkOrchestrator.java:[1174,17] error: cannot find symbol
21:06:07   symbol:   variable XenServer
21:06:07   location: class NetworkOrchestrator
21:06:07 [INFO] 8 errors

I think it would be a merge problem, but notice that the GHA build also fails with this error: https://github.com/apache/cloudstack/actions/runs/8935283845/job/24543531187?pr=8654#step:7:18232

hsato03 · 2024-07-11T16:32:53Z

@blueorangutan package

blueorangutan · 2024-07-11T16:34:03Z

@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-07-11T17:45:18Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10329

JoaoJandre · 2024-07-24T12:23:37Z

@blueorangutan package

blueorangutan · 2024-07-24T12:24:03Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-07-24T13:32:57Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 10457

JoaoJandre · 2024-08-13T19:46:56Z

@blueorangutan package

blueorangutan · 2024-08-13T19:48:03Z

@JoaoJandre a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-08-13T20:45:21Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10643

blueorangutan · 2024-08-14T14:31:59Z

Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 10662

github-actions · 2024-09-06T05:26:18Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

hsato03 · 2024-10-16T18:36:05Z

@blueorangutan package

blueorangutan · 2024-10-16T18:38:03Z

@hsato03 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-10-16T19:40:24Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 11368

DaanHoogland · 2024-10-28T14:13:19Z

@blueorangutan test

blueorangutan · 2024-10-28T14:14:04Z

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2024-10-29T06:41:52Z

[SF] Trillian test result (tid-11710)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 56931 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8654-t11710-kvm-ol8.zip
Smoke tests completed. 140 look OK, 2 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
test_07_vpc_and_tier_with_routed_mode	`Error`	2.34	test_ipv4_routing.py
test_09_connectivity_between_network_and_vpc_tier	`Failure`	0.12	test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode	`Error`	7.80	test_ipv4_routing.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	412.71	test_vpc_redundant.py

DaanHoogland · 2024-10-29T09:33:47Z

[SF] Trillian test result (tid-11710) Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8 Total time taken: 56931 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr8654-t11710-kvm-ol8.zip Smoke tests completed. 140 look OK, 2 have errors, 0 did not run Only failed and skipped tests results shown below:
Test Result Time (s) Test File
test_07_vpc_and_tier_with_routed_mode Error 2.34 test_ipv4_routing.py
test_09_connectivity_between_network_and_vpc_tier Failure 0.12 test_ipv4_routing.py
test_12_vpc_and_tier_with_dynamic_routed_mode Error 7.80 test_ipv4_routing.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL Error 412.71 test_vpc_redundant.py

@hsato03 these errors seem related to the change, an you have a look?

Update vpc.max.networks setting

5096a0b

boring-cyborg bot added component:networking component:vpc labels Feb 14, 2024

hsato03 marked this pull request as draft February 23, 2024 15:19

Change vpc.max.networks to account scope & create vm.max.nics settings

1b34e03

boring-cyborg bot added the component:orchestration label Mar 1, 2024

Adjusts

e0cddb7

github-actions bot added the status:has-conflicts label Mar 8, 2024

hsato03 changed the title ~~Update vpc.max.networks setting~~ Update vpc.max.networks setting & settings to limit the number of NICs for each hypervisor Mar 19, 2024

Henrique Sato added 2 commits March 21, 2024 16:08

Limit number of tiers

e0e5801

Resolve conflicts

209dd67

github-actions bot removed the status:has-conflicts label Mar 21, 2024

hsato03 marked this pull request as ready for review March 21, 2024 19:52

Henrique Sato added 2 commits March 21, 2024 17:56

Fix validation

0c13784

Desconsider redundant routers in validation

10d58f0

weizhouapache reviewed Mar 21, 2024

View reviewed changes

server/src/main/java/com/cloud/network/vpc/VpcManagerImpl.java Outdated Show resolved Hide resolved

Henrique Sato added 2 commits March 22, 2024 17:19

Change query

88d92ee

Compare single VR

63ab94a

github-actions bot added the status:has-conflicts label Jun 20, 2024

Resolve Conflicts

bf70339

github-actions bot removed the status:has-conflicts label Jul 10, 2024

github-actions bot added the status:has-conflicts label Sep 6, 2024

Resolve conflicts

acde1d9

github-actions bot removed the status:has-conflicts label Oct 16, 2024

Update vpc.max.networks setting & settings to limit the number of NICs for each hypervisor #8654

Are you sure you want to change the base?

Update vpc.max.networks setting & settings to limit the number of NICs for each hypervisor #8654

Conversation

hsato03 commented Feb 14, 2024 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

codecov bot commented Feb 14, 2024 • edited Loading

Codecov Report

weizhouapache commented Feb 14, 2024

hsato03 commented Feb 14, 2024

weizhouapache commented Feb 14, 2024

hsato03 commented Feb 15, 2024

weizhouapache commented Feb 16, 2024

weizhouapache commented Feb 17, 2024

hsato03 commented Feb 21, 2024

weizhouapache commented Feb 21, 2024

github-actions bot commented Mar 8, 2024

DaanHoogland commented Mar 29, 2024

blueorangutan commented Mar 29, 2024

blueorangutan commented May 7, 2024

hsato03 commented May 31, 2024

blueorangutan commented May 31, 2024

blueorangutan commented May 31, 2024

github-actions bot commented Jun 20, 2024

DaanHoogland commented Jun 20, 2024

hsato03 commented Jul 11, 2024

blueorangutan commented Jul 11, 2024

blueorangutan commented Jul 11, 2024

JoaoJandre commented Jul 24, 2024

blueorangutan commented Jul 24, 2024

blueorangutan commented Jul 24, 2024

JoaoJandre commented Aug 13, 2024

blueorangutan commented Aug 13, 2024

blueorangutan commented Aug 13, 2024

blueorangutan commented Aug 14, 2024

github-actions bot commented Sep 6, 2024

hsato03 commented Oct 16, 2024

blueorangutan commented Oct 16, 2024

blueorangutan commented Oct 16, 2024

DaanHoogland commented Oct 28, 2024

blueorangutan commented Oct 28, 2024

blueorangutan commented Oct 29, 2024

DaanHoogland commented Oct 29, 2024

Update `vpc.max.networks` setting & settings to limit the number of NICs for each hypervisor #8654

Update `vpc.max.networks` setting & settings to limit the number of NICs for each hypervisor #8654

hsato03 commented Feb 14, 2024 •

edited

Loading

codecov bot commented Feb 14, 2024 •

edited

Loading