make Private IP of the internal LB of the API Server configurable

[nawazkh]

What type of PR is this?
/kind feature

What this PR does / why we need it:

• This PR makes the private IP of the internal LB of the API Server configurable.
• We need private IP of the workload's API server to be configurable when deploying multiple workload clusters.
  • CAPZ defaults the Private IP to 10.0.0.100 (strongly coupled with DefaultVnetCIDR and DefaultControlPlaneSubnetCIDR)
  • While working with AKS as the management cluster; we peer mgmt VNet with Workload cluster's VNet so that mgmt cluster can reach to FQDN of the workload cluster's controlplane.
  • Since we peer VNets, we need the VNet CIDRs to be non-overlapping.
  • Since the VNet CIDRs have to be non-overlapping, we cannot default to 10.0.0.100 as the VNet CIDR will be opinionated when peering mgmt and workload cluster VNets.
  • Therefore, we want to have a configurable Private IP of the Internal LB of the API Server.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #5264

Special notes for your reviewer:

• cherry-pick candidate

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

Private IP of the Internal LB of the API Server will be configurable

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[codecov]

Codecov Report

Attention: Patch coverage is 92.55319% with 7 lines in your changes missing coverage. Please review.

Project coverage is 52.85%. Comparing base (8a89dfa) to head (2a44ea8).
Report is 8 commits behind head on main.

Files with missing lines	Patch %	Lines
api/v1beta1/azurecluster_default.go	64.28%	4 Missing and 1 partial ⚠️
api/v1beta1/azurecluster_validation.go	94.28%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5274      +/-   ##
==========================================
+ Coverage   52.82%   52.85%   +0.03%     
==========================================
  Files         270      270              
  Lines       29085    29121      +36     
==========================================
+ Hits        15365    15393      +28     
- Misses      12928    12935       +7     
- Partials      792      793       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[willie-yao]

/assign

[nawazkh]

A couple of questions:

• We havent documented in our docs that the CAPZ templates will now allow specifying private IP for the API Server of the workload cluster. I think we should update the documentation as well.
• We havent documented that an internal LB is created along side a public lb every time a user creates a template with public LB. And we are also updating the hosts file of the worker nodes. We should document that too.

Shall I do them in this PR or the uber PR #5248 that is revamping Tilt workflow ?

[willie-yao]

@nawazkh I think the documentation can be added in the uber PR if it will land prior to the release. Otherwise we'd want the docs to be up to date for the release.

[willie-yao]

/lgtm
/assign nojnhuh

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 76b1b171fd6a9e42e3fbcb5cbfcf3e0f7af1d1fd

[nojnhuh]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nojnhuh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nojnhuh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment