Configure the level of IPAM with UDNs #1708
Conversation
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
phoracek
force-pushed
the
ipam-level
branch
2 times, most recently
from
595d0c8
to
aa47656
Compare
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: maiqueb The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
| @@ -464,6 +465,7 @@ Suggested API validation rules: | |||
| - `Subnets` are mandatory for `Layer3` topology. | |||
| - `Localnet` topology is not supported for primary network. | |||
| - `IPAMLifecycle` is supported for `Layer2` and `Localnet` topology. | |||
| - When `IPAMLevel` is `Disabled`, `Subnets` cannot be set. When it is `FullyManaged`, `Subnets` are required. | |||
There was a problem hiding this comment.
I assume for layer3 there can never be a disabled mode right? for both role primary and role secondary?
There was a problem hiding this comment.
I think so too - L3 has to do routing between nodes, and it could not do routing without subnet awareness.
There was a problem hiding this comment.
Tell me if you would prefer me to reword this line. I think that the condition is already present in the list, but it is not obvious (subnets and ipamlevel points need to be merged).
The current proposal for UDN's IPAM configuration is limited and would not allow a simple expansion covering other use-cases. This patch aims to make it more future-proof. There are two limitations that this should solve: 1) The current proposal does not leave space for default subnets value 2) The current proposal does not leave space for granular control of IPAM This patch should solve both by introducing a new attribute called `IPAMLevel`. This attribute dictates how much of the network IP configuration will be done by OVN and how much is left to the user. The initial implementation of 4.18 will need only two values: `Disabled` and `FullyManaged`. Later releases should introduce a new level, when OVN will manage configuration of logical routers (routing, NAT, default GW), but configuration of individual Pod's / VM's IPs will be left to the user. Apart from introducing this new field, this patch also changes the behavior of nil subnets. While with the current enhancement not setting subnets means disabling IPAM, with this patch, not setting subnets is not allowed unless `Disabled` `IPAMLevel` is explicitly set. This will allow us to introduce default subnet value in the future, if we decide to go that path. Signed-off-by: Petr Horacek <[email protected]>
|
@phoracek: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
I think if we already have some future options like partially-managed in mind, it would be nice to include those options into this enhancement, to make sure our current API will be properly extended when needed. we don't need to implement all of the options at the same time though. Trying to figure out important parts of the subnet management process
@phoracek does that ^ sound right? |
|
@npinaeva thanks for the review. My goal with this PR is to keep the API open for the anticipated enhancements. I wanted to avoid designing the specific IPAM feature here - I don't trust myself to do it right and I'd rather leave that to you subject matter experts. But if you would find it useful to have an example of such an extension in this PR, I'm happy to do that (just take it with a grain of salt). Your summary of the 3 options matches my understanding / expectations +1. @npinaeva what would you like me to add to this PR? I can add the third option for |
|
The easiest-to-implement-now option I see, that would still allow adding different IPAM modes and default values in the future, is making |
We do use that. In fact, right now, that's the only option we support for virt workloads. We do not support IPAM in layer2 or localnet networks. IIUC, this proposal will not work for us. |
| @@ -415,6 +415,7 @@ The CRDs spec defines as follows: | |||
| | ExcludeSubnets | List of CIDRs.<br/>IP addresses are removed from the assignable IP address pool and are never passed to the pods. | Yes | | |||
| | JoinSubnets | Subnet used inside the OVN network topology. When omitted, this means no opinion and the platform is left to choose a reasonable default which is subject to change over time. | Yes | | |||
| | IPAMLifecycle | Control IP addresses management lifecycle. When `Persistent` is specified it enable workloads have persistent IP addresses. For example: Virtual Machines will have the same IP addresses along their lifecycle (stop, start migration, reboots). Supported by Topology `Layer2` & `Localnet`. | Yes | | |||
| | IPAMLevel | Control how much of the IP configuration will be managed by OVN-Kubernetes. Must be one of `Disabled`, `FullyManaged`. | Yes | | |||
There was a problem hiding this comment.
Naming suggestions (open for discussion)
IPAMLevel -> IPAMMode (level makes me think they are sorted, but I am not sure if that will always be the case)
FullyManaged -> Full
| // When `Disabled`, OVN-Kubernetes will only provide layer 2 communication, letting users configure IP addresses for the pods. `Disabled` is only available | ||
| // for `Layer2` and `Localnet` topologies. By disabling IPAM, you are also opting-out from features depending on it, e.g. IP spoof filtering and | ||
| // namespace/pod selectors in network policies. |
There was a problem hiding this comment.
we also have https://github.com/ovn-kubernetes/ovn-kubernetes/blob/f369f780b2bbc559d0a428761d7ba55c0097bc97/go-controller/pkg/crd/userdefinednetwork/v1/shared.go#L118-L119 piece of documentation, but it would be nice to have a full list of side-effects when disabling IPAM (or a list of what is supported if that is shorter). I can try to figure this out
There was a problem hiding this comment.
it's described here. This is more accurate.
The current proposal for UDN's IPAM configuration is limiting as it would not allow a (simple) expansion covering other anticipated use-cases. This patch aims to make it more future-proof.
There are two limitations that this patch should solve:
SubnetsvalueThis patch should solve both by introducing a new attribute called
IPAMLevel. This attribute dictates how much of the network IP configuration will be done by OVN and how much is left to the user.The initial implementation of 4.18 will need only two values:
DisabledandFullyManaged. Later releases should introduce a new level, where OVN will manage configuration of logical routers (routing, NAT, default GW), but configuration of individual Pod's / VM's IPs will be left to the user.Apart from introducing this new field, this patch also changes the behavior of
nilSubnets. While with the current enhancement not setting subnets means disabling IPAM, with this patch, not setting subnets is not allowed unlessDisabledIPAMLevelis explicitly set. This will allow us to introduce default subnet value in the future, if we decide to go that path.