Configure bond0 interface

[larsks]

We want to use bonded interface pairs on these system. The nodes
aren't yet wired for it, but setting this up now will allow us to
refer to the bond0 interface in e.g. VLAN configurations (and means
we won't have to re-work those later).

Because we're using OVNKubernetes, we can't use nmstate 1 to enact
the configuration. The recommendation is to apply the configuration
using a MachineConfig 2 resource, but this is complicated by the
fact that our nodes don't all have the same interface names, and it's
not possible to apply node-specific machineconfigs 3.

We work around this solution by:

1. Copying nmconnection files for all hosts to every host, but
   placing them in /etc/mco (just because that's a convenient
   available directory, it seems relatively topical, and it's not
   possible to create new directories using the directories section
   of an ignition config).

2. Installing a systemd unit that runs a shell script at boot that
   copies the host-specific configs from /etc/mco into
   /etc/NetworkManager/system-connections.

[larsks]

Because we're working around limitations in the MachineConfig operator, this is a pretty hairy solution. While I've been trying other things out "live", I really want some eyes on this before I push it out.

[dystewart]

Going to leave some questions/statements throughout to make sure I'm understanding the networking (which I'm not very familiar with)

[larsks]

@dystewart thanks for the questions; hope my answers are clear. Let me know if not!

[larsks]

An alternative solution here might be to be disable "predictable device names", because they don't appear to be predictable. In theory, the device name is based on the PCI bus/slot/etc, but in this case all three control nodes have an identical PCI configuration:

# lspci | grep -i ethernet
06:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
06:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
31:00.0 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 09)
31:00.1 Ethernet controller: Intel Corporation Ethernet Connection X722 for 10GBASE-T (rev 09)

But for whatever reason, the interface base name is ens6 on ctl-0 and ens5 on ctl-1 and ctl-2.

[larsks]

I think that disabling predicable interface names is the right idea. I've submitted an alternative implementation of this PR in #22. This is a dramatically simpler change, and some limited testing suggests that it works as advertised.

[larsks]

I've removed all the reviews and converted this PR to a draft because I think we'll end up going with #22.