Cumulus Point to point links with switch port specified #1452

SudarshanVK · 2024-10-28T05:24:35Z

SudarshanVK
Oct 28, 2024

Hey All,

I have a rather large cumulus environment that we are looking to lab out and maintain as a digital twin. Currently we are using vanilla Vagrant and the topology convert that Cumulus provides. I would love to move to netlab as it seems more manageable but having one issue.

I want to be able to specify connectivity between by specifying the port number.

For example , if i wanted swp6 on router1 connect to swp7 on router2, what I gather from the documentation is I would define my topology like the below? All of my links are either Layer2 or unnumbered. We run a vxlan fabric. Any help on this would be appreciated.

provider: libvirt
defaults.device: cumulus_nvue

nodes:
  - {"name": "router1", "image": "CumulusCommunity/cumulus-vx:5.10.0"}
  - {"name": "router2", "image": "CumulusCommunity/cumulus-vx:5.10.0"}

links:
  - {"router1": {"ifindex": 6}, "router2": {"ifindex": 7}}

addressing:
  mgmt:
    ipv4: 192.168.100.0/24
    mac: 08-4F-A9-00-00-00
    prefix: 24
    start: 20

SudarshanVK · 2024-10-28T06:27:55Z

SudarshanVK
Oct 28, 2024
Author

Some observations,

Just mentioning links without ifindex works fine, it connects the swp1 on both the devices to each other, shows up in lldp, has an ipaddress assigned and able to ping eachother.
With the ifindex mentioned, the swithchport is configured. when i cat /etc/network/interfaces i see a description and an ipaddress? but does not show up in lldp and unable to ping eachother.

Also, how can I configure them as unnumbered?

1 reply

ipspace Oct 28, 2024
Maintainer

With the ifindex mentioned, the swithchport is configured. when i cat /etc/network/interfaces i see a description and an ipaddress? but does not show up in lldp and unable to ping eachother.

OK, so you must be using VMs. KVM creates what look like virtual NICs when starting the VM, and the VM operating system detect those NICs and assign them names (usually starting with swp1).

We can specify whatever interface we wish in the /etc/network/interfaces, if it does not exist on the system, it's ignored.

Also, how can I configure them as unnumbered?

Once we'll figure out the interface naming everything else will click together.

ipspace · 2024-10-28T06:28:03Z

ipspace
Oct 28, 2024
Maintainer

I want to be able to specify connectivity between by specifying the port number.

You're doing exactly the right thing, but there's a gotcha -- the static "ifindex" changes the interface name in device configuration, but not necessarily the underlying virtualization configuration.

It works well with physical devices (for obvious reasons), and it works with containers (because containerlab allows us to name the container Ethernet interfaces whatever we wish), but not with VMs.

When using VMs, we configure the VM interfaces linearly from the node interfaces list, and the VM operating system maps those virtual NIC cards into interface names in whatever way it wishes.

I see two potential ways to solve this:

Reshuffle the interfaces based on ifindex and insert bogus interfaces before creating Vagrantfile
Renaming interface within Cumulus Linux.

It would be ideal if you could figure out how to do #2 ;) and then I could integrate that into the initial configuration template, or I could write a plugin (because I'm not sure I want to have that as part of core functionality) to do #1.

Finally, it seems you're using a really old way of specifying the nodes. It would be great if you could point me to where you found it so I could fix that. A much simpler way of specifying the nodes is with a dictionary. Instead of:

nodes:
  - {"name": "router1", "image": "CumulusCommunity/cumulus-vx:5.10.0"}
  - {"name": "router2", "image": "CumulusCommunity/cumulus-vx:5.10.0"}

... you could have:

nodes:
  router1:
  router2:

0 replies

SudarshanVK · 2024-10-28T06:40:11Z

SudarshanVK
Oct 28, 2024
Author

Hey Ivan, Please excuse my lack of understanding. Its only been a few months of me working with Cumulus. I have taken it over from someone who was not very forthcoming with information. I have no idea how I would even approach #2.

With regards to #1, Do you mean I add bogus interfaces from swp1 to swp5 on router1? Can you give me an example of how the topology file would look in that case?

With respect to the format, I can not seem to re-collect where i picked it from, obviously some example but I am just not able to get the link.

5 replies

ipspace Oct 28, 2024
Maintainer

I have no idea how I would even approach #2.

OK, I'll work on #1.

With regards to #1, Do you mean I add bogus interfaces from swp1 to swp5 on router1?

That would work, but it's a nice challenge, so I'll fix it with a plugin.

Can you give me an example of how the topology file would look in that case?

You would have to add links with a single node attached to them and using l2only pool so they don't get IPv4/IPv6 addresses. But please don't waste you time on that, wait for the plugin.

Example:

links:
- router1:
  pool: l2only
- router1:
  pool: l2only

I have a few topologies like that in the BGP labs repository, for example https://github.com/bgplab/bgplab/blob/master/session/2-asoverride/topology.yml

SudarshanVK Oct 28, 2024
Author

ah ok.. let me try with pool: l2only

By plugin do you mean add the dummy interfaces while generating the Vagrant file?
I was thinking worst case, I would write my won jinja2 template to generate the dummy interfaces and then use netlab.

SudarshanVK Oct 28, 2024
Author

Not quite, even after setting it to pool: l2only It seems to assign an IP address.

ipspace Oct 28, 2024
Maintainer

Not quite, even after setting it to pool: l2only It seems to assign an IP address.

Are you setting pool: l2only on the link (left-aligned with the device name) or within the device name?

SudarshanVK Oct 28, 2024
Author

ok, got it working now.. thank you..

SudarshanVK · 2024-10-28T06:58:19Z

SudarshanVK
Oct 28, 2024
Author

Ok, some progress this works.
But it seems to assign an IP address.

---
provider: libvirt
defaults.device: cumulus_nvue

nodes:
  router1:
    image: CumulusCommunity/cumulus-vx:5.10.0
  router2:
    image: CumulusCommunity/cumulus-vx:5.10.0

links:
  - router1:
      ifindex: 1
  - router2:
      ifindex: 1
  - router1:
      ifindex: 2
    router2:
      ifindex: 2

0 replies

SudarshanVK · 2024-10-28T20:17:03Z

SudarshanVK
Oct 28, 2024
Author

Adding the mock interfaces works. But i am just thinking about scalability. My lab has about 80 devices and each device has about 40 ports. Just worried that my topology file is going to become too huge to manage pretty quick.

While using vanilla Vagrant, I notice that it manages to create only interfaces which are specified. Below is the vagrant file maybe that helps figure out what we need to do to make it work in networklab?

# Created by Topology-Converter v4.7.1
#    Template Revision: v4.7.1
#    https://gitlab.com/cumulus-consulting/tools/topology_converter
#    using topology data from: topology_converter/topology.dot
#    built with the following args: topology_converter/topology_converter.py topology_converter/topology.dot -p libvirt
#
#    NOTE: in order to use this Vagrantfile you will need:
#       -Vagrant(v2.0.2+) installed: http://www.vagrantup.com/downloads
#       -the "helper_scripts" directory that comes packaged with topology-converter.py
#        -Libvirt Installed -- guide to come
#       -Vagrant-Libvirt Plugin installed: $ vagrant plugin install vagrant-libvirt
#       -Start with \"vagrant up --provider=libvirt --no-parallel\n")

#  Libvirt Start Port: 8000
#  Libvirt Port Gap: 1000

#Set the default provider to libvirt in the case they forget --provider=libvirt or if someone destroys a machine it reverts to virtualbox
ENV['VAGRANT_DEFAULT_PROVIDER'] = 'libvirt'

# Check required plugins
REQUIRED_PLUGINS_LIBVIRT = %w(vagrant-libvirt)
exit unless REQUIRED_PLUGINS_LIBVIRT.all? do |plugin|
  Vagrant.has_plugin?(plugin) || (
    puts "The #{plugin} plugin is required. Please install it with:"
    puts "$ vagrant plugin install #{plugin}"
    false
  )
end

Vagrant.require_version ">= 2.0.2"

# Fix for Older versions of Vagrant to Grab Images from the Correct Location
unless Vagrant::DEFAULT_SERVER_URL.frozen?
  Vagrant::DEFAULT_SERVER_URL.replace('https://vagrantcloud.com')
end

$script = <<-SCRIPT
function setup_ztp(){
    echo "### Disabling ZTP service..."
    systemctl stop ztp.service
    ztp -d 2>&1
    echo "### Resetting ZTP to work next boot..."
    ztp -R 2>&1
    ztp -i &> /dev/null

    if [ -e /tmp/cumulus-ztp ]; then
        echo "  ### Found ZTP Script, moving into preload directory... ###"
        mv /tmp/cumulus-ztp /var/lib/cumulus/ztp/cumulus-ztp
        chmod +x /var/lib/cumulus/ztp/cumulus-ztp
        ls -lha /var/lib/cumulus/ztp/cumulus-ztp
    fi
}

function disable_remap(){
    echo "### Disabling default remap on Cumulus VX..."
    if [ -f /etc/hw_init.d/S10rename_eth_swp.sh ]; then
        chmod -x /etc/hw_init.d/S10rename_eth_swp.sh
    fi
}

function vagrant_user_nclu(){
    echo "### Giving Vagrant User Ability to Run NCLU Commands ###"
    adduser vagrant netedit
    adduser vagrant netshow
}

function vagrant_user_nvue(){
    echo "### Giving Vagrant User Ability to Run NVUE Commands ###"
    adduser vagrant nvapply
}

if grep -q -i 'cumulus' /etc/lsb-release &> /dev/null; then
    echo "### RUNNING CUMULUS EXTRA CONFIG ###"
    source /etc/lsb-release
    echo "  INFO: Detected Cumulus Linux v$DISTRIB_RELEASE Release"
    if [ -e /etc/app-release ]; then
        echo "  INFO: Detected NetQ TS Server"
        source /etc/app-release
        echo "  INFO: Running NetQ TS Appliance Version $APPLIANCE_VERSION"
        disable_remap
        vagrant_user_nclu
        setup_ztp
    else
        if [[ $DISTRIB_RELEASE =~ ^2.* ]]; then
            echo "  INFO: Detected a 2.5.x Based Release"
            echo "     2.5.x: adding fake cl-acltool..."
            echo -e "#!/bin/bash\nexit 0" > /usr/bin/cl-acltool
            chmod 755 /usr/bin/cl-acltool
            echo "     2.5.x: adding fake cl-license..."
            echo -e "#!/bin/bash\nexit 0" > /usr/bin/cl-license
            chmod 755 /usr/bin/cl-license
            echo "     2.5.x: Disabling default remap on Cumulus VX..."
            mv -v /etc/init.d/rename_eth_swp /etc/init.d/rename_eth_swp.backup
        elif [[ $DISTRIB_RELEASE =~ ^3.* ]]; then
            echo "  INFO: Detected a 3.x Based Release ($DISTRIB_RELEASE)"
            echo "### Disabling default remap on Cumulus VX..."
            mv -v /etc/hw_init.d/S10rename_eth_swp.sh /etc/S10rename_eth_swp.sh.backup &> /dev/null
            if [[ $DISTRIB_RELEASE =~ ^3.[1-9].* ]]; then
                echo "### Fixing ONIE DHCP to avoid Vagrant Interface ###"
                echo "     Note: Installing from ONIE will undo these changes."
                mkdir /tmp/foo
                mount LABEL=ONIE-BOOT /tmp/foo
                sed -i 's/eth0/eth1/g' /tmp/foo/grub/grub.cfg
                sed -i 's/eth0/eth1/g' /tmp/foo/onie/grub/grub-extra.cfg
                umount /tmp/foo
            fi
            if [[ $DISTRIB_RELEASE =~ ^3.2.* ]]; then
                if [[ $(grep "vagrant" /etc/netd.conf | wc -l ) == 0 ]]; then
                    echo "### Giving Vagrant User Ability to Run NCLU Commands ###"
                    sed -i 's/users_with_edit = root, cumulus/users_with_edit = root, cumulus, vagrant/g' /etc/netd.conf
                    sed -i 's/users_with_show = root, cumulus/users_with_show = root, cumulus, vagrant/g' /etc/netd.conf
                fi
            elif [[ $DISTRIB_RELEASE =~ ^3.[3-9].* ]]; then
                vagrant_user_nclu
            fi
            setup_ztp
        elif [[ $DISTRIB_RELEASE =~ ^4.* ]]; then
            echo "  INFO: Detected a 4.x Based Release ($DISTRIB_RELEASE)"
            disable_remap
            vagrant_user_nclu
            setup_ztp
        elif [[ $DISTRIB_RELEASE =~ ^5.* ]]; then
            echo "  INFO: Detected a 5.x Based Release ($DISTRIB_RELEASE)"
            disable_remap
            vagrant_user_nvue
            setup_ztp
        fi
    fi
fi
echo "### DONE ###"
echo "### Rebooting Device to Apply Remap..."
nohup bash -c 'sleep 10; shutdown now -r "Rebooting to Remap Interfaces"' &
SCRIPT

Vagrant.configure("2") do |config|
  config.ssh.forward_agent = true
  VAGRANT_COMMAND = ARGV[0]


  wbid = 1
  offset = wbid * 100


  config.vm.provider :libvirt do |domain|
    domain.management_network_address = "10.255.#{wbid}.0/24"
    domain.management_network_name = "wbr#{wbid}"
    # increase nic adapter count to be greater than 8 for all VMs.
    domain.nic_adapter_count = 130
  end




  ##### DEFINE VM for router1 #####
  config.vm.define "router1" do |device|
    
    device.vm.hostname = "router1"
    
    device.vm.box = "CumulusCommunity/cumulus-vx"
    device.vm.box_version = "5.10.0"

    device.vm.provider :libvirt do |v|
      v.memory = 2048

    end
    #   see note here: https://github.com/pradels/vagrant-libvirt#synced-folders
    device.vm.synced_folder ".", "/vagrant", disabled: true



    # NETWORK INTERFACES
      # link for swp6 --> router2:swp7
      device.vm.network "private_network",
            :mac => "48:b0:2d:f0:2c:d4",
            :libvirt__tunnel_type => 'udp',
            :libvirt__tunnel_local_ip => '127.0.0.1',
            :libvirt__tunnel_local_port => "#{ 8001 + offset }",
            :libvirt__tunnel_ip => '127.0.0.1',
            :libvirt__tunnel_port => "#{ 9001 + offset }",
            :libvirt__iface_name => 'swp6',
            auto_config: false



    # Copy over Topology.dot File
    device.vm.provision "file", source: "topology_converter/topology.dot", destination: "~/topology.dot"
    device.vm.provision :shell, privileged: false, inline: "sudo mv ~/topology.dot /etc/ptm.d/topology.dot"



    # Install Rules for the interface re-map
    device.vm.provision :shell , :inline => <<-delete_udev_directory
if [ -d "/etc/udev/rules.d/70-persistent-net.rules" ]; then
    rm -rfv /etc/udev/rules.d/70-persistent-net.rules &> /dev/null
fi
rm -rfv /etc/udev/rules.d/70-persistent-net.rules &> /dev/null
delete_udev_directory

device.vm.provision :shell , :inline => <<-udev_rule
echo "  INFO: Adding UDEV Rule: 48:b0:2d:f0:2c:d4 --> swp6"
echo 'ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="48:b0:2d:f0:2c:d4", NAME="swp6", SUBSYSTEMS=="pci"' >> /etc/udev/rules.d/70-persistent-net.rules
udev_rule
     
      device.vm.provision :shell , :inline => <<-vagrant_interface_rule
echo "  INFO: Adding UDEV Rule: Vagrant interface = eth0"
echo 'ACTION=="add", SUBSYSTEM=="net", ATTR{ifindex}=="2", NAME="eth0", SUBSYSTEMS=="pci"' >> /etc/udev/rules.d/70-persistent-net.rules
echo "#### UDEV Rules (/etc/udev/rules.d/70-persistent-net.rules) ####"
cat /etc/udev/rules.d/70-persistent-net.rules
vagrant_interface_rule



    # Run Any Platform Specific Code and Apply the interface Re-map
    #   (may or may not perform a reboot depending on platform)
    device.vm.provision :shell , :inline => $script

end

  ##### DEFINE VM for router2 #####
  config.vm.define "router2" do |device|
    
    device.vm.hostname = "router2"
    
    device.vm.box = "CumulusCommunity/cumulus-vx"
    device.vm.box_version = "5.10.0"

    device.vm.provider :libvirt do |v|
      v.memory = 2048

    end
    #   see note here: https://github.com/pradels/vagrant-libvirt#synced-folders
    device.vm.synced_folder ".", "/vagrant", disabled: true



    # NETWORK INTERFACES
      # link for swp7 --> router1:swp6
      device.vm.network "private_network",
            :mac => "48:b0:2d:0a:f9:95",
            :libvirt__tunnel_type => 'udp',
            :libvirt__tunnel_local_ip => '127.0.0.1',
            :libvirt__tunnel_local_port => "#{ 9001 + offset }",
            :libvirt__tunnel_ip => '127.0.0.1',
            :libvirt__tunnel_port => "#{ 8001 + offset }",
            :libvirt__iface_name => 'swp7',
            auto_config: false



    # Copy over Topology.dot File
    device.vm.provision "file", source: "topology_converter/topology.dot", destination: "~/topology.dot"
    device.vm.provision :shell, privileged: false, inline: "sudo mv ~/topology.dot /etc/ptm.d/topology.dot"



    # Install Rules for the interface re-map
    device.vm.provision :shell , :inline => <<-delete_udev_directory
if [ -d "/etc/udev/rules.d/70-persistent-net.rules" ]; then
    rm -rfv /etc/udev/rules.d/70-persistent-net.rules &> /dev/null
fi
rm -rfv /etc/udev/rules.d/70-persistent-net.rules &> /dev/null
delete_udev_directory

device.vm.provision :shell , :inline => <<-udev_rule
echo "  INFO: Adding UDEV Rule: 48:b0:2d:0a:f9:95 --> swp7"
echo 'ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="48:b0:2d:0a:f9:95", NAME="swp7", SUBSYSTEMS=="pci"' >> /etc/udev/rules.d/70-persistent-net.rules
udev_rule
     
      device.vm.provision :shell , :inline => <<-vagrant_interface_rule
echo "  INFO: Adding UDEV Rule: Vagrant interface = eth0"
echo 'ACTION=="add", SUBSYSTEM=="net", ATTR{ifindex}=="2", NAME="eth0", SUBSYSTEMS=="pci"' >> /etc/udev/rules.d/70-persistent-net.rules
echo "#### UDEV Rules (/etc/udev/rules.d/70-persistent-net.rules) ####"
cat /etc/udev/rules.d/70-persistent-net.rules
vagrant_interface_rule



    # Run Any Platform Specific Code and Apply the interface Re-map
    #   (may or may not perform a reboot depending on platform)
    device.vm.provision :shell , :inline => $script

end



end

The other small improvement we can certainly make is avoiding to template out the provisioning script for each device in the Vagrantfile.

1 reply

ipspace Oct 29, 2024
Maintainer

Adding the mock interfaces works. But i am just thinking about scalability. My lab has about 80 devices and each device has about 40 ports. Just worried that my topology file is going to become too huge to manage pretty quick.

That's why I'll write that plugin to make it easier to do what you want to do. We can only hope that nothing else (Vagrant or KVM) will break.

While using vanilla Vagrant, I notice that it manages to create only interfaces which are specified. Below is the vagrant file maybe that helps figure out what we need to do to make it work in networklab?

What they're doing is exactly what I described as option #1 in the beginning of our discussion -- remapping (for example) interface#1 into swp6. However, that seemingly requires a reboot.

I will definitely keep the information you provided handy as it might solve other issues, but I don't think I'm ready to implement the "interface renaming" hack in netlab. It would be applicable to a very small subset of supported platforms, so why bother.

ipspace · 2024-11-01T09:21:32Z

ipspace
Nov 1, 2024
Maintainer

Implemented in cba50fb, documentation in https://github.com/ipspace/netlab/blob/cba50fbcc4afb752b5bffeaf5fffb00a6c798124/docs/links.md#changing-interface-names

You can try it out in a local copy of the GitHub repo (https://netlab.tools/install/clone/) or wait for the next release (coming in early November)

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cumulus Point to point links with switch port specified #1452

{{title}}

Replies: 6 comments 7 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Cumulus Point to point links with switch port specified #1452

SudarshanVK Oct 28, 2024

Replies: 6 comments · 7 replies

SudarshanVK Oct 28, 2024 Author

ipspace Oct 28, 2024 Maintainer

ipspace Oct 28, 2024 Maintainer

SudarshanVK Oct 28, 2024 Author

ipspace Oct 28, 2024 Maintainer

SudarshanVK Oct 28, 2024 Author

SudarshanVK Oct 28, 2024 Author

ipspace Oct 28, 2024 Maintainer

SudarshanVK Oct 28, 2024 Author

SudarshanVK Oct 28, 2024 Author

SudarshanVK Oct 28, 2024 Author

ipspace Oct 29, 2024 Maintainer

ipspace Nov 1, 2024 Maintainer

SudarshanVK
Oct 28, 2024

Replies: 6 comments 7 replies

SudarshanVK
Oct 28, 2024
Author

ipspace Oct 28, 2024
Maintainer

ipspace
Oct 28, 2024
Maintainer

SudarshanVK
Oct 28, 2024
Author

ipspace Oct 28, 2024
Maintainer

SudarshanVK Oct 28, 2024
Author

SudarshanVK Oct 28, 2024
Author

ipspace Oct 28, 2024
Maintainer

SudarshanVK Oct 28, 2024
Author

SudarshanVK
Oct 28, 2024
Author

SudarshanVK
Oct 28, 2024
Author

ipspace Oct 29, 2024
Maintainer

ipspace
Nov 1, 2024
Maintainer