need help on multilab issues

[nagraj-kulkarni]

Hello, i am trying to build a topo with multilab, with provider as clab. It shows the below error. I am using multilab plugin in the topology file. I do see the docker network showing a bridge "nl_mgmt_23". Tried to use a different multilab id, brought down the topo with cleanup but nothing helps. I see this errors quiet often, not sure how to address it. Please help.. Thanks..

netlab up --plugin multilab -s defaults.multilab.id=23

┌──────────────────────────────────────────────────────────────────────────────────┐
│ CREATING configuration files │
└──────────────────────────────────────────────────────────────────────────────────┘
[CREATED] provider configuration file: clab.yml
[CREATED] transformed topology dump in YAML format in netlab.snapshot.yml
[GROUPS] group_vars for all
[GROUPS] group_vars for modules
[GROUPS] group_vars for eos
[HOSTS] host_vars for r1
[HOSTS] host_vars for r2
[CREATED] minimized Ansible inventory hosts.yml
[CREATED] Ansible configuration file: ansible.cfg

┌──────────────────────────────────────────────────────────────────────────────────┐
│ CHECKING virtualization provider installation │
└──────────────────────────────────────────────────────────────────────────────────┘
[SUCCESS] clab installed and working correctly

┌──────────────────────────────────────────────────────────────────────────────────┐
│ STARTING clab nodes │
└──────────────────────────────────────────────────────────────────────────────────┘
provider clab: executing sudo -E containerlab deploy -t clab.yml
INFO[0000] Containerlab v0.54.2 started
INFO[0000] Parsing & checking topology file: clab.yml
INFO[0000] Creating docker network: Name="nl_mgmt_23", IPv4Subnet="192.168.23.0/24", IPv6Subnet="", MTU=1500
Error: bridge "ml_23_3" referenced in topology but does not exist
bridge "ml_23_4" referenced in topology but does not exist
Error executing sudo -E containerlab deploy -t clab.yml:
Command '['sudo', '-E', 'containerlab', 'deploy', '-t', 'clab.yml']' returned non-zero exit status 1.
[FATAL] netlab up: sudo -E containerlab deploy -t clab.yml failed, aborting...

[nagraj-kulkarni]

nl_mgmt_23 bridge is created but its looking for ml_bridge_23. appreciate any pointers on what to look for... Thanks..

[ipspace]

Please post your topology, that could help us figure out something. Also, please post the output of lsb_release -a.

Next, please start the lab with the -vv flag to get more details. Please do the topology transformation with netlab create first, and then start the lab with netlab up --snapshot -vv --no-config so you'll get the verbose printout just for the relevant part of the process.

Thank you!

[nagraj-kulkarni]

Sure, trying to use your example itself :)

`cat topology.yml
defaults.device: eos
provider: clab
defaults.devices.eos.clab.image: "ceos:4.31.0F"
addressing.p2p.ipv4: True
module: [ ospf ]

nodes: [ r1, r2 ]
links: [ r1-r2, r1-r2, r1, r2 ]

`

`
netlab create --plugin multilab -s defaults.multilab.id=23
[CREATED] provider configuration file: clab.yml
[CREATED] transformed topology dump in YAML format in netlab.snapshot.yml
[GROUPS] group_vars for all
[GROUPS] group_vars for modules
[GROUPS] group_vars for eos
[HOSTS] host_vars for r1
[HOSTS] host_vars for r2
[CREATED] minimized Ansible inventory hosts.yml
[CREATED] Ansible configuration file: ansible.cfg

netlab up --snapshot -vv --no-config
Read YAML data from netlab.snapshot.yml
Using transformed lab topology from snapshot file netlab.snapshot.yml

┌──────────────────────────────────────────────────────────────────────────────────┐
│ CHECKING virtualization provider installation │
└──────────────────────────────────────────────────────────────────────────────────┘
Checking virtualization provider installation
run_command executing: containerlab version
run_command executing: ['bash', '-c', "[[ containerlab version|awk '/version/ {print $2}' > '0.42' ]] && echo OK"]
[SUCCESS] clab installed and working correctly
loading netsim.providers.clab...
... found <class 'netsim.providers.clab.Containerlab'>
clab: validating node r1 image ceos:4.31.0F
run_command executing: ['docker', 'image', 'ls', '--format', 'json', 'ceos:4.31.0F']
clab: validating node r2 image ceos:4.31.0F

┌──────────────────────────────────────────────────────────────────────────────────┐
│ STARTING clab nodes │
└──────────────────────────────────────────────────────────────────────────────────┘
pre-start hook for Containerlab - create any bridges
run_command executing: ['brctl', 'show', 'ml_23_3']
Linux bridge ml_23_3 already exists, skipping
run_command executing: ['brctl', 'show', 'ml_23_4']
Linux bridge ml_23_4 already exists, skipping
provider clab: executing sudo -E containerlab deploy -t clab.yml
run_command executing: sudo -E containerlab deploy -t clab.yml
INFO[0000] Containerlab v0.54.2 started
INFO[0000] Parsing & checking topology file: clab.yml
Error: bridge "ml_23_3" referenced in topology but does not exist
bridge "ml_23_4" referenced in topology but does not exist
Error executing sudo -E containerlab deploy -t clab.yml:
Command '['sudo', '-E', 'containerlab', 'deploy', '-t', 'clab.yml']' returned non-zero exit status 1.
[FATAL] netlab up: sudo -E containerlab deploy -t clab.yml failed, aborting...
`

[ipspace]

Sure, trying to use your example itself :)

Works for me (what else did we expect ;). Please do post the output of lsb_release -a

run_command executing: ['brctl', 'show', 'ml_23_3'] Linux bridge ml_23_3 already exists, skipping

This means that the brctl show ml_23_3 command executed successfully (no error code) and that it returned some output. Could you execute it manually to see what it returns? Also execute brctl show ml_23_3 && echo OK so we can check the error code.

[nagraj-kulkarni]

here you go..

==>>>brctl show ml_23_3
bridge name bridge id STP enabled interfaces
ml_23_3 can't get info No such device

====> brctl show ml_23_3 && echo OK
bridge name bridge id STP enabled interfaces
ml_23_3 can't get info No such device
OK

====> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.6 LTS
Release: 18.04
Codename: bionic

[ipspace]

Looks like a bug in brctl to me. It should not report an error and return an OK exit code.

Not much we can do. You can try sudo apt-get update followed by sudo apt-get upgrade. Rebooting the box should also help. Finally, is there any reason you won't go for Ubuntu 20.04 or 22.04? I don't think I tested netlab on 18.04 for years.

[nagraj-kulkarni]

i am trying this on kvm, borrowed one :) its been running fine, i have put up many netlab examples..As i said, it starts trouble at times for some unknown triggers :)

[nagraj-kulkarni]

these bridge issues doesnt happen all the time, but when it starts it troubles a lot :) not sure whats the trigger point

[ipspace]

Does reload help?

[ipspace]

One more thing (if the upgrade doesn't fix the problem): do ip link show dev ml_23_3 (or whatever the bridge name is) as well as ip link show dev ml_23_3 && echo OK Maybe we can use that command for a more reliable bridge detection.

[nagraj-kulkarni]

It complains that there are no bridge ml_23_3 but it says
"run_command executing: ['brctl', 'show', 'ml_23_3']
Linux bridge ml_23_3 already exists, skipping".... thats weird

ip link show dev ml_23_3
Device "ml_23_3" does not exist.

will try update,upgrades

[ipspace]

netlab executes 'brctl show ml_23_3' command. As the command generates some output and does not return an error, netlab assumes the Linux bridge already exists.

What do the 'ip link' commands do?

[nagraj-kulkarni]

surprisingly, it worked with this topology file

cat topology.yml

provider: clab
defaults.device: eos # Change to your preferred device type
defaults.devices.eos.clab.image: "ceos:4.31.0F"
addressing.p2p.ipv4: True # Use IPv4 unnumbered P2P links
module: [ ospf ] # Change to OSPF if needed
nodes: [ r1, r2 ]
links: [ r1-r2 ]

[ipspace]

Of course, point-to-point links are implemented as vEth pairs and do not need Linux bridges.

[nagraj-kulkarni]

issue isolated. problem is happening when i include below in topology
links: [ r1-r2,r1-r2,r1,r2 ]

if i use links:[r1-r2,r1-r2], it works fine !!! i am so curious what would [r1,r2] do ?

[ipspace]

... not much. You would have one unconnected interface on each node, connected to a Linux bridge. See https://netlab.tools/links/#sample-link-formats

[nagraj-kulkarni]

ok, why its not liking that ? do you think using ovs is an option instead of linux bridge ?

[nagraj-kulkarni]

it worked well with ovs-bridge
provider: clab
defaults.providers.clab.bridge_type: ovs-bridge
defaults.device: eos # Change to your preferred device type
defaults.devices.eos.clab.image: "ceos:4.31.0F"
addressing.p2p.ipv4: True # Use IPv4 unnumbered P2P links
module: [ ospf ] # Change to OSPF if needed
nodes: [ r1, r2 ]
links: [ r1-r2,r1-r2,r1-r2,r1,r2 ]