microcloud init: failed to bind to any multicast udp port #254

myr4htw · 2023-12-19T13:51:46Z

Hello,
I try to set up MicroCloud with 3 virtual machines. Every machine has 2 network interfaces: one is assigned an IP address, the other is without IP address:
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1a:4a:16:01:96 brd ff:ff:ff:ff:ff:ff
altname enp0s3
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1a:4a:16:01:98 brd ff:ff:ff:ff:ff:ff
altname enp0s9
inet 10.216.50.200/24 brd 10.216.50.255 scope global noprefixroute ens9
valid_lft forever preferred_lft forever

What's going wrong?
Kind regards
Margit

roosterfish · 2024-01-02T08:45:53Z

Hi @myr4htw, can you please provide some more context and reproducer steps. At which point of the microcloud init process do you see this error?

myr4htw · 2024-01-03T12:49:29Z

Hi,
I am now trying microcloud init with the following preseed file:

lookup_subnet: 10.216.50.0/24
systems:

name: stl-s-microcl1
ovn_uplink_interface: ens3
disks:
local:
- path: /dev/sdb
  wipe: false
name: stl-s-microcl2
ovn_uplink_interface: ens3
disks:
local:
- path: /dev/sdb
  wipe: false
name: stl-s-microcl3
ovn_uplink_interface: ens3
disks:
local:
- path: /dev/sdb
  wipe: false
  ovn:
  ipv4_gateway: 10.216.50.1/24
  ipv4_range: 10.216.50.200-10.216.50.204
  network:
  ethernets:
  ens3:
  dhcp4: false
  dhcp6: false
  ens9:
  addresses:
  - 10.216.50.0/24
  dhcp4: false
  dhcp6: false

I receive the error
"Scanning for eligible servers ...
Error: Failed lookup: write udp4 0.0.0.0:50381->224.0.0.251:5353: sendto: network is unreachable"

But multicast seems to be ok!?

Please see the output of the commands netstat -gn and nc:

root@stl-s-microcl1:~# netstat -gn
IPv6/IPv4-Gruppenmitgliedschaften
Schnittstelle RefZäh Grupp

lo 1 224.0.0.251
lo 1 224.0.0.1
ens3 1 224.0.0.1
ens9 2 224.0.0.251
ens9 1 224.0.0.1
lo 1 ff02::fb
lo 1 ff02::1
lo 1 ff01::1
ens3 1 ff02::1
ens3 1 ff01::1
ens9 2 ff02::fb
ens9 1 ff02::1:fff8:8ae0
ens9 1 ff02::1
ens9 1 ff01::1
.... same result on second server (microcl2) and third server (microcl3)

Test connection to the second server (microcl2):
root@stl-s-microcl1:~# nc -u -v 10.216.50.202 5353
Connection to 10.216.50.202 5353 port [udp/mdns] succeeded!

Test connection to the third server (microcl3):
root@stl-s-microcl1:~# nc -u -v 10.216.50.204 5353
Connection to 10.216.50.204 5353 port [udp/mdns] succeeded!

Hope that helps.

Kind regards
Margit

roosterfish · 2024-01-04T15:24:39Z

Make sure multicast is enabled on your network. MicroCloud uses mDNS for discovery and in your case tries to send to the multicast address 224.0.0.251:5353 to discover it's peers. See the comment about cloud providers here: https://canonical-microcloud.readthedocs-hosted.com/en/latest/explanation/initialisation/#automatic-server-detection.

There is also this issue which looks to be the same #134.

myr4htw · 2024-01-11T13:51:19Z

Hi,
problem is solved. It was a network issue We had to add a router for our internal network.
Now setup is complete, but I can't ping the virtual router.
Network config of our servers looks strange with the interfaces in state down.....

micro1@stl-s-microcl1:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1a:4a:16:01:96 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 10.216.50.200/24 brd 10.216.50.255 scope global noprefixroute ens3
valid_lft forever preferred_lft forever
inet6 fe80::e42:9822:698:6b55/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 00:1a:4a:16:01:98 brd ff:ff:ff:ff:ff:ff
altname enp0s9
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 4a:e1:c9:bb:5a:79 brd ff:ff:ff:ff:ff:ff
5: lxdovn1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:1a:4a:16:01:98 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 6a:21:1e:63:93:6c brd ff:ff:ff:ff:ff:ff
7: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether ba:e9:58:06:65:cf brd ff:ff:ff:ff:ff:ff
inet6 fe80::7857:e1ff:fe4a:618b/64 scope link
valid_lft forever preferred_lft forever

Do you have any ideas?

Kind regards
Margit

roosterfish · 2024-01-11T15:11:13Z

I guess you are trying to ping the OVN virtual router from one of the LXD networks e.g. default? From the MicroCloud cluster nodes there is no route to this network. You can confirm this by getting the output of ip r on one of the cluster nodes and check that the LXD networks range (ipv4.address) from lxc network show default isn't in there.

Can you reach the virtual router/gateway from an instance connected to the respective LXD network?

lxc launch ubuntu:jammy c1 -n {network}
lxc exec c1 -- ping {gateway}

myr4htw · 2024-01-15T07:41:06Z

Hi,
unfortunately I don't understand your statement :-(
Please see my config. Does this look correct?

Kind regards
Margit
micro1@stl-s-microcl1:/etc/default$ ip r
default via 10.216.50.152 dev ens3 proto static metric 20100
10.216.50.0/24 dev ens3 proto kernel scope link src 10.216.50.200 metric 100
169.254.0.0/16 dev ens3 scope link metric 1000
micro1@stl-s-microcl1:/etc/default$ lxc network show default
config:
bridge.mtu: "1442"
ipv4.address: 10.183.27.1/24
ipv4.nat: "true"
ipv6.address: fd42:a418:68ab:5d4d::1/64
ipv6.nat: "true"
network: UPLINK
volatile.network.ipv4.address: 134.96.216.206
description: ""
name: default
type: ovn
used_by:

/1.0/profiles/default
managed: true
status: Created
locations:
stl-s-microcl1
stl-s-microcl2
stl-s-microcl3

myr4htw · 2024-01-15T08:05:19Z

Could you please give me a description how I can test the network and how to see if it is ok or not?
Is there any documentation describing the relations?

roosterfish · 2024-01-15T09:26:34Z

In the output of ip r you can see there is no route to reach the virtual network 10.183.27.0/24 (default) in which 10.183.27.1/24 is the gateway as seen from LXD instances. Egress traffic from this network has the source IP 134.96.216.206 which comes from the range of addresses that you have specified during MicroCloud installation.

Now if you want to ping the public facing side of the virtual network you could try to ping 134.96.216.206. Make sure the network this address resides in is properly routed in your infrastructure.

From your messages I still don't see the exact error you are facing, please elaborate on this.

myr4htw · 2024-01-15T11:14:09Z

Hi,

134.96.216.206 is not reachable by ping.
I think the main problem is that the MicroCloud is not reachable - neither internally nor from the internet.
I will try to tell you the steps we have taken to build the MicroCloud. Perhaps you then see a mistake.

We've got 3 ubuntu servers. These are virtual machines in our RedHat Virtualization.
Every server has 2 network interfaces. One interface (ens3) has an IP (10.216.50.200, 10.216.50.202, 10.216.50.204). The second interface has no IP (ens9), it is connected to the network 134.96.216.0 which is our network with internet connection.

micro1@stl-s-microcl1:/etc/default$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:1a:4a:16:01:96 brd ff:ff:ff:ff:ff:ff
altname enp0s3
inet 10.216.50.200/24 brd 10.216.50.255 scope global noprefixroute ens3
valid_lft forever preferred_lft forever
inet6 fe80::e42:9822:698:6b55/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: ens9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master ovs-system state UP group default qlen 1000
link/ether 00:1a:4a:16:01:98 brd ff:ff:ff:ff:ff:ff
altname enp0s9
4: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 4a:e1:c9:bb:5a:79 brd ff:ff:ff:ff:ff:ff
5: lxdovn1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:1a:4a:16:01:98 brd ff:ff:ff:ff:ff:ff
6: br-int: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 6a:21:1e:63:93:6c brd ff:ff:ff:ff:ff:ff
7: genev_sys_6081: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65000 qdisc noqueue master ovs-system state UNKNOWN group default qlen 1000
link/ether ba:e9:58:06:65:cf brd ff:ff:ff:ff:ff:ff
inet6 fe80::7857:e1ff:fe4a:618b/64 scope link
valid_lft forever preferred_lft forever

In the MicroCloud init process we selected the addresses 10.216.50.200 - 204 for internal traffic
and the interfaces ens9 for external connectivity.
IPv4 gateway: 134.96.216.200/24
first IP address in the range... 134.96.216.206
last IP address in the range... 134.96.216.208

Then the cluster was built successfully.
Perhaps these informations tell you what is going wrong or what I still have to do to get a working MicroCloud.

Kind regards
Margit

roosterfish · 2024-01-15T12:12:16Z

Okay so from what I can see this configuration looks ok.

Maybe let's first check it the other way around. When you deploy a LXD instance within the MicroCloud, can it reach the internet and/or gateway?

lxc launch ubuntu:jammy c1
lxc exec c1 -- bash -c "apt update && apt install -y traceroute && traceroute 1.1.1.1"

This should print something like this:

traceroute to 1.1.1.1 (1.1.1.1), 30 hops max, 60 byte packets
 1  _gateway ({gateway of your LXD default network})  0.787 ms  0.822 ms  0.839 ms
 2  _gateway.lxd (134.96.216.200)  2.163 ms  2.164 ms  2.161 ms
...
10  one.one.one.one (1.1.1.1)  28.644 ms  23.253 ms  23.247 ms

myr4htw · 2024-01-15T13:09:08Z

Hi,
creation of an instance doesn't work, please see:

micro1@stl-s-microcl1:~$ lxc launch ubuntu:jammy c1
Creating c1
Error: Failed instance creation: Failed getting image: Failed parsing stream: Get "https://cloud-images.ubuntu.com/releases/streams/v1/index.json": lookup cloud-images.ubuntu.com on 127.0.0.53:53: server misbehaving

roosterfish · 2024-01-15T13:23:11Z

This looks to be an issue related to domain name resolution on your machine and not LXD.
What happens when you nslookup cloud-images.ubuntu.com?

myr4htw · 2024-01-15T13:32:32Z

root@stl-s-microcl1:~# nslookup cloud-images.ubuntu.com
Server: 127.0.0.53
Address: 127.0.0.53#53

** server can't find cloud-images.ubuntu.com: SERVFAIL
--> ok, there is no nameserver on localhost.
And when I add our nameserver (e.g. 134.96.216.214) to /etc/resolv.conf it cannot be contacted ---> no internet connection!
--> I am moving in a circle .... ???

myr4htw · 2024-01-16T11:51:22Z

Hello,
perhaps I can ask my question like that:
Which network / routing requirements must be met in order MicroCloud to work?
We' d like to use network 10.216.50.0/24 (internal network without internet connection) for internal communication and addresses from network 134.96.216.0/24 (this network has internet connection) for the uplink network.

Kind regards
Margit

roosterfish · 2024-01-18T08:19:10Z

You can find the networking requirements listed on this page https://canonical-microcloud.readthedocs-hosted.com/en/latest/explanation/microcloud/#explanation-networking.

Have you managed to get DNS working?

myr4htw · 2024-01-18T08:21:56Z

No, the problem is the lack of internet connection...

roosterfish · 2024-01-18T08:30:40Z

When launching an instance, downloading the image would be done through 10.216.50.0/24 and not the OVN network. Can you nslookup cloud-images.ubuntu.com from the host where LXD/MicroCloud is running on?

myr4htw · 2024-01-18T08:42:14Z

No, I can't.
I have no internet connection from network 10.216.50.1 - I think that is the main problem.
I thought the microcloud-init process would establish the internet connection via OVN-router?

roosterfish · 2024-01-18T08:53:42Z

That seems to be a problem yes. Please check the link I have posted earlier. For the creation of the MicroCloud/LXD cluster and the download of images the first network interface is used.

The second network interface is dedicated for connecting OVN to the uplink network. That is egress traffic originating from LXD instances deployed in MicroCloud.

myr4htw · 2024-01-18T09:45:27Z

The network the first interfaces belongs to has no internet connection. My understanding was it is only used for the communication between the MicroCloud Servers. That is why we chose this internal network.
What does it mean now? What do we have to do to solve this problem?

roosterfish · 2024-11-15T14:10:10Z

Hi @myr4htw, are you still facing this issue?
We have just released a new version of MicroCloud using a slightly different mechanism for multicast disocovery. See the release post here.

roosterfish added the Incomplete Waiting on more information from reporter label Jan 2, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

microcloud init: failed to bind to any multicast udp port #254

microcloud init: failed to bind to any multicast udp port #254

myr4htw commented Dec 19, 2023

roosterfish commented Jan 2, 2024 •

edited

Loading

myr4htw commented Jan 3, 2024 •

edited

Loading

roosterfish commented Jan 4, 2024 •

edited

Loading

myr4htw commented Jan 11, 2024

roosterfish commented Jan 11, 2024

myr4htw commented Jan 15, 2024

myr4htw commented Jan 15, 2024

roosterfish commented Jan 15, 2024

myr4htw commented Jan 15, 2024

roosterfish commented Jan 15, 2024

myr4htw commented Jan 15, 2024

roosterfish commented Jan 15, 2024

myr4htw commented Jan 15, 2024

myr4htw commented Jan 16, 2024

roosterfish commented Jan 18, 2024 •

edited

Loading

myr4htw commented Jan 18, 2024

roosterfish commented Jan 18, 2024

myr4htw commented Jan 18, 2024

roosterfish commented Jan 18, 2024

myr4htw commented Jan 18, 2024

roosterfish commented Nov 15, 2024

microcloud init: failed to bind to any multicast udp port #254

microcloud init: failed to bind to any multicast udp port #254

Comments

myr4htw commented Dec 19, 2023

roosterfish commented Jan 2, 2024 • edited Loading

myr4htw commented Jan 3, 2024 • edited Loading

roosterfish commented Jan 4, 2024 • edited Loading

myr4htw commented Jan 11, 2024

roosterfish commented Jan 11, 2024

myr4htw commented Jan 15, 2024

myr4htw commented Jan 15, 2024

roosterfish commented Jan 15, 2024

myr4htw commented Jan 15, 2024

roosterfish commented Jan 15, 2024

myr4htw commented Jan 15, 2024

roosterfish commented Jan 15, 2024

myr4htw commented Jan 15, 2024

myr4htw commented Jan 16, 2024

roosterfish commented Jan 18, 2024 • edited Loading

myr4htw commented Jan 18, 2024

roosterfish commented Jan 18, 2024

myr4htw commented Jan 18, 2024

roosterfish commented Jan 18, 2024

myr4htw commented Jan 18, 2024

roosterfish commented Nov 15, 2024

roosterfish commented Jan 2, 2024 •

edited

Loading

myr4htw commented Jan 3, 2024 •

edited

Loading

roosterfish commented Jan 4, 2024 •

edited

Loading

roosterfish commented Jan 18, 2024 •

edited

Loading