How does multipass connect to instances?

[zenarcher007]

I have been using Multipass for a couple of years and it recently stopped working. On MacOS Sonoma 14.6.1 (M1) with Multipass 1.14.0 and qemu, I am having the same issues as this and this. I have tried debugging steps such as following the network troubleshooting guide, checking my firewall, and downgrading to 1.13.1. Rather than opening another issue on this problem, I wanted to ask for clarification that the troubleshooting guide does not explicitly provide so I can further understand this, as this could be the fault of my system.

Multipass launch times out for me. When you run a command such as multipass launch <instance>, and it waits for an SSH connection, under what interface and ip does it know to check for a connection on? Can this be configured?




I also noted from running the qemu command manually, that the VM does not obtain a 192.168.64.x address, although other attached interfaces work and the bridge100 interface is up and available (somewhat related). The relevant arguments passed to the qemu command were -nic vmnet-shared,model=virtio-net-pci,mac=52:54:00:c8:e9:89. (Edit: this seems to be the case when launching new VMs, but older VMs have a .64 address)

Thanks for the clarification (in advance). I would be happy to further debug and discuss this issue.

[ricab]

Hi @zenarcher007, I am sorry that you are facing this problem.

Multipass relies on the instance having an IP on the default interface, to connect to through SSH. Launches/starts won't complete without it. There are 3 main possible reasons for that:

1. the VM didn't manage to boot properly up to the point where it requests an IP (DHCP)
2. the VM does request an IP, but it doesn't get one

If we go a little beyond, there are similar situations where:

3. the VM actually gets an IP, but:
   1. Multipass can't find this IP
   2. Multipass finds an IP that doesn't match the one that the instance actually gets
   3. SSH doesn't function properly inside the VM, or Multipass is blocked from accessing it for some reason

People experience all these situations, with different causes, and report timeouts + state unknown in very different scenarios, so we are trying to improve the troubleshoot documentation for this and hope to make it available soon.

There are a few things that you can do to figure out which is your case. Given your observations, I am excluding firewall-related issues and others.

• look at the Multipass logs for a message mentioning NIC_RX_FILTER_CHANGED (this is QEMU specific); if you don't have it, the VM didn't even manage to get the interface up and you are in situation 1.
• otherwise (you have that message), inspect DHCP traffic on your host to find out if there are requests and replies. You can try something like sudo tcpdump -i bridge100 udp port 67 and port 68, right after starting the instance, to achieve that. Note that bridge100 won't exist until an instance starts.
  • if you don't see DHCP requests, that would point to situation 1, but it would be a rare occurrence
  • if you see a DHCP request but no reply, you are in situation 2 and the VM is "fine", but waiting for an IP
  • if you see DHCP requests and replies, you are probable in situation 3

If you're in situation 2: as you've found in the QEMU command line, this all rests on top of Apple's vmnet, which in turn relies on bootp to give out addresses. This is all a black box to us and we've unfortunately found these things to be unreliable (especially if the firewall is involved). There could be other reasons for this though, like interfering software, VPNs, or misconfiguration.

If you're in situation 1: you can try to run the QEMU command line that Multipass logs but without -serial chardev:char0 -nographic. This will bring up a QEMU window, where you should be able to see the boot output (you may need to select Serial or VGA in the menu), to find out where it is getting stuck. In practice, this is frequently caused by the VM image getting corrupted. Unfortunately, that can happen when the VM is killed without a proper shutdown, which is necessary when it previously failed to start for whatever reason. If you took a snapshot before you got into this situation, you could try to restore it. However, QEMU snapshots are stored as layers in the same image file, so they may not be enough. If you try this, please let us know if that fixes it for you.

If you are in situation 3: one thing that may help is to delete /var/db/dhcpd_leases and try again. This file may get duplicate or obsolete IPs and confuse things. It gets re-generated when the next DHCP lease is requested. If you can't delete the whole file (you may want to keep some of those leases), you can try to delete the ones that pertain to Multipass. For information, people report that a cleanup with a utility called OnyX fixes it for them. I don't personally know this utility so I am not recommending it. Reported successes might be because dhcpd_leases gets removed, but there might also be something more.

[zenarcher007]

!! I have found that disabling the MacOS firewall makes multipass function normally again. As it is probably not the best idea, however, to leave the firewall disabled, I am wondering where the communication process breaks down, and wondering what additional items I would need to allow through the firewall or any additional configuration I would need to make for multipass to function correctly. I have already added bootpd and multipassd to the firewall, but the issue persists while enabled.

Currently, when launching the VM with the firewall enabled, I do get NIC_RX_FILTER_CHANGED in the multipass logs, and I see no replies to the DHCP requests on bridge100 via tcpdump. An entry for the instance does not appear in /var/db/dhcpd_leases.

Additionally, I will note that when I add an additional hardware adapter via --network, such as en0 for WiFi, I can indeed SSH into the VM over the second interface (once I port scan to guess its IP address). When doing so, I can see that there is no interface which contains a .64.X address within the VM's ifconfig.

This would point to situation 2.

Since it is not getting a /var/db/dhdpd_leases entry, and the VM isn't acquiring an IP over bridge100, could this be a DHCP issue, perhaps?

[ricab]

Hi @zenarcher007, it is a DHCP issue caused by the macOS firewall. Surprising as it is, Apple's firewall blocks its own bootp process from giving out DHCP addresses. I excluded that from my reply because you said you had checked the firewall and I thought you were aware of this problem.

See #2387 for info on this topic. You can find a couple of commands at the top that usually fix it for most people, though they need you to be admin on your machine and you may need to repeat them after reboots.

[zenarcher007]

Thanks! I hadn't actually noticed issue #2387 before, and I had not known that allowing incoming connections via the firewall UI would not work as would running the two commands, so I suspected something other than bootpd. Perhaps this clarification could be added to the network troubleshooting page?

Since it is my own personal Mac, I added the commands:

@reboot /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd && /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd

to my root's crontab, and it seems multipass will (hopefully) forever work now!

Köszönöm!