Raspberry Pi 5 network boot using IPv6 does not respond to Neighbor Solicitation packets.

[MiddelkoopT]

Describe the bug

During the IPv6 network boot the bootloader does not respond to IPv6 "neighbor solicitation" (NS) packets. This blocks the boot process as dnsmasq (and radvd) needs know the MAC address of the Pi 5 before it can send the "neighbor advertisement" (NA) packet with the MAC address of the Pi 5. If the address of the Pi 5 is manually added to the kernel of the dnsmasq server the the boot process proceeds as expected.

Here is the relevant tcpdump trace of the failed boot (full tcpdump attached). Note the repeating "neighbor solicitation" (NS) packets that are not responded to. The dnsmasq machine is at fe80::da3a:ddff:fe47:a110 the target is at d8:3a:dd:98:09:77.

23:45:08.861745 IP6 (class 0xc0, flowlabel 0x40337, hlim 255, next-header ICMPv6 (58) payload length: 88) fe80::da3a:ddff:fe47:a110 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 88
        hop limit 64, Flags [managed, other stateful], pref medium, router lifetime 1800s, reachable time 0ms, retrans timer 0ms
          prefix info option (3), length 32 (4): fd00:400:3::/64, Flags [onlink], valid time 1800s, pref. time 1800s
            0x0000:  4080 0000 0708 0000 0708 0000 0000 fd00
            0x0010:  0400 0003 0000 0000 0000 0000 0000
          mtu option (5), length 8 (1):  1500
            0x0000:  0000 0000 05dc
          source link-address option (1), length 8 (1): d8:3a:dd:47:a1:10
            0x0000:  d83a dd47 a110
          rdnss option (25), length 24 (3):  lifetime 1800s, addr: fe80::da3a:ddff:fe47:a110
            0x0000:  0000 0000 0708 fe80 0000 0000 0000 da3a
            0x0010:  ddff fe47 a110
23:45:45.046490 IP6 (flowlabel 0x360c7, hlim 1, next-header UDP (17) payload length: 94) fe80::da3a:ddff:fe47:a110.546 > ff02::1:2.547: [bad udp cksum 0x5589 -> 0x4c85!] dhcp6 solicit (xid=c313d0 (rapid-commit) (IA_NA IAID:4174257057 T1:0 T2:0) (IA_PD IAID:4174257057 T1:0 T2:0) (Client-FQDN) (option-request DNS-server SNTP-servers NTP-server opt_82) (client-ID vid 0000ab112e525167) (elapsed-time 65535))
23:46:11.073298 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::da3a:ddff:fe98:977 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): d8:3a:dd:98:09:77
            0x0000:  d83a dd98 0977
23:46:11.074030 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::da3a:ddff:fe47:a110 > ff02::1:ff98:977: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::da3a:ddff:fe98:977
          source link-address option (1), length 8 (1): d8:3a:dd:47:a1:10
            0x0000:  d83a dd47 a110
23:46:12.085106 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::da3a:ddff:fe47:a110 > ff02::1:ff98:977: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::da3a:ddff:fe98:977
          source link-address option (1), length 8 (1): d8:3a:dd:47:a1:10
            0x0000:  d83a dd47 a110
23:46:13.109093 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) fe80::da3a:ddff:fe47:a110 > ff02::1:ff98:977: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::da3a:ddff:fe98:977
          source link-address option (1), length 8 (1): d8:3a:dd:47:a1:10
            0x0000:  d83a dd47 a110
23:46:15.073299 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 16) fe80::da3a:ddff:fe98:977 > ff02::2: [icmp6 sum ok] ICMP6, router solicitation, length 16
          source link-address option (1), length 8 (1): d8:3a:dd:98:09:77
            0x0000:  d83a dd98 0977

The workaround to add the MAC address to the Pi server kernel is as follows

ip neigh replace fe80::da3a:ddff:fe98:977 dev eth0 lladdr d8:3a:dd:98:09:77

Steps to reproduce the behaviour

Setup a Pi server running bookworm with the networking on wlan0 and install dnsmasq (be sure to disable it in systemd). Connect the Pi5 and the Pi server on an empty switch (or use a direct connection).

As root and with the following dnsmasq.conf

## DHCPv6
interface=eth0
dhcp-range=fd00:0400:3::2:01,fd00:0400:3::2:FF,600s
dhcp-option=option6:bootfile-url,tftp://[fd00:400:3::8]/
#dhcp-option=option6:ntp-server,[fd00:400:3::1]
dhcp-leasefile=./dnsmasq.leases

## RA v6
enable-ra

## tftp
enable-tftp
tftp-root=/data

## run forground to stdout
#keep-in-foreground
no-daemon
log-facility=-

Run dnsmasq server with

dnsmasq --conf-file=dnsmasq.conf | tee 1-pi5-no-na-dnsmasq.txt

On another terminal on the Pi server run

tcpdump -n -vvv -i eth0 not port 22 |tee 1-pi5-no-na-tcpdump.txt

Device (s)

Raspberry Pi 5

Bootloader configuration.

[all]
BOOT_UART=1
POWER_OFF_ON_HALT=1
BOOT_ORDER=0xf412
USE_IPV6=1

System

root@k51:~# rpi-eeprom-config 
[all]
BOOT_UART=1
POWER_OFF_ON_HALT=1
BOOT_ORDER=0xf412
USE_IPV6=1
root@k51:~# vcgencmd bootloader_version
2024/02/16 15:28:41
version 4c845bd37c8a7c7ff79173cdc50dd3facf63996f (release)
timestamp 1708097321
update-time 1709973075
capabilities 0x0000007f

root@k51:~# vcgencmd version
2024/02/16 15:28:41 
Copyright (c) 2012 Broadcom
version 4c845bd3 (release) (embedded)
root@k51:~# uname -a 
Linux k51 6.6.20+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux

Bootloader logs

RPi: BOOTSYS release VERSION:4c845bd3 DATE: 2024/02/16 TIME: 15:28:41
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1708097321 serial be512421 boardrev d04170 stc 1622082
AON_RESET: 00000003 PM_RSTS 00001000
RP1_BOOT chip ID: 0x20001927
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
PMIC reset-event 00000000 rtc 00000001 alarm 00000000 enabled 0
uSD voltage 3.3V
Initialising SDRAM 'Micron' 32Gb x2 total-size: 64 Gbit 4267
DDR 4267 1 0 64 152
RP1_BOOT chip ID: 0x20001927

RP1_BOOT chip ID: 0x20001927
RP1_BOOT: fw size 25968
PCI2 init
PCI2 reset
PCIe scan 00001de4:00000001
RP1_CHIP_INFO 20001927

RPi: BOOTLOADER release VERSION:4c845bd3 DATE: 2024/02/16 TIME: 15:28:41
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1708097321 serial be512421 boardrev d04170 stc 4591582
AON_RESET: 00000003 PM_RSTS 00001000
usb_pd_init status 1
XHCI-STOP
xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
USBSTS 1
xHC0 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
xHC0 ports 3 slots 64 intrs 4
XHCI-STOP
xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
USBSTS 1
xHC1 ver: 272 HCS: 03000440 140000f1 07ff000a HCC: 0240fe6d
xHC1 ports 3 slots 64 intrs 4
Boot mode: NETWORK (02) order f41
NET_BOOT: d8:3a:dd:98:09:77 wait for link TFTP6: (null)
LINK STATUS: speed: 1000 full duplex
Link ready
RX6: 0 IP: 0 MAC: 0 ICMP: 0/0 UDP: 0/0 ICMP_CSUM_ERR: 0 UDP_CSUM_ERR: 0
Stopping network
RX6: 0 IP: 0 MAC: 0 ICMP: 0/0 UDP: 0/0 ICMP_CSUM_ERR: 0 UDP_CSUM_ERR: 0
RP1_NET NETWORK_CONTROL : 0x0000001c
RP1_NET NETWORK_CONFIG : 0x00d40502
RP1_NET NETWORK_STATUS : 0x00000006
RP1_NET DMA_CONFIG : 0x4020071f
RP1_NET TRANSMIT_STATUS : 0x00000021
RP1_NET RECEIVE_STATUS : 0x00000000
RP1_NET INT_STATUS : 0x00000088
RP1_NET FRAMES_TXED_64 : 0x00000000
RP1_NET FRAMES_TXED_65 : 0x00000006
RP1_NET FRAMES_TXED_128 : 0x00000000
RP1_NET FRAMES_TXED_256 : 0x00000000
RP1_NET FRAMES_TXED_512 : 0x00000000
RP1_NET FRAMES_TXED_1024 : 0x00000000
RP1_NET FRAMES_TXED_1519 : 0x00000000
RP1_NET TX_UNDERRUNS : 0x00000000
RP1_NET FRAMES_RXED_OK : 0x00000000
RP1_NET BROADCAST_RXED : 0x00000000
RP1_NET MULTICAST_RXED : 0x00000000
RP1_NET FRAMES_RXED_64 : 0x00000000
RP1_NET FRAMES_RXED_65 : 0x00000000
RP1_NET FRAMES_RXED_128 : 0x00000000
RP1_NET FRAMES_RXED_256 : 0x00000000
RP1_NET FRAMES_RXED_512 : 0x00000000
RP1_NET FRAMES_RXED_1024 : 0x00000000
RP1_NET FRAMES_RXED_1519 : 0x00000000
RP1_NET FCS_ERRORS : 0x00000000
RP1_NET RX_RESOURCE_ERRORS : 0x00000000
RP1_NET RX_OVERRUNS : 0x00000000
RP1_NET TRANSMIT_Q_PTR : 0x3c320920
RP1_NET RECEIVE_Q_PTR : 0x3c3203c0
RP1_NET tx_q_base : 0xfc3208c0
RP1_NET rx_q_base : 0xfc3203c0
RP1_NET rx dropped : 0
RX6: 0 IP: 0 MAC: 0 ICMP: 0/0 UDP: 0/0 ICMP_CSUM_ERR: 0 UDP_CSUM_ERR: 0
NETBOOT CANCEL
NETBOOT init failed

USB boot

No response

NVMe boot

No response

Network (TFTP boot)

0-commands.txt

Failed boot
1-pi5-no-na-serial.txt
1-pi5-no-na-tcpdump.txt
dnsmasq.conf.txt

Successful boot with workaround
5-pi5-static-serial.txt
5-pi5-static-tcpdump.txt