Redeploying containers with macvlan + static IP does not inform of the new mac address

[jerome59]

Issue Description

Hello,

Let's assume that we have a macvlan network and in that network we have a container with a static IP.
When redeploying the container (rm + run), the new container has the same IP as before but a different mac address, which is the expected behavior.

The problem is that the new container does not inform the network that the mac address has changed, so the switches don't know that they need to update their routing table.
The switches still keep in memory the old couple IP/mac address which is no longer valid.
On a small setup (only one switch between the different hardware), it's not a problem because the switch quickly detects that the mac address is no longer reachable and updates its table.
On a larger setup (distributed network), this becomes a problem. All the switches have to learn the new MAC address, but since they have not been notified of the change, it takes time (on our setup we see delays of up to 30 minutes before the container becomes pingable from anywhere else on the network).

This problem can be solved by creating a Gratuitous ARP request (https://wiki.wireshark.org/Gratuitous_ARP).
Let's say my container has an IP 200.200.200.252, if I enter it (podman exec) and run arping -U -I eth0 200.200.200.252, then an ARP request is sent out on the network and all the switches update their table. My container will then be directly pingable from any node on our network.

I guess that problem can also happen with ipvlan if we undeploy a container from a hardware and tries to deploy it on another hardware with the same IP.

Steps to reproduce the issue

Steps to reproduce the issue

1. Create a macvlan network sudo podman network create --driver=macvlan --subnet=200.200.200.0/22 --opt parent=eno1 testnetwork
2. Start a container sudo podman run --name=test_container --replace --rm --network=testnetwork --sdnotify=conmon -d --ip 200.200.200.252 --privileged httpd:2.4
3. From another hardware on a distributed network, try to ping the container ping 200.200.200.252 -> It should work as it is the first deployment
4. Redeploy the container sudo podman run --name=test_container --replace --rm --network=testnetwork --sdnotify=conmon -d --ip 200.200.200.252 --privileged httpd:2.4
5. From another hardware on a distributed network, try to ping the container ping 200.200.200.252 -> it should failed in a timeout as the switches haven't updated their routing table
6. Enter the container sudo podman exec -it test_container /bin/bash, run apt update && apt install arping then generate an ARP Gratuitous request arping -U -I eth0 200.200.200.252
7. From another hardware on a distributed network, try to ping the container ping 200.200.200.252 -> It should work as the switches have updated their table due to the ARP Gratuitous request

Describe the results you received

When redeploying a container in a macvlan network with a static IP, the new container has a new MAC address associated to the same IP.
That container does not send an ARP Gratuitous request on the network (confirmed with wireshark by capturing the traffic on my ethernet interface) so the switches are not informed that they need to update their routing table.

Describe the results you expected

I was expecting that when redeploying a container in a macvlan network, the container will generate an ARP Gratuitous request in order to inform the switches of the new couple IP / MAC address.

podman info output

host:
arch: amd64
buildahVersion: 1.37.4
cgroupControllers:

• cpuset
• cpu
• io
• memory
• hugetlb
• pids
• rdma
• misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
  package: conmon_2.1.6+ds1-1_amd64
  path: /usr/bin/conmon
  version: 'conmon version 2.1.6, commit: unknown'
  cpuUtilization:
  idlePercent: 98.78
  systemPercent: 0.27
  userPercent: 0.95
  cpus: 12
  databaseBackend: boltdb
  distribution:
  codename: trixie
  distribution: debian
  version: unknown
  eventLogger: journald
  freeLocks: 2021
  hostname: orion-docker
  idMappings:
  gidmap: null
  uidmap: null
  kernel: 6.1.0-13-rt-amd64
  linkmode: dynamic
  logDriver: journald
  memFree: 23622979584
  memTotal: 33438621696
  networkBackend: netavark
  networkBackendInfo:
  backend: netavark
  dns:
  package: aardvark-dns_1.12.2-1_amd64
  path: /usr/lib/podman/aardvark-dns
  version: aardvark-dns 1.12.2
  package: netavark_1.12.1-3_amd64
  path: /usr/lib/podman/netavark
  version: netavark 1.12.1
  ociRuntime:
  name: runc
  package: runc_1.1.5+ds1-1+deb12u1_amd64
  path: /usr/bin/runc
  version: |-
  runc version 1.1.5+ds1
  commit: 1.1.5+ds1-1+deb12u1
  spec: 1.0.2-dev
  go: go1.19.8
  libseccomp: 2.5.4
  os: linux
  pasta:
  executable: /usr/bin/pasta
  package: passt_0.0git20240326.4988e2b-1_amd64
  version: |
  pasta 0.0git20240326.4988e2b-1
  Copyright Red Hat
  GNU General Public License, version 2 or later
  https://www.gnu.org/licenses/old-licenses/gpl-2.0.html
  This is free software: you are free to change and redistribute it.
  There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
  exists: true
  path: /run/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
  apparmorEnabled: true
  capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
  rootless: false
  seccompEnabled: true
  seccompProfilePath: /usr/share/containers/seccomp.json
  selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
  executable: /usr/bin/slirp4netns
  package: slirp4netns_1.2.0-1_amd64
  version: |-
  slirp4netns version 1.2.0
  commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
  libslirp: 4.7.0
  SLIRP_CONFIG_VERSION_MAX: 4
  libseccomp: 2.5.4
  swapFree: 1024454656
  swapTotal: 1024454656
  uptime: 5h 37m 28.00s (Approximately 0.21 days)
  variant: ""
  plugins:
  authorization: null
  log:
• k8s-file
• none
• passthrough
• journald
  network:
• bridge
• macvlan
• ipvlan
  volume:
• local
  registries:
  search:
• docker.io
  store:
  configFile: /usr/share/containers/storage.conf
  containerStore:
  number: 12
  paused: 0
  running: 11
  stopped: 1
  graphDriverName: overlay
  graphOptions:
  overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 249365385216
  graphRootUsed: 70269030400
  graphStatus:
  Backing Filesystem: extfs
  Native Overlay Diff: "true"
  Supports d_type: "true"
  Supports shifting: "true"
  Supports volatile: "true"
  Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
  number: 18
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
  version:
  APIVersion: 5.2.4
  Built: 1729434149
  BuiltTime: Sun Oct 20 16:22:29 2024
  GitCommit: ""
  GoVersion: go1.23.2
  Os: linux
  OsArch: linux/amd64
  Version: 5.2.4

Podman in a container

No

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

[Luap99]

This would need to be implemented in netavark

[jerome59]

@corrosive4354 :

You can consider automatically sending Gratuitous ARP requests when the container starts. You can add this command to the startup script of the container to ensure that it is executed every time the container starts.

Yes we can indeed send the Gratuitous ARP requests ourselves at startup but I guess that's only a "fallback solution". The real solution should be in netavark and someone deploying a container should not have to think about that.

In addition, I saw that you used docker to set this thing up.

We are only using podman. In the podman info command, the docker you see in the hostname is because I reused a hardware that we wanted to use with dockers but at the end we're using podman. The docker.io is just the registry

[M1cha]

This might have to be an optional default-off feature though, because ARP solicitations can be used for attacks and could thus trigger intrusion detection systems.

[Luap99]

Looking at the code I noticed we do set the arp_notify sysctl to 1 for the veth inside the container which per kernel docs should send out a Generate gratuitous arp requests already actually.