VXLanCrossSubnet issue on v3.29

[valentin2105]

Hello,

I got an issue with Calico v3.29 and VXLanCrossSubnet :

I have a cluster (v1.31.2) with nodes in "admin" LAN et two nodes in "dmz" LAN (the ingresses nodes).

In the firewall between the LANs, due of the issue, we opened all network traffic (tcp/udp/icmp).

The issue is :

From the Ingress node, I can ping a pod on an admin node, but when I try to curl the TCP Port (of the pod), the packet doesn't reach the node.

With TCPDump, I see the ICMP, but the http packet doesn't arrive on the node.

• On the Calico Node status, all BGP is Established,
• On the Ingress node, I have the route to the node throught VXLAN, and the ping to the pod work.

My config :

  calicoNetwork:
    ipPools:
    - name: default-ipv4-ippool
      blockSize: 26
      cidr: 192.168.0.0/16
      encapsulation: VXLANCrossSubnet
      natOutgoing: Enabled
      nodeSelector: all()
    nodeAddressAutodetectionV4:
      canReach: 1.1.1.1

   bpfConnectTimeLoadBalancing: TCP                                                                                                                                                             
   bpfEnabled: false                                                                                                                                                                            
   bpfHostNetworkedNATWithoutCTLB: Enabled                                                                                                                                                      
   bpfLogLevel: ""                                                                                                                                                                              
   floatingIPs: Disabled                                                                                                                                                                        
   healthPort: 9099                                                                                                                                                                             
   logSeverityScreen: Info                                                                                                                                                                      
   nftablesMode: Disabled                                                                                                                                                                       
   reportingInterval: 0s                                                                                                                                                                        
   vxlanVNI: 4096  

On my ingress node :

root@ingress01-k8s:~# ip route |grep 192.168.3
192.168.3.64/26 via 192.168.3.65 dev vxlan.calico onlink 

root@ingress01-k8s:~# ping 192.168.3.65 -c 1
PING 192.168.3.65 (192.168.3.65) 56(84) bytes of data.
64 bytes from 192.168.3.65: icmp_seq=1 ttl=64 time=0.558 ms

root@ingress01-k8s:~# ping 192.168.3.79 -c 1  ( my POD)
PING 192.168.3.79 (192.168.3.79) 56(84) bytes of data.
64 bytes from 192.168.3.79: icmp_seq=1 ttl=63 time=0.437 ms

root@ingress01-k8s:~# curl -v 192.168.3.79:8080
*   Trying 192.168.3.79:8080...
(nothing happen)

------ FROM another worker on the "admin" LAN : 

root@master01-k8s:~# curl  192.168.3.79:8080 
default backend - 404

The really strange thing, is that I have the same setup on a v1.30 cluster with Calico v3.28.1 and it work as expected.

It seem to be related to VXLan cause it happend only over Cross Subnets.

Any idea to help me out ?

Thanks 🙏

[valentin2105]

It work by replacing VXLan by IPinIP.

[caseydavenport]

With TCPDump, I see the ICMP, but the http packet doesn't arrive on the node.

Hm, this is pretty strange because VXLAN doesn't handle ICMP vs TCP / HTTP traffic differently. I wonder if there is something else (like a policy rule or similar?) that might be blocking the TCP/HTTP traffic on the ingress node?

Do you see the TCP traffic egress the Ingress node using tcpdump?