Cross-Cluster Service Connectivity Fails with "Host is Unreachable" Despite Successful DNS Resolution in Submariner GlobalNet Setup #3204
Assignees
Comments
|
Thanks for reaching out @aswinayyolath. A. As mentioned in Slack discussion, inter-cluster libreswan tunnel is up and communication between gw nodes is fine while communication from non-GW node to gw node is failing. further dapath investigation is needed here, I assume that for some reason (maybe infra firewall, connection tracking) ingress packet is being dropped in gwnode@clusterX to nongwnode@clusterX segment. Can you please run ping from non-gw node@sub1 to gw-node@sub2 (for gw-node@sub2 IP address you should use endpoint healthcheck IP == 242.0.255.254) and tcpdump the gw node and non-gw node on cluster sub1 ? B. Also, this is not relevant to datapath issue, but I noticed that Submariner detected the CNI as generic instead of flannel, Submariner uses this code to discover network details for flannel CNI. |
yboaron
added
flannel
|
Please note: I have to create a new cluster as I messed up the old one trying various stuffs Endpoint health check IP for the gateway node in sub2 :
|
DaemonSet List:Checked PodsCNI Configuration |
Is there flannel daemonset in another namespace? |
|
Yes |
|
The kube-flannel-ds DaemonSet has the following volumes volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni-plugin
hostPath:
path: /opt/cni/bin
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg
- name: xtables-lock
hostPath:
path: /run/xtables.lock
type: FileOrCreate
CM detailsapiVersion: v1
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"EnableNFTables": false,
"Backend": {
"Type": "vxlan"
}
}
kind: ConfigMap
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","data":{"cni-conf.json":"{\n \"name\": \"cbr0\",\n \"cniVersion\": \"0.3.1\",\n \"plugins\": [\n {\n \"type\": \"flannel\",\n \"delegate\": {\n \"hairpinMode\": true,\n \"isDefaultGateway\": true\n }\n },\n {\n \"type\": \"portmap\",\n \"capabilities\": {\n \"portMappings\": true\n }\n }\n ]\n}\n","net-conf.json":"{\n \"Network\": \"10.244.0.0/16\",\n \"EnableNFTables\": false,\n \"Backend\": {\n \"Type\": \"vxlan\"\n }\n}\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"flannel","k8s-app":"flannel","tier":"node"},"name":"kube-flannel-cfg","namespace":"kube-flannel"}}
creationTimestamp: "2024-11-03T16:24:55Z"
labels:
app: flannel
k8s-app: flannel
tier: node
name: kube-flannel-cfg
namespace: kube-flannel
resourceVersion: "282"
uid: f0058e4b-4ba9-49be-a759-fd0c9843a88d
|
|
Thanks for the information, Regarding flannel discovery, it looks like we need to update flannel discovery code. QQ: does Could you please report a new issue for flannel CNI discovery? please attach relevant information, we welcome any code contribution here :-) . As per the datapath issue, traffic initiated at nongw node@clusterA towards remoter cluster is encapsulated in VxLAN (port 4800, interface vx-submariner) towards gw node@clusterA and gw node should forward it to remote cluster gw. Can you double check (maybe use tcpdump -pi ) that no packet is sent in nonGW node ? I can see that on gw node iptables (filter table) packet counter for input traffic on vx-submariner interface is > 0 , check [1] . [1] |
|
QQ: does kubectl get ds -A -l k8s-app=flannel return flannel ds ? I will report a new issue and see if I can contribute (I guess changes should be relatively small) ... Packet Transmission on the Non-GW Node in sub1 (ClusterA)
Run Packet Capture on the Non-Gateway Node in ClusterAVerified Reception on the Gateway Node in ClusterAis this what you want me to do? I am not 100% sure |
|
I have created a new issue: #3210. A draft change has been pushed here: #3268. @yboaron, I haven't yet looked into linting, unit tests, or e2es testing; I'm just checking if the changes look something like this (Draft linked above). I also modified the loop structure from for k := range daemonsets.Items {
if strings.Contains(daemonsets.Items[k].Name, "flannel") {
volumes = daemonsets.Items[k].Spec.Template.Spec.Volumesto for _, ds := range daemonsets.Items {
if strings.Contains(ds.Name, "flannel") {
flannelDaemonSet = &ds
volumes = ds.Spec.Template.Spec.Volumes
break
}
}to enhance code readability and clarity. I think thid approach makes it clear that ds represents a DaemonSet obj, eliminating the need for indexing. Additionally, by storing a pointer to the found DS and breaking the loop upon finding it, I believe if we do something like this the code becomes more efficient and reduces the risk of errors associated with accessing elements via an index. |
Submariner only handles egress routing and only for packets destined to remote clusters (dest IP is from remote pod,service CIDRs, in your case it is globalNet CIDR for remote cluster) , please tcpdump while pinging remote endpoint healthcheck IP address |
|
|
Can you try running |
|
I am seeing a lot of output from |
|
Hmmmm, its ICMP/IPv6 traffic , don't you get any ICMP/IPv4 ( |
|
|
Hmm, strange, can't see the V4 icmp sent to remote cluster. |
|
I don't have the cluster with me 😔. But I will create one (in fact 2). @yboaron I would like to check with you if the steps I am following is correct or not. Could you please review the Steps here (https://kubernetes.slack.com/archives/C010RJV694M/p1730390398271589?thread_ts=1730390376.380879&cid=C010RJV694M) and let me know If I am missing anything please? |
|
I would also like to test the same in AWS across 2 regions. I just want to know if the steps I followed is correct and I will try it in Both the VM I used before as well as I will create 2 EKS cluster in 2 diff regions in AWS and see if that works |
Yep, looks fine. Can you try reinstalling without adding --globalnet-cidr 242.0.0.0/16 flag in subctl join command for both clusters |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What happened:
I deployed Submariner with GlobalNet across two Kubernetes clusters. DNS resolution works as expected, but connectivity to services across clusters fails with a
Host is unreachableerror.More info is available in below link
https://kubernetes.slack.com/archives/C010RJV694M/p1730390376380879
What you expected to happen:
curlrequests from a pod in cluster2 to a service exposed via Submariner in cluster1 should succeed, indicating that cross-cluster communication is functioning.How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
subctl diagnose all):Cluster 1 info
Cluster 2 info
subctl gather):sub1.zip
sub2.zip
K8S is installed on ubuntu VM
OS INFO
The text was updated successfully, but these errors were encountered: