113 Host is unreachable

[MinhDuc010194]

i had follow this example to config Kong ingress point to foo Service https://konghq.com/blog/kubernetes-ingress-api-gateway. The last step is to check the kong gateway i got a message "An invalid response was received from the upstream server". Then I log kong_ingress controller by command "kubectl logs -n kong ingress-kong-74d8d78f57-gvv5z -c proxy". and I got

2021/08/16 02:42:02 [error] 23#0: *763 recv() failed (113: Host is unreachable), context: ngx.timer, client: 127.0.0.1, server: 127.0.0.1:8444
2021/08/16 02:42:02 [error] 23#0: *763 recv() failed (113: Host is unreachable), context: ngx.timer, client: 127.0.0.1, server: 127.0.0.1:8444
2021/08/16 02:42:03 [error] 23#0: *696 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.0.10.50, server: kong, request: "GET /foo HTTP/1.1", upstream: "http://192.168.235.212:5000/foo", host: "10.0.10.50"

How can i fix this?
my nodeip is 10.0.10.50 and i using metallb to create LB at external Ip 10.0.10.50

[fffonion]

@MinhDuc010194 The ip address of upstream looks weird, what's the output of kubectl get service foo-service?

[MinhDuc010194]

this is IP of pod foo, not foo-service. here is IP of foo_service
foo-service ClusterIP 10.97.248.8 5000/TCP 11h

[MinhDuc010194]

here is what i got from logs of kong_proxy. maybe this host had a problem

2021/08/16 12:40:27 [error] 24#0: *152051 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:27 [error] 24#0: *152051 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:29 [error] 24#0: *152061 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:29 [error] 24#0: *152061 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:31 [error] 24#0: *152061 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:31 [error] 24#0: *152079 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:33 [error] 24#0: *152079 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:33 [error] 24#0: *152079 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:35 [error] 24#0: *152099 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:35 [error] 24#0: *152099 recv() failed (113: Host is unreachable), context: ngx.timer

2021/08/16 12:40:37 [error] 24#0: *152099 recv() failed (113: Host is unreachable), context: ngx.timer
2021/08/16 12:40:37 [error] 24#0: *152099 recv() failed (113: Host is unreachable), context: ngx.timer

this error is thrown every 2 seconds

[fffonion]

@MinhDuc010194 The error for 2 seconds is likely from healthchecks. If you look at the error for the request

2021/08/16 02:42:03 [error] 23#0: *696 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.0.10.50, server: kong, request: "GET /foo HTTP/1.1", upstream: "http://192.168.235.212:5000/foo", host: "10.0.10.50"

It indicates Kong resolved the upstream to be 192.168.235.212, which looks weird and I feel like that's the Host is unreachable error comes from, the pod could have route with 10.97.248.8, and may have no route to route to 192.168.235.212. But I don't immediately have an idea on why it resolves to 192.168.x.x instead of the 10.x.x.x

[MinhDuc010194]

@fffonion because of a firewall. I had to disable it! and now everything works fine! thank you for pointing me an important hint!

[bungle]

I guess this can be closed then. @fffonion / @MinhDuc010194, please reopen if you have further questions.

[cleanet]

The logs:

2021/08/16 02:42:02 [error] 23#0: *763 recv() failed (113: Host is unreachable), context: ngx.timer, client: 127.0.0.1, server: 127.0.0.1:8444
2021/08/16 02:42:02 [error] 23#0: *763 recv() failed (113: Host is unreachable), context: ngx.timer, client: 127.0.0.1, server: 127.0.0.1:8444
2021/08/16 02:42:03 [error] 23#0: *696 connect() failed (113: Host is unreachable) while connecting to upstream, client: 10.0.10.50, server: kong, request: "GET /foo HTTP/1.1", upstream: "http://192.168.235.212:5000/foo", host: "10.0.10.50"

means that nginx is accessing at application since the endpoint 192.168.235.212:5000.

This socket, is the endpoint of you service. You can see it, do it:

kubectl get endpoints -n kong

In this case, is the endpoints of service.
But seeing that throw a 502 Bad Gateway and the logs, this means that the ingress controller is trying access at service via endpoint (trying with all the endpoints of ingress controller). And the ingress controller's pod cannot access.

For test it, entry in the pod of ingress controller and checks the connection.

$ kubectl exec -it pod/ingress-nginx-controller-57ff8464d9-pvjpc -n ingress-nginx -- bash
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ nc -zv 192.168.235.212 5000
nc: 10.85.0.12 (10.85.0.12:8080): Host is unreachable
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ 

As we see exactly , this cannot access.

You look that IP has the service and try access

$ kubectl describe service
$ kubectl exec -it pod/ingress-nginx-controller-57ff8464d9-pvjpc -n ingress-nginx -- bash
ingress-nginx-controller-57ff8464d9-pvjpc:/etc/nginx$ nc -zv <clusterIP> <port>
<clusterIP> (<clusterIP>:<port>) open

And as we see, the pod has access. With the ClusterIP and Port of the service.

So that a solution would be do the follow.

You must tell at Ingress, that uses the ClusterIP:port instead of use endpoints list of kong controller.

For this you edit the Ingress resource and add the follow annotation.

ingress.kubernetes.io/service-upstream: "true"

FYI

ingress.kubernetes.io/service-upstream

Available since controller 0.6

By default, the Kong Ingress Controller distributes traffic amongst all the Pods of a Kubernetes Service by forwarding the requests directly to Pod IP addresses. One can choose the load-balancing strategy to use by specifying a KongIngress resource.

However, in some use-cases, the load-balancing should be left up to kube-proxy, or a sidecar component in the case of Service Mesh deployments.

Setting this annotation to a Service resource in Kubernetes will configure the Kong Ingress Controller to directly forward the traffic outbound for this Service to the IP address of the service (usually the ClusterIP).

kube-proxy can then decide how it wants to handle the request and route the traffic accordingly. If a sidecar intercepts the traffic from the controller, it can also route traffic as it sees fit in this case.

Following is an example snippet you can use to configure this annotation on a Service resource in Kubernetes, (please note the quotes around true):

annotations:
  ingress.kubernetes.io/service-upstream: "true"