NodePort only working on nodes with pod

[nvollmar]

Environmental Info:
K3s Version:
k3s version v1.28.2+k3s1 (6330a5b)
go version go1.20.8

Node(s) CPU architecture, OS, and Version:
Linux k3s-worker-0 6.5.0-rc5-edge-rockchip-rk3588 #3 SMP PREEMPT Sun Aug 6 22:07:51 UTC 2023 aarch64 GNU/Linux

Cluster Configuration:
2 master nodes, 2 worker nodes

Describe the bug:
I've set up a service with a NodePort and it is only reachable on the node which the pod is running on.

Service definition:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mosquitto
  name: mosquitto
  namespace: home
spec:
  clusterIP: 10.43.208.9
  clusterIPs:
  - 10.43.208.9
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  ports:
  - name: mqtt
    nodePort: 31883
    port: 1883
    protocol: TCP
    targetPort: mqtt
  selector:
    app: mosquitto
  sessionAffinity: None
  type: NodePort
status:
  loadBalancer: {}

Steps To Reproduce:
k3s installed using k3s-ansible project with HA. Server args:

apiserver_endpoint: "10.3.0.1"
apiserver_dns: "k3s-lb.vollmar.local"

extra_server_args: >
  --tls-san={{ apiserver_endpoint }} --tls-san={{ apiserver_dns }} --advertise-address={{ apiserver_endpoint }} 
  --disable local-storage --disable traefik --disable metrics-server --disable-cloud-controller --disable-helm-controller 
  --egress-selector-mode=disabled

Expected behavior:
NodePort should be reachable on all nodes

Actual behavior:
NodePort only working on node with the pod running

[brandond]

What Linux distribution are you using? Do you have an existing iptables firewall (firewall or ufw) with conflicting rules, or have you configured iptables on your nodes to have a default deny or drop policy?

[nvollmar]

Thanks for the fast response.

I'm running the Debian Bullseye version of Armbian 23.8.1 on an OrangePi 5.
I only installed iptables-legacy and k3s, no manual modifications to iptables on my side.

iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N FLANNEL-FWD

after that only rules that were added by k3s.

[Nikola-Milovic]

What was the reason for this issue? I am also experiencing it

Ufw is disabled via ufw disable

Only being able to access the apps from the node that is actually running the pod.

Linux control1 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

k3s --version
k3s version v1.30.4+k3s1 (98262b5d)
go version go1.22.5

I do run this script inside the VMs

#!/bin/bash
#
# Sets up the kernel with the requirements for running Kubernetes
set -e

# Add br_netfilter kernel module
cat <<EOF >>/etc/modules
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
br_netfilter
nf_conntrack
EOF
systemctl restart systemd-modules-load.service

# Set network tunables
cat <<EOF >>/etc/sysctl.d/10-kubernetes.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
net.bridge.bridge-nf-call-iptables=1
net.ipv4.ip_forward=1
EOF

sysctl --system

ufw disable