KinD no longer working in GitHub Codespaces

[avillela]

What happened:

Up until sometime this summer (~July 2024) I used to be able to run KinD in GitHub codespaces. Recently, however, I am no longer able to do so.

What you expected to happen:

After installing KinD in my Codespace, I expect it to spin up a Kubernetes cluster; however, it fails to do so.

How to reproduce it (as minimally and precisely as possible):

After creating my codepsace, I run the following:

# Install KinD
[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind
kind --version

# Create KinD cluster
kind create cluster --name otel-python-lab

After running kind create cluster, I get the following error:

ERROR: failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1
Command Output: Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o br-6ded57439d75 -j MASQUERADE: ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
 (exit status 3))

I used to not get this error before and was able to create a cluster successfully.

Anything else we need to know?:

Environment:

• kind version: (use kind version): v0.24.0
• Runtime info: (use docker info, podman info or nerdctl info):

Client:
 Version:    27.0.3-1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 27.0.3-1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88c3d9bc5b5a193f40b7c14fa996d23532d6f956
 runc version: 2c9f5602f0ba3d9da1c2596322dfc4e156844890
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.5.0-1025-azure
 Operating System: Ubuntu 22.04.5 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.744GiB
 Name: codespaces-bdba78
 ID: d9fc66fe-981c-4c91-80e6-521f00d0feae
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: codespacesdev
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

• OS (e.g. from /etc/os-release): Ubuntu 22.04.5 LTS
• Kubernetes version: (use kubectl version): Not sure, because it doesn't install
• Any proxies or other special environment settings?: None.

I am aware of #2412; however, it does not seem to apply to me, as far as I can tell, unless I'm missing something??

[BenTheElder]

#3677

Docker enabled ipv6 by default but it doesn't work in codespaces.

ERROR: failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1
Command Output: Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule: (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o br-6ded57439d75 -j MASQUERADE: ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
(exit status 3))

We can probably detect and handle this docker error, in the meantime please explicitly disable ipv6 in the daemon if ipv6 kernel modules are not available on the host, or roll back to an earlier version of docker.

[BenTheElder]

The problem is ipv6 now being enabled in docker (so we don't get the expected error that ipv6 is disabled) while not actually being functional. It's not a codespaces specific issue, though IMHO codespaces should really either provide the relevant kernel modules or configure docker to disable ipv6.

[BenTheElder]

Can you try from the latest sources? #3749

there are instructions in the docs for building from source: https://kind.sigs.k8s.io/docs/user/quick-start/#installing-from-source (note to use HEAD commit use @latest, or better yet clone the repo and run make install)

[BenTheElder]

@avillela were you able to test the fix?

[avillela]

Hi @BenTheElder - let me give it a shot...

[avillela]

@BenTheElder just tested it and got the following error when trying to create the KinD cluster (looks like same error as original):

ERROR: failed to create cluster: failed to ensure docker network: command "docker network create -d=bridge -o com.docker.network.bridge.enable_ip_masquerade=true -o com.docker.network.driver.mtu=1500 --ipv6 --subnet fc00:f853:ccd:e793::/64 kind" failed with error: exit status 1
Command Output: Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule:  (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o br-1abdd1de1641 -j MASQUERADE: ip6tables v1.8.7 (legacy): can't initialize ip6tables table `nat': Table does not exist (do you need to insmod?)
Perhaps ip6tables or your kernel needs to be upgraded.
 (exit status 3))

KinD version: 0.25.0-alpha.29+57da3f7c9eaaa2

[avillela]

For funsies, based on your notes above, I did a little experiment and manually created the docker network without the --ipv6 flag before running kind create cluster, and that seemed to allow me to create my KinD cluster.

[BenTheElder]

@BenTheElder just tested it and got the following error when trying to create the KinD cluster (looks like same error as original):

If your copy-paste exactly matches the terminal, for some reason there is an extra space in the output from docker 😕
Command Output: Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule: (iptables failed: ip6tables --wait -t nat -I POSTROUTING -s fc00:f853:ccd:e793::/64 ! -o br-1abdd1de1641 -j MASQUERADE: ip6tables v1.8.7 (legacy): can't initialize ip6tables table nat': Table does not exist (do you need to insmod?)`

Versus:

kind/pkg/cluster/internal/providers/docker/network.go

Line 273 in 57da3f7

const dockerIPV6TablesError = "Error response from daemon: Failed to Setup IP tables: Unable to enable NAT rule: (iptables failed: ip6tables"

I think that's the issue, the error doesn't match the check because the additional space isn't present in the string in kind ...

[BenTheElder]

#3759

[avillela]

That worked!

[BenTheElder]

Thanks for confirming!

I think another workaround would be disable ipv6 in docker daemon config, but we should handle this gracefully in the next release.