ip tables legacy vs nf_tables issue persists #467
Comments
|
Can you run the following one-liner on affected infrastructure and provide the full output?
It should look something like this: $ docker run -it --rm --privileged docker:dind sh -euxc 'modprobe nf_tables > /dev/null 2>&1 || :; if ! iptables -nL > /dev/null 2>&1; then modprobe ip_tables || :; /usr/local/sbin/.iptables-legacy/iptables -nL > /dev/null 2>&1; echo success legacy; else echo success nftables; fi'
+ modprobe nf_tables
+ :
+ iptables -nL
+ echo success nftables
success nftablesor: $ docker run -it --rm --privileged docker:dind sh -euxc 'modprobe nf_tables > /dev/null 2>&1 || :; if ! false iptables -nL > /dev/null 2>&1; then modprobe ip_tables || :; /usr/local/sbin/.iptables-legacy/iptables -nL > /dev/null 2>&1; echo success legacy; else echo success nftables; fi'
+ modprobe nf_tables
+ :
+ iptables -nL
+ modprobe ip_tables
ip: can't find device 'ip_tables'
ip_tables 36864 0
x_tables 53248 8 ip_tables,xt_mark,xt_nat,xt_tcpudp,xt_conntrack,xt_MASQUERADE,xt_addrtype,nft_compat
modprobe: can't change directory to '/lib/modules': No such file or directory
+ :
+ /usr/local/sbin/.iptables-legacy/iptables -nL
+ echo success legacy
success legacy |
|
Actually, instead, could someone test #468? 👀
|
|
I tried the fix on a COS 105 instance, without success unfortunately: Same as before. I had a look at the fix and checked from the host and container side which IP table chains are visible to the container - maybe I overlooked something in the first place.
This looks interesting, the container does not see a rule chain I also came across
|
|
The only way I could make my system use the legacy ip tables is to check for the warning as described in: https://github.com/docker-library/docker/pull/468/files#r1430804593 See https://github.com/kautsig/docker/blob/better-iptables2/24/dind/dockerd-entrypoint.sh#L160 I don't know enough about this code to judge if it is even close to correct. What I see is that, the entrypoint always chooses |
|
For me, the suggestion by @tianon failes. My Gitlab dind services keep failing to resolve. Build and run And exec into that container and run nslookup. There is nothing in the above containers that suggest why it is failing. Contrary to @kautsig, I do not see errors related to However, with the container from @kautsig it succeeds. Build and run And exec into that container and run nslookup. The scripts from @tianon |
|
@frederikbosch On which host distribution are you on? We have the issue on our gitlab runners on COS 105 where the first fix #465 and current proposal #468 do not work. On gitlab prod (COS 85) both fixes were confirmed to work: #468 (comment) |
|
Old one, Ubuntu 18.04. |
This was referenced Dec 19, 2023
|
I faced same issue on gke Container-Optimized OS. nf_tables is available and loaded into kernel Run
Workaround is either use image Ubuntu with containerd or specify docker image to 24.0.7-dind-alpine3.18 GKE version: 1.27.3-gke.100 @kautsig Will #468 (comment) solve this issue? |
|
@saintnoah The comment you linked refers production gitlab saas runners. They use COS 85, for them the change done in the MR at that time didn't break it again (they need legacy ip tables). For our issue, I am on COS 105, I'd say, this depends on how the discussion in the MR progresses... the My current perspective on the wider issue is that the use of |
|
@kautsig Agree, docker version should be specified to avoid similar upstream issues. What I found out is with COS 105, And for the ubuntu OS |
|
On COS 85: stanhu-old-cos /home/stanhu # iptables -V
iptables v1.6.2
stanhu-old-cos /home/stanhu # docker run --privileged --rm -it docker:24.0.7-dind sh
/ # iptables -V
iptables v1.8.10 (nf_tables)
/ # iptables -nL
iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument
/ # echo $?
4On COS 105: stanhu-test /home/stanhu # iptables -V
iptables v1.8.5 (legacy)
stanhu-test /home/stanhu # docker run --privileged --rm -it docker:24.0.7-dind sh
/ # iptables -V
iptables v1.8.10 (nf_tables)
/ # iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
/ # echo $?
0Both the hosts of COS 85 and 105 use the legacy However, in COS 105, It seems that the auto-detection doesn't always work because the host Maybe for now we can just set a default environment variable, such as What do you think @tianon? |
|
Just wanted to confirm the problem is solved, For readers using the docker+machine gitlab runner (I think it came up in the discussion above), adding the environment variable to the |
adamconnelly
added a commit
to spacelift-io/spacelift-helm-charts
that referenced
this issue
Jan 12, 2024
It looks like the latest version of dind is causing issues in certain hosting environments. It may be due to this issue: docker-library/docker#467. Let's pin to a known-working version of `dind`.
adamconnelly
added a commit
to spacelift-io/spacelift-helm-charts
that referenced
this issue
Jan 12, 2024
It looks like the latest version of dind is causing issues in certain hosting environments. It may be due to this issue: docker-library/docker#467. Let's pin to a known-working version of `dind`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
For us, the issue described in #463 persists, even after the fix. Our host image is a Google COS 105.
Initially I added my findings as a comment to #466, but after getting the logs, the problems looks different than the one described there.
What we observe is that the host uses a legacy ip tables version, but the
nf_tableskernel module is present and loads fine. My guess is, that this makes the fix ineffective, because it usesmodprobe nf_tablesto determine which iptables version to use.Running the latest dind image on this system shows the original error.
The text was updated successfully, but these errors were encountered: