Document alternatives to mounting docker.sock

[charlesritchea]

A couple of our Java/Kotlin based deployments and jobs execute containerized c++ or python tools (to avoid dependency hell). By using docker-in-docker, this has allowed our k8s versions to behave the same way as our bare metal (i.e. run from IntelliJ or docker-compose). In other words, it doesn't make sense for us to spin up yet another job with a shared volume when we can use dind.

I see that this will no longer be supported in 1.24 and dds identifies the expected deployment; however, I can't find anywhere what to do about this? Do I just do the same thing with containerd.sock and nerdctl? Any alternative documented would be appreciated

[rothgar]

There are some alternative container build tools mentioned in the FAQ https://kubernetes.io/blog/2022/02/17/dockershim-faq/#what-should-i-look-out-for-when-changing-cri-implementations

The buildkit cli for kubectl seems like the most generic and flexible option https://github.com/vmware-tanzu/buildkit-cli-for-kubectl

[charlesritchea]

I don't use docker-in-docker to build, just to run other CLI containers. Like running different containerized image transformers on a shared folder. All the alternatives mentioned are for build servers

…

On Wed, Nov 9, 2022, 5:24 PM Justin Garrison ***@***.***> wrote: There are some alternative container build tools mentioned in the FAQ https://kubernetes.io/blog/2022/02/17/dockershim-faq/#what-should-i-look-out-for-when-changing-cri-implementations The buildkit cli for kubectl seems like the most generic and flexible option https://github.com/vmware-tanzu/buildkit-cli-for-kubectl — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMUMHKVVVUZO4W3NO2N5NLWHQQBXANCNFSM6AAAAAAR34E45U> . You are receiving this because you authored the thread.Message ID: <aws-containers/kubectl-detector-for-docker-socket/issues/7/1309464832@ github.com>

[rothgar]

Can you give an example of commands you run? If you’re not running `docker` commands you should be fine.

[charlesritchea]

Yes, I am using e.g. docker run -it --rm -v /my-folder:/data my-cli-program:v3 like if I was using bash -c my-cli-program. For example, I have a k8s pod that consumes messages from a queue, processing these messages can tke a long time, like an hour. While processing the message, I download like 1000 jpegs or a large 6gb file. I need to do some magic on these large binary files and have this magic c++ CLI program that has been containerized because it requires Ubuntu 14.06alpha27 or whatever. I also have a CLI program that is python 2.6 based (these are ridiculous examples but you get the picture, they are containerized to avoid dependency hell). Both of these CLI programs process the large binary files on the same volume as the host message processor. But ultimately what I am doing is running containerized CLI programs with shared data. Also, I am doing this both in k8s as well as locally on bare metal during development, so docker-in-docker let me use the same docker run command whether it was k8s or bare metal.

[rothgar]

on your local machine docker run will be the same and inside k8s I'm assuming your have a manifest file that uses the my-cli-program:v3 and mounts a volume but nothing in the CLI requires the docker command. Kubernetes will still fully support this use case so long as you are not mounting the /var/lib/docker.sock and using the docker CLI inside the container.

[charlesritchea]

Sorry I wasn't clear, I am executing docker run from my Kotlin code, so yes I am using docker run. What it comes down to is it was my (and my team leads) understanding that you can containerize any program and use it like a native program, so we chose to containerize all of our custom CLI programs (as well as use containers for other programs like gdal), the only change is wrapping the executable commands with docker run. I'm sorry I don't know how else to explain what to me is a very basic use case

[charlesritchea]

I just remembered your manifest/volume comment. Yes, there is a pod that is running a container with a volume. The program on that pod also needs to execute other docker containers with access to folders on that volume.

[rothgar]

Do you have an example kubernetes manifest and dockerfile to build the containers you use? It would probably help me understand how it's used more.

If you don't want to add them to this ticket you can email me [email protected]

[charlesritchea]

No, it's an inhouse spring application that runs as either a deployment or a job. We use inhouse programs and gdal for the containers we run inside the other container. And by inside I mean exec shell with docker run. Here is a reddit post I created about it, with lots of helpful solutions, looks like sidecar dind might be the best short term solution. https://www.reddit.com/r/kubernetes/comments/yrf215/eks_ive_been_struggling_with_the_upcoming_removal/?utm_source=share&utm_medium=android_app&utm_name=androidcss&utm_term=1&utm_content=share_button

…

On Thu, Nov 10, 2022, 5:17 PM Justin Garrison ***@***.***> wrote: Reopened #7 <#7> . — Reply to this email directly, view it on GitHub <#7 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMUMHP2OQMMXVZPNEEUXELWHVYA3ANCNFSM6AAAAAAR34E45U> . You are receiving this because you authored the thread.Message ID: <aws-containers/kubectl-detector-for-docker-socket/issue/7/issue_event/7787477755 @github.com>

[charlesritchea]

I don't think the manifest or dockerfile would show you anything, it just shows that docker.sock is mounted and docker is installed. All of the containers I'm talking about using are not represented because they are called with docker run

…

On Thu, Nov 10, 2022, 5:31 PM Charles Ritchea ***@***.***> wrote: No, it's an inhouse spring application that runs as either a deployment or a job. We use inhouse programs and gdal for the containers we run inside the other container. And by inside I mean exec shell with docker run. Here is a reddit post I created about it, with lots of helpful solutions, looks like sidecar dind might be the best short term solution. https://www.reddit.com/r/kubernetes/comments/yrf215/eks_ive_been_struggling_with_the_upcoming_removal/?utm_source=share&utm_medium=android_app&utm_name=androidcss&utm_term=1&utm_content=share_button On Thu, Nov 10, 2022, 5:17 PM Justin Garrison ***@***.***> wrote: > Reopened #7 > <#7> > . > > — > Reply to this email directly, view it on GitHub > <#7 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAMUMHP2OQMMXVZPNEEUXELWHVYA3ANCNFSM6AAAAAAR34E45U> > . > You are receiving this because you authored the thread.Message ID: > <aws-containers/kubectl-detector-for-docker-socket/issue/7/issue_event/7787477755 > @github.com> >