Add option to run containers with podman instead of docker

[mbestavros]

My team and I are looking into using the Github Actions runner for self-hosted tests of our project. For various reasons, we'd like to use Podman to run Actions containers instead of Docker, and we're interested in an option that would allow us to specify which container framework to use in the runner app.

I'll note that we are not advocating for the replacement of Docker; rather, the addition of a Podman option.

We'd potentially be interested in implementing this ourselves and submitting a patch, but we wanted to start a discussion around it and see if it was something the maintainers would be interested in merging.

Thoughts welcome.

[bryanmacfarlane]

I'm currently investigating this while looking into native k8s runners (without dind). It might be awhile but it's on our radar.

[npmccallum]

@bryanmacfarlane I just happened to notice that you live near me. Could I take you out to lunch (my treat!)? I'd love to discuss our experiences of GitHub Actions with you (and also just make a new acquaintance).

[bryanmacfarlane]

my email is my github alias and that's at github

[davidkarlsen]

@bryanmacfarlane native k8s runners would be totally awesome. Any news on that or a roadmap?

[sycured]

Any ETA about Podman ?
Podman v2 has a rest api: https://podman.io/blogs/2020/06/29/podman-v2-announce.html

[missedone]

now that Kubernetes announced that deprecate docker in 1.20 and remove in 1.22, we're looking for the solution of running github actions with DinD in k8s cluster.

[trevor-vaughan]

Not sure what GitHub is using under the hood but I just added this PR to docker-api that allows it to work with podman. As noted, it's not "great" but it certainly works. If pointed at a system-level podman API socket, you can get cross-node communication without the fuss of k8s.

upserve/docker-api#569

[pevik]

Any update on this, please? FYI due another problem with docker glibc2.34 docker blocks clone3 syscall any project which use Tumbleweed container (and very soon Fedora and other distros which start to use glibc 2.34) on GitHub Actions (e.g. snapper, LTP, iputils) is hit by this issue because GitHub Actions does not allow to use podman.

Sometimes it feels like keeping up the CI is harder than the project itself.

[ssbarnea]

Any updates? As everyone is moving away from docker GHA should act and address the issue.

[stevehipwell]

@bryanmacfarlane have you made any progress with this? It'd be great if the action runner could use Kubernetes or another system such as Podman to run containers.

My primary use case is to be able to run Kaniko from an actions-runner-controller based ephemeral runner without needing to use DinD; I can do this successfully with GitLab runners and this is the big pain point with the current runner implementation.

It looks like #1461 & #1522 have been independently opened to look at solving this but I'm not sure if they're officially sanctioned?

[mrserverless]

Podman and Buildah are drop in, daemonless replacements for docker CLI. There is no need for Kubernetes nor DinD. Any standard linux runner will do.

In reference to the original poster's query, Redhat has wrote a blog here: https://www.redhat.com/en/blog/build-ubi-containers-github-actions-buildah-and-podman.

[moubre]

We're experiencing this issue trying to create RHEL based self-hosted runners. Given the market resistance to Docker, we'd like to see GitHub support the buildah/podman tooling. This could be supported either by configuration directives for the runner or by workflow syntax additions to provide replacement command alternatives for docker.

[ssbarnea]

Installing docker manually on ubuntu-latest runners is decently fast (~1min) but on macos-latest is extremely slow, basically ~9-14 minutes as it needs to install several brew packages and download the linux machine and start it.

It would be very useful to have podman pre-install on both platforms. We do develop vscode extensions which required containers to run and we need to test them on macos+podman in order to detect platform specific bugs. We do it now, but with a huge delay caused by the fact that we do need to install it manually.

[jmorcar]

k8s pod runners have the risk problem of docker in docker, so GitHub actions images pulled via Marketplace need an alternative to use another command different to “docker”, like podman, crictl…

[VolkerSchiewe]

We are trying to run a container job on a self-hosted runner without docker. Instead of docker we are having podman with an alias to docker. Unfortunately the job fails already on login (Docker login for 'registry.enterprise.example.com' failed with exit code 125) I assume the issue is that podman login is not fully compatible with docker login :/

Is there any progress in making the actions runner compatible with docker alternatives like podman?

[coiby]

We are trying to run a container job on a self-hosted runner without docker. Instead of docker we are having podman with an alias to docker. Unfortunately the job fails already on login (Docker login for 'registry.enterprise.example.com' failed with exit code 125) I assume the issue is that podman login is not fully compatible with docker login :/

Is there any progress in making the actions runner compatible with docker alternatives like podman?

@VolkerSchiewe I'm not sure if you have exactly the same issue as mine. If it's the same one, you could make podman ignore "--config" as a workaround.

[l3ender]

While not ideal, we were able to use podman (on RHEL9) by adding a symlink:

ln -s /usr/bin/podman /usr/bin/docker

Please add full support for podman! For any new server installation, podman is preferred over docker.

[traylenator]

While not ideal, we were able to use podman (on RHEL9) by adding a symlink:

ln -s /usr/bin/podman /usr/bin/docker

On EL9 installing the package podman-docker will create this symlink for you.

[thesayyn]

Also currently, if you have a matrix of macos jobs that installs podman, it gets stuck because of brew lock.

[chrispat]

The new Actions Runner Scale Set has an option to use native Kubernetes apis instead of docker in docker for running container actions in a workflow https://github.com/actions/actions-runner-controller/releases/tag/gha-runner-scale-set-0.4.0. This should resolve the majority of the potential security issues faced when using docker-in-docker.

As for Podman, running container actions is a pluggable piece for the runner. You could implement a hook for using podman instead of docker or kubernetes and install that into your runners.

[seanlafferty-ibm]

There are things in the runner which are docker-specific, that make creating a podman runner with the hooks frustrating.

1. The runner provides a list of systemMounts that don't actually exist- docker will create these on your host, but other container runtimes (like podman) will error Non-existing systemMount folders prevent use of podman runner-container-hooks#176
2. The runner also hardcodes the docker socket in the list of mounts

   runner/src/Runner.Worker/Container/ContainerInfo.cs

   Line 57 in 2979fba

   this.MountVolumes.Add(new MountVolume("/var/run/docker.sock", "/var/run/docker.sock"));

In your custom hook, you could filter the later, and build a safe-guard script to protect against the former, but it's been a real challenge to get to the bottom of this.

[ritzk]

This needs #1754 . This would allow one to set environment for local unix socket ( not over network/tcp).

DOCKER_HOST=unix:///run/user/$UID/podman/podman.sock

Without this patch, on centos/fedora to run as non-root user

$ sudo dnf install podman-docker
$ sudo ln -sf /run/user/$UID/podman/podman.sock /var/run/docker.sock