Difference between dind and kubernetes modes

[gerardsegarra]

Hi community!

Context

We are trying to implement the new runner scale sets, and we need as most of the community, to build docker images.
The documentation talks about dind and kubernetes containerModes, but it doesn't leave it very clear when to use each.

DinD

We see a main drawback in the dind mode, and that is basically that the requests for the dind container and the runner container have to be separated. This means that if a workflow does not use docker to build something, the request for the dind container will be unused and on the other hand if a team uses docker, the requests for the runner will be much less used. This creates the problem that we will always will be over requesting resources.

Kubernetes Mode

Then we have kubernetes mode, that the documentation doesn't clarify much what use cases it is for, and we also have the problem that when we run a docker command, it basically fails saying the it doesn't have the docker daemon running. Neither the documentation explains how to setup docker for it.

TL;DR

Do you have a suggestion on what configuration would you use to avoid over requesting resources? Is kubernetes mode the right one, and if so, how can we configure it?

Thank you in advance!

[gerardsegarra]

This is the current values.yaml we have:

listenerTemplate:
  spec:
    containers:
      - name: listener
    tolerations: [
      {
        "key": "kubernetes.azure.com/scalesetpriority",
        "operator": "Equal",
        "value": "spot",
        "effect": "NoSchedule"
      }
    ]

template:
  spec:
    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
    securityContext:
      fsGroup: 1001
    tolerations: [
      {
        "key": "kubernetes.azure.com/scalesetpriority",
        "operator": "Equal",
        "value": "spot",
        "effect": "NoSchedule"
      }
    ]

tolerations:
  - effect: NoSchedule
    key: kubernetes.azure.com/scalesetpriority
    operator: Equal
    value: spot

githubConfigUrl: "https://github.com/org/repo"
githubConfigSecret: "pre-defined-secret"
containerMode:
  type: "kubernetes"
  kubernetesModeWorkVolumeClaim:
    accessModes: ["ReadWriteOnce"]
    storageClassName: "managed"
    resources:
      requests:
        storage: 1Gi

[grzleadams]

Ultimately, the problem is that the new ARC/RSS does not support the "all-in-one" image that was available with the old ARC. Until GitHub supports that, I don't see how we can work around the over-provisioning of resources in dind mode, which is disappointing and one of the main reasons we haven't switched our K8s-based Runners over from legacy ARC.

[grzleadams]

It's also worth mentioning that (I believe) it's possible to use kaniko or other Docker alternatives to do container image builds in kubernetes mode since they don't rely on the Docker daemon, but I haven't done so. But if that does work, then you might be able to specify resource requests/limits via the template spec/hook script/however it's handled these days (although there are a bunch of discussions here where people have difficulty doing so).

[jef]

Hey @gerardsegarra, I wanted to address your concern on this topic:

We see a main drawback in the dind mode, and that is basically that the requests for the dind container and the runner container have to be separated. This means that if a workflow does not use docker to build something, the request for the dind container will be unused and on the other hand if a team uses docker, the requests for the runner will be much less used. This creates the problem that we will always will be over requesting resources.

I originally had this thought, but we can circumvent this behavior:

The DinD container and the runner container in the same pod will share the node's resources, but they do not directly share each other's allocated resources. We can request the resources in the runner container and completely remove the resource requests and limits in the DinD container. This is because once a pod is assigned to a node, Kubernetes ensures all containers in that pod are guaranteed to run on the same node.