Using dind-rootless, updating dind or kubernetes mode configurations

[Link-]

With the new Autoscaling runner set for ARC we introduced containerMode to handle GitHub Actions workflows with jobs running in a container or for running service containers.

TLDR; This is causing a lot of confusion for many of you because of the way it's designed. Many are under the assumption that the configuration we inject is hardcoded and not configurable. It's not hardcoded and it is configurable. Explanation below 👇

[Link-]

The illusion of 2 options

Whenever you uncomment containerMode in the values.yaml for the gha-runner-scale-set helm chat you have 2 options dind or kubernetes.

When you choose dind for example, this configuration will be automatically injected into the following template section and when you choose kubernetes this will be injected into the template section.

The 3rd option

If you want to customize either of these options (dind or kubernetes), just keep containerMode commented out and simply paste whatever configuration you want under template!

Example: changing the DOCKER_HOST value for dind

Let's say you don't want the DOCKER_HOST to be set to: unix:///run/docker/docker.sock and instead, you want it to be unix:///var/docker/docker.sock

This is how your values.yaml should look like:

## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: "https://github.com/actions/actions-runner-controller"

## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## You can choose to use GitHub App or a PAT token
githubConfigSecret: my-super-safe-secret

## maxRunners is the max number of runners the autoscaling runner set will scale up to.
maxRunners: 5

## minRunners is the min number of runners the autoscaling runner set will scale down to.
minRunners: 0

runnerGroup: "my-custom-runner-group"

## name of the runner scale set to create.  Defaults to the helm release name
runnerScaleSetName: "my-awesome-scale-set"

## Container mode is an object that provides out-of-box configuration
## for dind and kubernetes mode. Template will be modified as documented under the
## template object.
##
## If any customization is required for dind or kubernetes mode, containerMode should remain
## empty, and configuration should be applied to the template.
# containerMode:
#   type: "dind"  ## type can be set to dind or kubernetes
#   ## the following is required when containerMode.type=kubernetes
#   kubernetesModeWorkVolumeClaim:
#     accessModes: ["ReadWriteOnce"]
#     # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
#     storageClassName: "dynamic-blob-storage"
#     resources:
#       requests:
#         storage: 1Gi
#   kubernetesModeServiceAccount:
#     annotations:

## template is the PodSpec for each listener Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
# listenerTemplate:
#   spec:
#     containers:
#     # Use this section to append additional configuration to the listener container.
#     # If you change the name of the container, the configuration will not be applied to the listener,
#     # and it will be treated as a side-car container.
#     - name: listener
#       securityContext:
#         runAsUser: 1000
#     # Use this section to add the configuration of a side-car container.
#     # Comment it out or remove it if you don't need it.
#     # Spec for this container will be applied as is without any modifications.
#     - name: side-car
#       image: example-sidecar

## template is the PodSpec for each runner Pod
## For reference: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec
template:
  spec:
    initContainers:
    - name: init-dind-externals
      image: ghcr.io/actions/actions-runner:latest
      command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
      volumeMounts:
        - name: dind-externals
          mountPath: /home/runner/tmpDir
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
        - name: DOCKER_HOST
          value: unix:///var/docker/docker.sock
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /run/docker
          readOnly: true
    - name: dind
      image: docker:dind
      args:
        - dockerd
        - --host=unix:///var/docker/docker.sock
        - --group=$(DOCKER_GROUP_GID)
      env:
        - name: DOCKER_GROUP_GID
          value: "123"
      securityContext:
        privileged: true
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /run/docker
        - name: dind-externals
          mountPath: /home/runner/externals
    volumes:
    - name: work
      emptyDir: {}
    - name: dind-sock
      emptyDir: {}
    - name: dind-externals
      emptyDir: {}

## Optional controller service account that needs to have required Role and RoleBinding
## to operate this gha-runner-scale-set installation.
## The helm chart will try to find the controller deployment and its service account at installation time.
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
## to help it finish RoleBinding with the right service account.
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
# controllerServiceAccount:
#   namespace: arc-system
#   name: test-arc-gha-runner-scale-set-controller

[Link-]

To give extra guidance, template maps to PodSpec for each runner Pod! That means that every field in this API is supported and can be used under template: https://kubernetes.io/docs/reference/kubernetes-api/workload-resources/pod-v1/#PodSpec

[alvicsam]

For people seeking a ready-made solution for dind-rootless I would like to put a link here for a PR that should've introduced rootless dind in ARC: #2919
Unfortunately it wasn't merged but in the comments you can find a template config that works (though it need some adjustments for your installation).

[shankarRaman]

@Link- we have policies where we need to specify container resources (memory/cpu) when deploying our pods. However I see that the override does not honor these values for the dind container

[kamigerami]

Hey, so most of our users that run our runners do some sort of an docker build, push workflow for our runners. We are however running these runners in k8s so I assumed I had to run it in containerMode=kubernetes however that way we don't get access to the docker socket so I have to run them in dind mode - so my question goes - what is the point of containerMode=kubernetes in that case? Anybody care to explain :)

[shankarRaman]

@kamigerami you can learn the difference here https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller#using-kubernetes-mode

[DevBey]

@shankarRaman

why do we get this error when in kubernetes mode, I don't understand
what's the point of kubernetes mode if you can't run container. I have read the document multiple times.

Error: Building container actions is not currently supported

[shankarRaman]

@DevBey sorry for the delay in getting back. I've not tried the kubernetes mode. Perhaps you can share your contanierMode configuration.
Tagging @nikola-jokic

[DevBey]

apologies @shankarRaman, @nikola-jokic,

this is what we are using right now, question is why are we getting this when using actions which spawn other containers, the same is working in DIND mode ofcourse.

for example or this action https://pr-agent-docs.codium.ai/installation/github/#run-as-a-github-action
we get this error

Error: Building container actions is not currently supported

## githubConfigUrl is the GitHub url for where you want to configure runners
githubConfigUrl: "https://github.com/Kavida-team"

## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
githubConfigSecret: pre-defined-secret

## Container mode configuration
containerMode:
  type: "kubernetes"

## maxRunners is the max number of runners the autoscaling runner set will scale up to
maxRunners: 5

## minRunners is the min number of idle runners
minRunners: 1

## template is the PodSpec for each runner Pod
template:
  spec:
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: "Deployment"
                  operator: "In"
                  values:
                    - "mongodb_azc"
    securityContext:
      fsGroup: 1001
      runAsUser: 1001
      runAsGroup: 1001
    initContainers:
          - name: kube-init
            image: ghcr.io/actions/actions-runner:latest
            command: ["sudo", "chown", "-R", "1001:123", "/home/runner/_work"]
            volumeMounts:
            - name: work
              mountPath: /home/runner/_work

    containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest
        command: ["/home/runner/run.sh"]
        resources:
          requests:
            cpu: "50m"
            memory: "1Gi"
          limits:
            cpu: "2"
            memory: "4Gi"
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOKS
            value: /home/runner/k8s/index.js
          - name: ACTIONS_RUNNER_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
            value: "false"
          - name: DOCKER_CONFIG
            value: /home/runner/.docker
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: tool-cache
            mountPath: /home/runner/_tool
          - name: docker-cache
            mountPath: /home/runner/.docker
        securityContext:
          allowPrivilegeEscalation: false
    volumes:
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              accessModes: [ "ReadWriteOnce" ]
              storageClassName: "gp3"
              resources:
                requests:
                  storage: 10Gi
      - name: tool-cache
        emptyDir:
          sizeLimit: 5Gi
      - name: docker-cache
        persistentVolumeClaim:
          claimName: docker-cache-pvc

#---
## PersistentVolumeClaim for Docker cache
#apiVersion: v1
#kind: PersistentVolumeClaim
#metadata:
#  name: docker-cache-pvc
#spec:
#  accessModes:
#    - ReadWriteMany  # This allows multiple pods to use the volume
#  resources:
#    requests:
#      storage: 20Gi
#  storageClassName: gp3  # Make sure this storage class supports ReadWriteMany, or change as needed