Issues with Docker Image Download in gha-runner-scale-set

[kpinarci]

Hello,

I'm currently working on deploying the Actions-runner-controller on Kubernetes using official GitHub supported Helm Packges. For this, I've followed the guide Deploying runner scale sets.

---

My using packeges:

• gha-runner-scale-set
• gha-runner-scale-set-controller
• gha-runner-scale-set-controller

Issue 1: Proxy Configuration

While the installation process was largely seamless and I can successfully execute actions on GHE, I've encountered challenges when running actions that involve Docker commands. Specifically, there's a failure in downloading images, resulting in timeouts. Upon inspecting the environments variables in the dind container, I noticed that the proxy settings are absent. However, these settings are present and visible in the runner container, as detailed in custom-values.yaml.

Issue 2: Certificate Configuration

I've set up a configmap with our root_ca.crt (in PEM format) and referenced it in custom-values.yaml. However, when pulling Docker images from our Artifactory using the dind container, I'm confronted with an x509 error. Why aren't the certificates functioning within the container? I've noticed a similar issue with the runner container.

Issue 3: Docker Registry Mirror

I'm looking to specify our own Docker registry mirror. Currently, images are being sourced from https://registry-1.docker.io/v2/. Our goal is to utilize our Docker Registry to avoid potential rate limiting. However, I'm unable to locate the dockerRegistryMirror variable within the gha-runner-scale-set and gha-runner-scale-set-controller charts.Could someone guide me on how to direct our runners to fetch images directly from our custom dockerRegistryMirror?

I hope someone can assist me with this issue. Thank you in advance!

Job log:

Checking docker version
Clean up resources from previous jobs
Create local container network
Starting job container
  /usr/bin/docker pull node:14.16
  Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning: Docker pull failed with exit code 1, back off 7.939 seconds before retry.
  /usr/bin/docker pull node:14.16
  Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Warning: Docker pull failed with exit code 1, back off 7.127 seconds before retry.
  /usr/bin/docker pull node:14.16
  Error response from daemon: Get "https://registry-1.docker.io/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  Error: Docker pull failed with exit code 1

---

Technical Details:

• Kubernetes Version: v1.20.11
• Helm Version: v3.12.1
• gha-runner-scale-set-controller-0.5.0
• gha-runner-scale-set-0.5.0
• actions-runner:2.308.0

---

Installation Setps:

Install action-runner-controller

INSTALLATION_NAME="arc"
NAMESPACE="arc-systems"
helm upgrade --install ${INSTALLATION_NAME} \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set image.tag="0.5.0" \
    --version "0.5.0" \
    --values /home/$USER/github/ghe/arc-configuration/controller/custom-values.yaml \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

Create configmap for certifcate

NAMESPACE="arc-runners"
kubectl create ns "${NAMESPACE}"
kubectl -n "${NAMESPACE}" create configmap example-bundle-cert --from-file=/home/$USER/github/ghe/arc-configuration/certs/example_ca_bundle.crt

Secret für token erstellen

NAMESPACE="arc-runners"
kubectl create ns "${NAMESPACE}"
kubectl create secret generic pre-defined-secret \
   --namespace="${NAMESPACE}" \
   --from-literal=github_token="${GITHUB_PAT}"

Install gha-runner-scale-set

INSTALLATION_NAME="custom-dind-set" 
NAMESPACE="arc-runners"
helm upgrade --install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --version "0.5.0" \
    --values /home/$USER/github/ghe/arc-configuration/runner-scale-set-1/custom-values.yaml \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

---

Configuration File (my-value.yaml):

## githubConfigUrl is the GitHub url for where you want to configure runners
## ex: https://github.com/myorg/myrepo or https://github.com/myorg
githubConfigUrl: "https://example.example.net/kpi/arc"
## githubConfigSecret is the k8s secrets to use when auth with GitHub API.
## You can choose to use GitHub App or a PAT token

githubConfigSecret: pre-defined-secret
  ### GitHub Apps Configuration
  ## NOTE: IDs MUST be strings, use quotes
  #github_app_id: ""
  #github_app_installation_id: ""
  #github_app_private_key: |

  ### GitHub PAT Configuration
  
## If you have a pre-define Kubernetes secret in the same namespace the gha-runner-scale-set is going to deploy,
## you can also reference it via `githubConfigSecret: pre-defined-secret`.
## You need to make sure your predefined secret has all the required secret data set properly.
##   For a pre-defined secret using GitHub PAT, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_token='ghp_your_pat'
##   For a pre-defined secret using GitHub App, the secret needs to be created like this:
##   > kubectl create secret generic pre-defined-secret --namespace=my_namespace --from-literal=github_app_id=123456 --from-literal=github_app_installation_id=654321 --from-literal=github_app_private_key='-----BEGIN CERTIFICATE-----*******'
# githubConfigSecret: pre-defined-secret

## proxy can be used to define proxy settings that will be used by the
## controller, the listener and the runner of this scale set.
#
proxy:
  http:
    url: http://gw-examplenet:3000
  #  credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
  https:
    url: http://gw-examplenet:3000
  #  credentialSecretRef: proxy-auth # a secret with `username` and `password` keys
  noProxy:
    - test.example.net
    - example.com
    - .example.net
    - "172.17.0.1"
    - localhost
    - "127.0.0.1"
    - 10.0.0.0/8
    - 172.16.0.0/12
    - 192.168.0.0/1

## maxRunners is the max number of runners the auto scaling runner set will scale up to.
# maxRunners: 5

## minRunners is the min number of runners the auto scaling runner set will scale down to.
minRunners: 1

# runnerGroup: "default"

## name of the runner scale set to create.  Defaults to the helm release name
runnerScaleSetName: "custom-dind-set"

## A self-signed CA certificate for communication with the GitHub server can be
## provided using a config map key selector. If `runnerMountPath` is set, for
## each runner pod ARC will:
## - create a `github-server-tls-cert` volume containing the certificate
##   specified in `certificateFrom`
## - mount that volume on path `runnerMountPath`/{certificate name}
## - set NODE_EXTRA_CA_CERTS environment variable to that same path
## - set RUNNER_UPDATE_CA_CERTS environment variable to "1" (as of version
##   2.303.0 this will instruct the runner to reload certificates on the host)
##
## If any of the above had already been set by the user in the runner pod
## template, ARC will observe those and not overwrite them.
## Example configuration:
#
githubServerTLS:
  certificateFrom:
    configMapKeyRef:
      name: example-bundle-cert
      key: example_ca_bundle.crt
  runnerMountPath: /usr/local/share/ca-certificates/

containerMode:
  type: "dind"  ## type can be set to dind or kubernetes
#   ## the following is required when containerMode.type=kubernetes
#   kubernetesModeWorkVolumeClaim:
#     accessModes: ["ReadWriteOnce"]
#     # For local testing, use https://github.com/openebs/dynamic-localpv-provisioner/blob/develop/docs/quickstart.md to provide dynamic provision volume with storageClassName: openebs-hostpath
#     storageClassName: "dynamic-blob-storage"
#     resources:
#       requests:
#         storage: 1Gi

## template is the PodSpec for each runner Pod
template:
  ## template.spec will be modified if you change the container mode
  ## with containerMode.type=dind, we will populate the template.spec with following pod spec
# template:
#   spec:
#     initContainers:
#     - name: init-dind-externals
#       image: ghcr.io/actions/actions-runner:2.308.0
#       command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
#       volumeMounts:
#         - name: dind-externals
#           mountPath: /home/runner/tmpDir
#     containers:
#     - name: runner
#       image: ghcr.io/actions/actions-runner:2.308.0
#       env:
#         - name: DOCKER_HOST
#           value: tcp://localhost:2376
#         - name: DOCKER_TLS_VERIFY
#           value: "1"
#         - name: DOCKER_CERT_PATH
#           value: /certs/client
#       volumeMounts:
#         - name: work
#           mountPath: /home/runner/_work
#         - name: dind-cert
#           mountPath: /certs/client
#           readOnly: true
#     - name: dind
#       image: docker:24.0.5-dind
# #      env:            # Set Proxy to evn #INFO: not working
# #        - name: HTTP_PROXY
# #          value: "http://gw-examplenet:3000"
# #        - name: HTTPS_PROXY
# #          value: "http://gw-examplenet:3000"
# #        - name: NO_PROXY
# #          value: "example.com,localhost,127.0.0.1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/1"
#       securityContext:
#         privileged: true
#       volumeMounts:
#         - name: work
#           mountPath: /home/runner/_work
#         - name: dind-cert
#           mountPath: /certs/client
#         - name: dind-externals
#           mountPath: /home/runner/externals
#     volumes:
#     - name: work
#       emptyDir: {}
#     - name: dind-cert
#       emptyDir: {}
#     - name: dind-externals
#       emptyDir: {}
  ######################################################################################################
  ## with containerMode.type=kubernetes, we will populate the template.spec with following pod spec
  ## template:
  ##   spec:
  ##     containers:
  ##     - name: runner
  ##       image: ghcr.io/actions/actions-runner:latest
  ##       env:
  ##         - name: ACTIONS_RUNNER_CONTAINER_HOOKS
  ##           value: /home/runner/k8s/index.js
  ##         - name: ACTIONS_RUNNER_POD_NAME
  ##           valueFrom:
  ##             fieldRef:
  ##               fieldPath: metadata.name
  ##         - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
  ##           value: "true"
  ##       volumeMounts:
  ##         - name: work
  ##           mountPath: /home/runner/_work
  ##     volumes:
  ##       - name: work
  ##         ephemeral:
  ##           volumeClaimTemplate:
  ##             spec:
  ##               accessModes: [ "ReadWriteOnce" ]
  ##               storageClassName: "local-path"
  ##               resources:
  ##                 requests:
  ##                   storage: 1Gi
  spec:
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:2.308.0
      command: ["/home/runner/run.sh"]

## Optional controller service account that needs to have required Role and RoleBinding
## to operate this gha-runner-scale-set installation.
## The helm chart will try to find the controller deployment and its service account at installation time.
## In case the helm chart can't find the right service account, you can explicitly pass in the following value
## to help it finish RoleBinding with the right service account.
## Note: if your controller is installed to only watch a single namespace, you have to pass these values explicitly.
#ontrollerServiceAccount:
# namespace: arc-systems
# name: arc-gha-runner-scale-set-controller

[snps-xlwu]

Yes I faced same issue, anyone can help?

[kanumuripa]

Hi,

Facing same issue...do you guy found work around of fix for this issue??

[corrigat]

I've just run into this myself. It looks like the deployment.yaml args for actions-runner-controller contains the argument brought in from the chart's values, but the scale set deployment does not. I'm not sure if the scale set controller is capable. However, I also just found the entrypoint-dind.sh script for the runner will use a provided DOCKER_REGISTRY_MIRROR environment var, if provided, independently of the a controller configuration. I'm going to give it a shot, and will report back.

[zzswang]

I've just run into this myself. It looks like the deployment.yaml args for actions-runner-controller contains the argument brought in from the chart's values, but the scale set deployment does not. I'm not sure if the scale set controller is capable. However, I also just found the entrypoint-dind.sh script for the runner will use a provided DOCKER_REGISTRY_MIRROR environment var, if provided, independently of the a controller configuration. I'm going to give it a shot, and will report back.

Hello corrigat

Do you have tested it? Does it work?

[newdeal123]

In my case, I solved it by adding docker mirror register as a parameter.
However, currently, the helm template supporting this is not available in gha-runner-scale-set helm chart, so I changed the autoscaling runnerset manually.

- image: docker:dind
  name: dind
  args:
    - dockerd
    - '--host=unix:///run/docker/docker.sock'
    - '--group=$(DOCKER_GROUP_GID)'
    - '--registry-mirror=https://mirror.gcr.io'
  env:
    - name: DOCKER_GROUP_GID
      value: '123'
  

If this PR is merged, I think we can solve it neatly with the helm chart values.