Pre-Populated Docker Cache

[harshaisgud]

The process to setup a docker cache described in the repository does not satisfy my orgs requirement.
I have also followed the discussions extensively and found a few interesting approaches like #1286 (comment)

Requirements for docker cache

• Every runner requires a clean environment with a pre-populated cache.
• Clean docker cache with no image artefacts from previous builds.

Listed Steps have been performed on EKS(1.23) but they should work with any Kubernetes flavour implementing the CSI spec such as GKE.

Pre-requisites

To use the snapshot functionality of the CSI driver, you must install the external snapshotter.

Please refer the following documentation for

• EKS(https://aws.amazon.com/blogs/containers/using-amazon-ebs-snapshots-for-persistent-storage-with-your-amazon-eks-cluster-by-leveraging-add-ons/)
• GKE(https://cloud.google.com/kubernetes-engine/docs/how-to/persistent-volumes/volume-snapshots)

Once the pre-requisites have been satisfied run the following steps.

1. Create a StorageClass with RetentionPolicy set to Delete.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gh-ebs
provisioner: ebs.csi.aws.com
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

2. Create RunnerSet

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: example-1
spec:
  labels: 
    - arc-1
    - linux
  selector:
    matchLabels:
      app: example
  serviceName: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: docker
        volumeMounts:
        - name: var-lib-docker
          mountPath: /var/lib/docker
  volumeClaimTemplates:
  - metadata:
      name: var-lib-docker
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 7Gi
      storageClassName: gh-ebs

3. Create a job pulling the images required for the docker cache. Add a step to sleep at the end of the job to facilitate the next step. The sleep is required because actions-runner-controller deletes the pod after a run and PVC is recreated.

name: 'Docker Cache Test'
on:
  push:
    branches:
      - master

jobs:

  docker-test:
    name: 'Test docker'
    runs-on: [self-hosted, linux, arc-1]
    steps:
      - name: Pull docker image
        run: docker pull traefik:latest
      - name: Check docker image
        run: docker image ls
      - name: Sleep
        run: sleep 100

5. Create Volume Snapshot and wait for it to be ready. Remember to replace the pvc name with the name of pvc created in previous step.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: ebs-volume-snapshot
spec:
  volumeSnapshotClassName: csi-aws-vsc
  source:
    persistentVolumeClaimName: {replace-pvc-name-from-previous-step}

6. Check Volume Snapshot Status

k get vs
NAME                  READYTOUSE   SOURCEPVC                        SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                    CREATIONTIME   AGE
ebs-volume-snapshot   true         var-lib-docker-example-1-mdr9d-0                           7Gi           csi-aws-vsc     snapcontent-590a6994-9896-4876-8cd6-f27bb0a5163c   98m            100m

7. Update the runnerSet VolumeClaimTemplate section such that volumes are populated from a snapshot.

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: example-1
spec:
  labels: 
    - arc-1
    - linux
  selector:
    matchLabels:
      app: example
  serviceName: example
  template:
    metadata:
      labels:
        app: example
    spec:
      containers:
      - name: docker
        volumeMounts:
        - name: var-lib-docker
          mountPath: /var/lib/docker
  volumeClaimTemplates:
  - metadata:
      name: var-lib-docker
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 7Gi
      dataSource:
        name: ebs-volume-snapshot
        kind: VolumeSnapshot
        apiGroup: snapshot.storage.k8s.io

8. Check if docker cache has been successful. The docker cache should be populated from a snapshot everytime.

I am sure this can be made much more elegant and I am open to suggestions 😄

[totogtr]

Hi @harshaisgud, thanks for this process.
Does it scale well with new volumes also populated from the snapshot ? If yes do you scale with an hra ?

I'm trying to reproduce your setup but the PVC created from the volumeClaimTemplates doesn't contain anything while a manually created PVC using the the snapshot as a datasource works fine. I guess something is wrong with the EBS CSI controller somewhere.

[harshaisgud]

@totogtr Thank you for trying this method.It scales perfectly with HRA.
I guess the reason you didn't find anything in the first pass is because ARC uses PV that match the labels and there was the initial PV left over which did not have the image. Once that PV is deleted, every new PV will get populated with the cache.
Like I said it is a little janky. :)
Also I have been facing problems with ReclaimPolicy set to delete and Ive had to had a cronjob to delete all the un used PVs every hour.

[totogtr]

Thanks for the explanation. I've tried to remove that initial PV but every new PVC is created without the datasource block, even if it is in my RunnerSet manifest. I am not sure where this issue can come from.

[totogtr]

Well after some tries, I can achieve a working set-up in another EKS cluster.
The PVC gets created with

  spec:
    accessModes:
    - ReadWriteOnce
    dataSource:
      apiGroup: snapshot.storage.k8s.io
      kind: VolumeSnapshot
      name: docker-vs
    dataSourceRef:
      apiGroup: snapshot.storage.k8s.io
      kind: VolumeSnapshot
      name: docker-vs
    resources:
      requests:
        storage: 14500Mi
    storageClassName: gh-ebs
    volumeMode: Filesystem
    volumeName: pvc-010a9626-d41f-4e87-b224-1e642fc58e1a

However, on the initial cluster the PVC gets created with :

  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 14500Mi
    storageClassName: gh-ebs
    volumeMode: Filesystem
    volumeName: pvc-ba0f6674-d0d7-490b-8873-8a9666569b6d

I deployed the same RunnerSet manifest to both clusters, except targeting repository on the set-up that works and organization on the one that doesn't. (Also tried with dataSource)

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerSet
metadata:
  name: cache-dock
spec:
  replicas: 1
  organization: MYORG # or repository: myrepo
  labels:
    - arc-1
    - linux
  selector:
    matchLabels:
      app: cache1
  serviceName: cache
  template:
    metadata:
      labels:
        app: cache1
    spec:
      containers:
        - name: docker
          volumeMounts:
            - name: docker-cache2
              mountPath: /var/lib/docker
  volumeClaimTemplates:
    - metadata:
        name: docker-cache2
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 14500Mi
        storageClassName: gh-ebs
        dataSourceRef:
          kind: VolumeSnapshot
          name: docker-vs
          apiGroup: snapshot.storage.k8s.io

The RunnerSet and StatefulSet do have a properly defined volumeClaimTemplates block :

    volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        creationTimestamp: null
        name: docker-cache2
      spec:
        accessModes:
        - ReadWriteOnce
        dataSourceRef:
          apiGroup: snapshot.storage.k8s.io
          kind: VolumeSnapshot
          name: docker-vs
        resources:
          requests:
            storage: 14500Mi
        storageClassName: gh-ebs
        volumeMode: Filesystem
      status:
        phase: Pending

So not really sure what is missing here. The EBS CSI driver is more recent on the cluster with the not working set-up and nothing obvious shows in the logs as the CSI controller is always succesful in creating a PVC... it just seems to ignore the dataSourceRef. If anyone has a clue what to look for that would be very useful.

[harshaisgud]

Strange I dont seem to have any issues with this setup. I am using organizational runners.

[totogtr]

Well it works fine on a newly created cluster, not such what is wrong with the other one. It could be due to an old EKS version with upgrade or some components not up to date probably.

[JohnYoungers]

I'm trying a slightly different approach where I'm hoping to use the existing runners to create the snapshot like so in a github workflow:

docker system prune --all --volumes --force

# pull images we'd like to cache
docker pull ....

# giving docker a moment to finalize pulls: without this, the images above will be in a partially saved state, and result in a `too many levels of symbolic links` error
sleep 30

# get volume id for running docker volume
volumeId=$(aws ec2 describe-volumes --filters Name=tag:kubernetes.io/created-for/pvc/name,Values=$HOSTNAME-var-lib-docker --query "Volumes[0].VolumeId" --output text)

# create snapshot
aws ec2 create-snapshot --volume-id $volumeId --description 'GitHub Runner docker volume (build ${{ github.run_number }})' >> $GITHUB_STEP_SUMMARY

With the snapshot id from that workflow run, I can populate the following resources:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: ebs-csi-aws-vsc
  namespace: github-runners
driver: ebs.csi.aws.com
deletionPolicy: Delete
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotContent
metadata:
  name: var-lib-docker-content
  namespace: github-runners
spec:
  volumeSnapshotRef:
    kind: VolumeSnapshot
    name: var-lib-docker-snapshot
    namespace: github-runners
  source:
    snapshotHandle: snap-.....
  driver: ebs.csi.aws.com
  deletionPolicy: Retain
  volumeSnapshotClassName: ebs-csi-aws-vsc
---
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: var-lib-docker-snapshot
  namespace: github-runners
spec:
  volumeSnapshotClassName: ebs-csi-aws-vsc
  source:
    volumeSnapshotContentName: var-lib-docker-content

That all seems to work; however, there must be some sort of issue with the snapshot itself: In my workflows that use docker (for service containers for example), they're currently throwing this error: Error response from daemon: error creating overlay mount to /var/lib/docker/overlay2/...-init/merged: too many levels of symbolic links.

I'll update this comment if I find the resolution

edit: issues resolved (see comment)

[JohnYoungers]

Looping back to this: it seems like it's just certain images that are impacted. If I snapshot after pulling busybox from dockerhub and mssql/server from mcr.microsoft.com, after restore the tag is missing for mssql/server:

REPOSITORY                       TAG       IMAGE ID       CREATED       SIZE
mcr.microsoft.com/mssql/server   <none>    e7fc0b49be3c   4 weeks ago   1.47GB
busybox                          latest    a416a98b71e2   6 weeks ago   4.26MB

using busybox works fine, but attempting to use the mssql/server image results in the too many levels of symbolic links error.

[JohnYoungers]

After further testing I noticed it was just the last image that didn't have a tag: it looks like docker is still doing work as the snapshot is being taken

Adding a sleep prior to snapshot creation seemed to fix it