Introducing Autoscaling Runner Scale Sets mode

[TingluoHuang]

Introducing Autoscaling Runner Scale Sets mode

Following the announcement of actions-runner-controller adoption by GitHub, we're excited to have reached the second phase of our plan by introducing the new Autoscaling Runner Scale Sets mode.

This new autoscaling mode will bring numerous enhancements (described in the following sections) that will make your experience more reliable and secure.

⚠️ This mode is currently only available for a limited number of organizations.

Private beta

While the code changes are publicly available, this new mode requires new GitHub APIs that are behind a feature flag. We would like to guarantee the quality of this release by running a private beta with a pre-selected list of organizations. These organizations have been chosen already, and we will not be including new ones in the pool for now.

As we run our tests, fix bugs, and increase our confidence with our changes we will gradually make the APIs available for you. The private beta trials will run between January - March 2023.

New features

In addition to the increased reliability of the automatic scaling, we have worked on these improvements:

• No longer require cert-manager as a prerequisite for installing actions-runner-controller
• Reliable scale-up based on job demands and scale-down to zero runner pods
• Reduce API requests to api.github.com, no more API rate-limiting problems
• The GitHub Personal Access Token (PAT) or the GitHub App installation token is no longer passed to the runner pod for runner registration
• Maximum flexibility for customizing your runner pod template

Demo

arc_autoscaling_runner_set_mode_LOWREZ.mp4

Future updates

We will continuously update this discussion with further announcements regarding this change, and we can't wait to make these features available to everyone, soon!

[jonathanbeber]

Hello, @TingluoHuang, is there a way to register for the private beta?

[macux]

@Link- is there a roadmap for future releases somewhere available? And do you have any idea when it's likely to reach GA? Thanks

[cmergenthaler]

@macux https://github.com/orgs/github/projects/4247/views/1?filterQuery=runner

[macux]

Thanks I'd already seen that but it wasn't clear to see what would happen and when - I see there is a roadmap item for self-hosted runners for Apr-Jun but it's very high level. It's also nearly July and I don't see anything after that.

[Link-]

We're still on track to GA by end of June, and might spillover to July. We have a bunch of features (like metrics) that we're wrapping up, we want to make those available before we GA. However, there's literally no reason why not to use ARC right now. It's been running stable in many environments.

[macux]

Thanks, we already are but is good to have this extra info.

[thepoppingone]

Would enterprise runners be supported? From the docs I was not able to find a way to connect as enterprise runners.

[Link-]

Yes, just set the githubConfigUrl to point to your enterprise

[ianandb]

@Link- do we have additional info on auto scaling mode setup. Thanks for demo , it will we great if we have additional steps and examples.

[Link-]

https://gh.io/arc-quickstart

[aosorio1]

Is this public yet? It says that the private beta would run until March.

I tried this on a private hosted GHE 3.7.9 and I cannot deploy AutoscalingRunnerSet resource. I used the following helm:

helm install arc-runner-set-2 \
    --namespace arc-systems \
    --set githubConfigUrl=https://git.mycompany.com/organization-a\
    --set githubConfigSecret=runner-secret \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set --version 0.4.0

Checking the logs I found out that autoscalingrunnerset_controller.go makes a call to /_apis/runtime/runnerscalesets and it's returning 404. I thought it had to do with some permission but if I use the action server url and token and make a call to /_apis/runtime/runnergroups it works. The permissions used for the GitHub App are the ones listed in the authentication with GitHub guide. I've even tried giving the GitHub App all the permissions but it didn't make any difference.

The error log of the controller:

2023-05-01T06:14:28Z	INFO	actions-clients	getting Actions tenant URL and JWT	{"registrationURL": "https://git.mycompany.com/api/v3/actions/runner-registration"}
2023-05-01T06:14:28Z	ERROR	AutoscalingRunnerSet	Failed to get runner scale set from Actions service	{"autoscalingrunnerset": "arc-systems/arc-runner-set-2", "runnerGroupId": "1", "runnerScaleSetName": "arc-runner-set-2", "error": "404 - had issue communicating with Actions backend: Request returned status: 404 Not Found"}
github.com/actions/actions-runner-controller/controllers/actions%2egithub%2ecom.(*AutoscalingRunnerSetReconciler).createRunnerScaleSet
	github.com/actions/actions-runner-controller/controllers/actions.github.com/autoscalingrunnerset_controller.go:368

[Link-]

Support for GHES will only come in version 3.9. This is not available on GHES yet.

[roshitho]

Is this general availability now ? Is there any usage/doc related to the values I need to pass. What is the equivalent to horizontal auto-scaler in - actions.summerwind.dev/v1alpha1

scaleDownDelaySecondsAfterScaleOut: 300
metrics:

• type: PercentageRunnersBusy
  scaleUpThreshold: '0.75'
  scaleDownThreshold: '0.25'
  scaleUpFactor: '2'
  scaleDownFactor: '0.5'

Thank you

[Link-]

It's in public beta. Not generally available yet.

Here are the docs: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/quickstart-for-actions-runner-controller

[roshitho]

I have seen this document and the values.yaml. I could not find where do we define the type of metrics (example - PercentageRunnersBusy) . I have seen max and min runner but not the equivalent of:

type: PercentageRunnersBusy
scaleUpThreshold: '0.75'
scaleDownThreshold: '0.25'
scaleUpFactor: '2'
scaleDownFactor: '0.5'

that we would like to use. Any documentation similar to - https://github.com/actions/actions-runner-controller/blob/master/docs/automatically-scaling-runners.md in current design. Or clarity on what should be the expectation wrt new auto-scaling mode.

Thank you for your time.

[Link-]

Metrics are not supported in the new mode, yet.

[roshitho]

One last question, given there are 2 different implementations [actions.github.com/v1alpha1] and [actions.summerwind.net/v1alpha1], can you please advise for new organizations adopting ARC which is the path forward will summerwind be maintained for a foreseeable future. Or should we go with actions.github.com/v1alpha1 ?

[Link-]

actions.github.com/v1alpha1 is the path forward :)

[simpson]

@Link- I have a simple Github workflow that terraforms an AKS Cluster in Azure, and then I try to install this but it isn't registering any runners in our org, though the workflow completes successfully. Do you have any advice on what I may be missing?
I'm also not sure how to ingest the value.yaml file

You also reference the autoscalingrunnerset.yaml file in your video but that isn't represented anywhere in the getting started doc and how to ingest that. Can we do something like kubectl apply -f manifest/autoscalingrunnerset.yaml? When I try to do what you reference I get this error:

Run kubectl apply -f manifest/autoscalingrunnerset.yaml
error: resource mapping not found for name: "arc-primary-runners" namespace: "arc-systems" from "manifest/autoscalingrunnerset.yaml": no matches for kind "AutoscalingRunnerSet" in version "actions.summerwind.dev/v1alpha1"

Sorry for all the questions I just can't seem to follow the documentation vs the video

Here is my sample Github Workflow after we terraform the AKS cluster

      - name: Install Helm
        run: |
          curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
          chmod 700 get_helm.sh
          ./get_helm.sh

      - name: Install kubectl
        run: |
          curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
          curl -LO "https://dl.k8s.io/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"
          echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check
          sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
          kubectl version --client
          kubectl cluster-info

      - name: Install actions-runner-controller
        run: |
          NAMESPACE="arc-systems"
          helm install arc \
              --namespace "${NAMESPACE}" \
              --create-namespace \
              oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller

      - name: Install autoscaling runner set
        run: |
          INSTALLATION_NAME="arc-runner-set"
          NAMESPACE="arc-runners"
          GITHUB_CONFIG_URL="https://github.com/<your_enterprise/org/repo>"
          GITHUB_PAT="${{ secrets.GH_TOKEN_ORG }}"
          helm install "${INSTALLATION_NAME}" \
              --namespace "${NAMESPACE}" \
              --create-namespace \
              --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
              --set githubConfigSecret.github_token="${GITHUB_PAT}" \
              oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

      - name: Apply autoscalingrunnerset.yaml manifest
        run: |
          kubectl apply -f manifest/autoscalingrunnerset.yaml
    
      - name: Check namespace and pods
        run: |
          sleep 30
          helm list -A
          kubectl get pods -n arc-systems

[Link-]

You don't ever have to run this: kubectl apply -f manifest/autoscalingrunnerset.yaml the helm charts should do their work and install everything necessary.

Our docs are quite detailed and should explain all the components needed: https://gh.io/arc-docs

And as a working example of workflows, you can check our end to end tests: https://github.com/actions/actions-runner-controller/blob/master/.github/workflows/e2e-test-linux-vm.yaml

[simpson]

@Link- Thank you. How do we reference values.yaml? That isn't covered here: https://gh.io/arc-docs

[Link-]

https://helm.sh/docs/chart_template_guide/values_files/

[pashcan]

@Link- Thanks for your ongoing work 🙏
Side note: e2e test are now at https://github.com/actions/actions-runner-controller/blob/master/.github/workflows/gha-e2e-tests.yaml

[Link-]

Yes, thank you @pashcan - we did some housekeeping :)

[onedr0p]

I am trying to run the public beta but it looks like the runner images are Debian?

❯ docker run --rm -it ghcr.io/actions/actions-runner:latest bash
runner@1eec5a14f330:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

This kind of breaks actions like setup-python which only work with Ubuntu Linux variant.

The error in the workflow logs:

Run actions/setup-python@57ded4d7d5e986d7296eab16560982c6dd7c923b
  with:
    python-version: 3.11.3
    check-latest: false
    token: ***
    update-environment: true
    allow-prereleases: false
Installed versions
  Version 3.11.3 was not found in the local cache
  Error: The version '3.11.3' with architecture 'x64' was not found for this operating system.
  The list of all available versions can be found here: https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json

What is the suggested route of building your own runner images now?

[simpson]

Having this same issue, haven't found a workaround yet. I wish they just used the same version of Ubuntu to make parity much easier

[onedr0p]

I can't even find the Dockerfile used to build this image in the repo. Maybe my github foo is not strong today.

[Link-]

This is the Dockerfile for the runner image we're shipping: https://github.com/actions/runner/blob/main/images/Dockerfile

You're not bound to use it, you can create your own as long as you follow this simple contract: https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/about-actions-runner-controller#creating-your-own-runner-image

I'll look into why this action is failing, it shouldn't.

[Link-]

I created an issue here, let's please not use this discussion for troubleshooting. Feel free to create another discussion and/or report these issues so that we can address them.

#2610

[furlangiacomo]

Hi, is there a way to associate the same runner to multiple repositories without creating an organization runner? githubConfigUrl seems to support only one value.

[Link-]

In short, no. You can deploy as many autoscaling runner sets as need with the same controller-manager.

[simpson]

@furlangiacomo You probably already know this but if you build out your infrastructure with terraform, it would be pretty easy to stand up as many repos or orgs as you need. I recently built out a repo that I'm planning to make public soon that sets up the new runner sets on Azure Kubernetes Service and it runs really smooth, using Spot instances makes it pretty cheap too

[furlangiacomo]

Hi, sorry to bother you with another question, we saw that our runners are logging quite a lot, I saw that there is an option to change the logLevel for the controller but I was not able to find a similar option for the runners, is something that is actually not present and maybe will be implemented in the future or I just did not find it? Thanks

[Link-]

I'm closing this discussion since this mode is now generally available: #2775