Skip to content

Latest commit

 

History

History
147 lines (114 loc) · 4.88 KB

README.md

File metadata and controls

147 lines (114 loc) · 4.88 KB
Raw

Gradle Build Cache Node StatefulSet Deployment in AKS

This project provides a Kubernetes StatefulSet deployment for Gradle Build Cache Nodes in Azure Kubernetes Service (AKS). The setup includes two replicas of the build cache node, a service to expose the cache, an ingress controller for external access, and AzureFile for persistent storage.

Prerequisites

  • Azure Kubernetes Service (AKS) cluster
  • Kubernetes CLI (kubectl)
  • Azure CLI (az)
  • Docker CLI

Project Structure

  • gradle-cache-stateful-set.yaml: Defines the StatefulSet for deploying Gradle Build Cache Nodes.
  • service.yaml: Defines the Kubernetes Service on port 5071.
  • ingress.yaml: Defines the Ingress configuration for external access.
  • config.yaml: Custom configuration file for user authentication. Masked by .gitignore. You should have own copy of that.

Setup Instructions

Note: The manifests are expected to be applied in the build-cache namespace. You can change this as needed.

1. Create Azure File Share

  1. Create a storage account:

    az storage account create --resource-group <resource-group> --name <storage-account-name> --location <location> --sku Standard_LRS

  2. Create a file share:

    az storage share create --account-name <storage-account-name> --name <file-share-name>

  3. Get storage account key:

    az storage account keys list --resource-group <resource-group> --account-name <storage-account-name>

  4. Create Kubernetes secret:

    kubectl create secret generic azure-build-cache-secret --from-literal=azurestorageaccountname=<storage-account-name> --from-literal=azurestorageaccountkey=<storage-account-key> -n build-cache

2. Deploy StatefulSet and Service

  1. Apply StatefulSet:

    kubectl apply -f gradle-cache-stateful-set.yaml

  2. Apply Service:

    kubectl apply -f service.yaml

3. Configure Ingress

  1. Apply Ingress: 
    kubectl apply -f ingress.yaml

4. User Authentication Setup

  1. Generate salted hash for password:

    docker run --interactive --tty gradle/build-cache-node:19.0 hash

  2. Create config.yaml with user credentials: This is complete file example

    version: 5
# Restrict UI access
uiAccess:
    type: "secure"
    username: "<UI access user>"
    password: "<salted-hash passwrod>"
# List users to read/write cache
cache:
    accessControl:
    anonymousLevel: "read"
    users:
        <user name/role>: # e.g developer, reader etc.
            password: "<salted-hash passwrod>"
            level: "readwrite"
            note: "Developer to read/write cache"

  3. Apply config to StatefulSet:

    The details of the applying custom configuration for build cache node is here

    • Make a secret for config.yaml file: 
      kubectl create secret generic gradle-build-cache-config-secret -n build-cache --from-file=config.yaml
    • You will use it as a volume to mount to InitialContainer 
      ...
  initContainers:
  - name: config-mounter
    image: "busybox:1.33.0"
    command: [
          "sh",
          "-ce",
          "cp /tmp/config.yaml /data/conf/config.yaml" ]
    volumeMounts:
      - name: tmp-build-cache-config-file
          mountPath: /tmp
      - name: build-cache-config-dir
          mountPath: /data/conf
...
  - name: tmp-build-cache-config-file
      secret:
      secretName: gradle-build-cache-config-secret

Accessing the Build Cache Node

  1. Find the external IP of your ingress:

    kubectl get ingress -n build-cache

  2. Access the Gradle Build Cache Node UI:

    • Navigate to http://<external-ip>:5071 in your browser.

Benefits of Using AzureFile

  • Durability: AzureFile ensures that your data is highly available and durable.
  • Simplicity: Easy to use and manage within the Azure ecosystem.
  • Flexibility: Can be used by multiple pods concurrently.

Custom Configuration

Custom configurations are necessary to handle user authentication for pushing cache from a Gradle build. The default Gradle setup poses challenges for saving changes through the UI. To configure a user later, create a special config.yaml file and provide a hash-salted password.

Troubleshooting

  • Ensure all Kubernetes resources are correctly applied.
  • Verify the external IP and DNS settings for ingress.
  • Check the StatefulSet and Service status using kubectl get statefulsets and kubectl get services in build-cache namespace.

Contributions

Feel free to contribute by submitting issues or pull requests.