cwm-keda-external-scaler

CWM KEDA external scaler scales the target resrouce based on the metrics fetched from the Redis server. The metrics are generated by the aggregated MinIO logs and pushed to the Redis server by the cwm-worker-logger using fluent-plugin-http-cwm as its underlying logging engine. The Redis server stores a number of metrics. Based on its local configuration, the external scaler fetches only the required one(s) to make its scaling decisions.

Flow Diagram

                                  CONFIGURATION (global and local)
                                ------------------------------------
                                Env Variables: { CWM_REDIS_HOST, ... }
        {metrics}               ScaledObject : { deploymentid, ... }
            |                                  |
            |                                  |
            |                                  |
            v                                  v
  +-------------------+              +-------------------+
  |                   |   {metric}   |                   |
  |    Redis Server   |------------->|  External Scaler  |
  |                   |              |                   |
  +-------------------+              +-------------------+
                                               |
                                               |
                                               v
                                     +-------------------+
                                     |                   |
                                     |     Kubernetes    |
                                     |                   |
                                     +-------------------+
                                               |
                                               |  scale
                                               v
                                     +-------------------+
                                     |                   |
                                     |  Target Resource  |
                                     |                   |
                                     +-------------------+

Configuration

The external scaler listens on port 50051.

Global Configuration: Environment Variables

Environment Variable	Default Value	Description
CWM_REDIS_HOST	localhost	ip/host of the Redis metrics server
CWM_REDIS_PORT	6379	port of the Redis metrics server
CWM_REDIS_DB	0	Redis database to use
LAST_UPDATE_PREFIX	deploymentid:last_action	prefix for last update key
METRICS_PREFIX	deploymentid:minio-metrics	prefix for metrics key

Local Configuration: Metadata in ScaledObject

Here is the generic YAML format of a ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: <scaledobject-name>
  namespace: <scaledobject-namespace>
spec:
  scaleTargetRef:
    name: <deployment-name>
  triggers:
    - type: external
      metadata:                       # <<< local configuration >>> #
        scalerAddress: <host:port>
        key1: <value1>
        key2: <value2>

The local configuration will be under metadata:

# ...
spec:
  # ...
  triggers:
    - type: external
      metadata:
        # <<< local confiugration >>>

The following table lists the supported local configuration:

Configuration (Key)	Default Value	Description
deploymentid	minio	value to append to the prefix
isActiveTtlSeconds	600	seconds since last update for workload to be active
scaleMetricName	bytes_out	metric for scaling (listed below)
scalePeriodSeconds	600	retention time for the metric value
targetValue	10	target value reported by the autoscaler

Here are the supported options for scaleMetricName:

Metric Name	Description
bytes_in	number of incoming bytes
bytes_out	number of outgoing bytes (default)
num_requests_in	number of incoming requests (WebUpload, PutObject, DeleteObject)
num_requests_out	number of outgoing requests (WebDownload, GetObject)
num_requests_misc	number of requests other than input and output
bytes_total	bytes_in + bytes_out
num_requests_in_out	num_requests_in + num_requests_out
num_requests_total	num_requests_in + num_requests_out + num_requests_misc

Sample Configuration

Here's the configuration format of a ScaledObject:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: {scaled-object-name}
spec:
  scaleTargetRef:
    apiVersion:    {api-version-of-target-resource}   # Optional. Default: apps/v1
    kind:          {kind-of-target-resource}          # Optional. Default: Deployment
    name:          {name-of-target-resource}          # Mandatory. Must be in the same namespace as the ScaledObject
    envSourceContainerName: {container-name}          # Optional. Default: .spec.template.spec.containers[0]
  pollingInterval: 30                                 # Optional. Default: 30 seconds
  cooldownPeriod:  300                                # Optional. Default: 300 seconds
  minReplicaCount: 0                                  # Optional. Default: 0
  maxReplicaCount: 100                                # Optional. Default: 100
  advanced:                                           # Optional. Section to specify advanced options
    restoreToOriginalReplicaCount: true/false         # Optional. Default: false
    horizontalPodAutoscalerConfig:                    # Optional. Section to specify HPA related options
      behavior:                                       # Optional. Use to modify HPA's scaling behavior
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
          - type: Percent
            value: 100
            periodSeconds: 15
  triggers:
  # {list of triggers to activate scaling of the target resource}

Assuming that the global configuration via environment variables has properly been set, the external scaler (cwm-keda-external-scaler) can be configured as a ScaledObject under triggers like this:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name:                     {scaled-object-name}
spec:
  scaleTargetRef:
    name:                   {name-of-target-resource}
  pollingInterval: 10
  triggers:
    - type: external
      metadata:
        scalerAddress:      {host:port}               # Mandatory.
        deploymentid:       {deployment-id}           # Optional. Default: deploymentid
        isActiveTtlSeconds: {seconds}                 # Optional. Default: 600
        scaleMetricName:    {supported-metric-name}   # Optional. Default: bytes_out
        scalePeriodSeconds: {seconds}                 # Optional. Default: 600
        targetValue:        {target-value}            # Optional. Default: 10

Build Docker Image

docker build -t cwm-keda-external-scaler:latest .

Testing

Prerequisite

Software	Version
minikube	v1.12.3
Kubernetes	v1.16.4
KEDA	v2.1.0

Make sure that KEDA is already deployed and running (use v2.1.0 via YAML file):

# apply
kubectl apply -f https://github.com/kedacore/keda/releases/download/v2.1.0/keda-2.1.0.yaml

# delete
kubectl delete -f https://github.com/kedacore/keda/releases/download/v2.1.0/keda-2.1.0.yaml

Deploy

Terminal-1: Watch resources in all the namespaces

watch -x kubectl get all --all-namespaces

Terminal-2: Apply test deployment

kubectl apply -f ./test/deploy.yaml

Terminal-3: Check logs of pod/keda-operator-* in keda namespace

kubectl logs -f -n keda pod/keda-operator-*

Terminal-4: Check logs of pod/keda-operator-metrics-apiserver-* in keda namespace

kubectl logs -f pod/keda-operator-metrics-apiserver-* -n keda

Terminal-5: Check logs of the custom external scaler cwm-keda-external-scaler

kubectl logs -f -n cwm-keda-external-scaler-ns pod/cwm-keda-external-scaler-* cwm-keda-external-scaler

NOTE: The trailing * in above pod/<pod-name>-* format denotes the actual complete name of the pod.

Run Tests

Start minikube:

minikube start --driver=docker --kubernetes-version=v1.16.14

Run:

./test/run_scaling_tests.sh

For details, please navigate to test subdirectory.

Contribute

• Fork the project.
• Check out the latest main branch.
• Create a feature or bugfix branch from main.
• Commit and push your changes.
• Make sure to add tests.
• Test locally.
• Submit the PR.

License

MIT