Limit Kafka Partitions KEDA operates on (#3879)

Signed-off-by: Tobias Krause <[email protected]>
Co-authored-by: Zbynek Roubalik <[email protected]>

CHANGELOG.md

@@ -38,7 +38,7 @@ To learn more about active deprecations, we recommend checking [GitHub Discussio
 
 ### New
 
-
+- **General**: **EXPERIMENTAL** Adding an option to cache metric values for a scaler during the polling interval ([#2282](https://github.com/kedacore/keda/issues/2282))
 - **General:** Introduce new CouchDB Scaler ([#3746](https://github.com/kedacore/keda/issues/3746))
 - **General**: Introduction deprecation & breaking change policy ([Governance #68](https://github.com/kedacore/governance/issues/68))
 - **General**: Consolidate all exposed Prometheus Metrics in KEDA Operator ([#3919](https://github.com/kedacore/keda/issues/3919))
@@ -47,6 +47,7 @@ To learn more about active deprecations, we recommend checking [GitHub Discussio
 - **General:** Introduce new Loki Scaler ([#3699](https://github.com/kedacore/keda/issues/3699))
 - **General:** Introduce new Etcd Scaler ([#3880](https://github.com/kedacore/keda/issues/3880))
 - **General**: Introduce rate-limitting parameters to KEDA manager to allow override of client defaults ([#3730](https://github.com/kedacore/keda/issues/3730))
+- **General**: Introduction deprecation & breaking change policy ([Governance #68](https://github.com/kedacore/governance/issues/68))
 - **General**: Provide off-the-shelf Grafana dashboard for application autoscaling ([Docs](https://keda.sh/docs/2.9/operate/prometheus/) | [#3911](https://github.com/kedacore/keda/issues/3911))
 - **General**: Provide Prometheus metric with indication of total number of custom resources per namespace for each custom resource type (CRD). ([#2637](https://github.com/kedacore/keda/issues/2637)|[#2638](https://github.com/kedacore/keda/issues/2638)|[#2639](https://github.com/kedacore/keda/issues/2639))
 - **General**: Provide Prometheus metric with indication of total number of triggers per trigger type in `ScaledJob`/`ScaledObject`. ([#3663](https://github.com/kedacore/keda/issues/3663))

CREATE-NEW-SCALER.md

@@ -20,20 +20,42 @@ If you want to deploy locally
 
 The scalers in KEDA are implementations of a KEDA `Scaler` Go interface declared in `pkg/scalers/scaler.go`. This documentation describes how scalers work and is targeted towards contributors and maintainers.
 
-### GetMetrics
+### GetMetricsAndActivity
 
-This is the key function of a scaler; it returns a value that represents a current state of an external metric (e.g. length of a queue). The return type is an `ExternalMetricValue` struct which has the following fields:
-- `MetricName`: this is the name of the metric that we are returning. The name should be unique, to allow setting multiple (even the same type) Triggers in one ScaledObject, but each function call should return the same name.
-- `Timestamp`: indicates the time at which the metrics were produced.
-- `WindowSeconds`: //TODO
-- `Value`: A numerical value that represents the state of the metric. It could be the length of a queue, or it can be the amount of lag in a stream, but it can also be a simple representation of the state.
+This is the key function of a scaler; it returns:
 
-Kubernetes HPA (Horizontal Pod Autoscaler) will poll `GetMetrics` regularly through KEDA's metric server (as long as there is at least one pod), and compare the returned value to a configured value in the ScaledObject configuration. Kubernetes will use the following formula to decide whether to scale the pods up and down:
+1. A value that represents a current state of an external metric (e.g. length of a queue). The return type is an `ExternalMetricValue` struct which has the following fields:
+    - `MetricName`: this is the name of the metric that we are returning. The name should be unique, to allow setting multiple (even the same type) Triggers in one ScaledObject, but each function call should return the same name.
+    - `Timestamp`: indicates the time at which the metrics were produced.
+    - `WindowSeconds`: //TODO
+    - `Value`: A numerical value that represents the state of the metric. It could be the length of a queue, or it can be the amount of lag in a stream, but it can also be a simple representation of the state.
+2. A value that represents an activity of the scaler. The return type is `bool`.
+
+    KEDA polls ScaledObject object according to the `pollingInterval` configured in the ScaledObject; it checks the last time it was polled, it checks if the number of replicas is greater than 0, and if the scaler itself is active. So if the scaler returns false for `IsActive`, and if current number of replicas is greater than 0, and there is no configured minimum pods, then KEDA scales down to 0.
+
+Kubernetes HPA (Horizontal Pod Autoscaler) will poll `GetMetricsAndActivity` regularly through KEDA's metric server (as long as there is at least one pod), and compare the returned value to a configured value in the ScaledObject configuration. Kubernetes will use the following formula to decide whether to scale the pods up and down:
 
 `desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )]`.
 
 For more details check [Kubernetes HPA documentation](https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/).
 
+<br>Next lines are an example about how to use it:
+>```golang
+func (s *artemisScaler) GetMetricsAndActivity(ctx context.Context, metricName string) ([]external_metrics.ExternalMetricValue, bool, error) {
+	messages, err := s.getQueueMessageCount(ctx)
+
+	if err != nil {
+		s.logger.Error(err, "Unable to access the artemis management endpoint", "managementEndpoint", s.metadata.managementEndpoint)
+		return []external_metrics.ExternalMetricValue{}, false, err
+	}
+
+	metric := GenerateMetricInMili(metricName, float64(messages))
+
+	return []external_metrics.ExternalMetricValue{metric}, messages > s.metadata.activationQueueLength, nil
+}
+>```
+
+
 ### GetMetricSpecForScaling
 
 KEDA works in conjunction with Kubernetes Horizontal Pod Autoscaler (HPA). When KEDA notices a new ScaledObject, it creates an HPA object that has basic information about the metric it needs to poll and scale the pods accordingly. To create this HPA object, KEDA invokes `GetMetricSpecForScaling`.
@@ -64,12 +86,6 @@ For example:
 >```
 
 
-### IsActive
-
-For some reason, the scaler might need to declare itself as in-active, and the way it can do this is through implementing the function `IsActive`.
-
-KEDA polls ScaledObject object according to the `pollingInterval` configured in the ScaledObject; it checks the last time it was polled, it checks if the number of replicas is greater than 0, and if the scaler itself is active. So if the scaler returns false for `IsActive`, and if current number of replicas is greater than 0, and there is no configured minimum pods, then KEDA scales down to 0.
-
 ### Close
 
 After each poll on the scaler to retrieve the metrics, KEDA calls this function for each scaler to give the scaler the opportunity to close any resources, like http clients for example.

Makefile

@@ -53,7 +53,7 @@ GO_LDFLAGS="-X=github.com/kedacore/keda/v2/version.GitCommit=$(GIT_COMMIT) -X=gi
 COSIGN_FLAGS ?= -a GIT_HASH=${GIT_COMMIT} -a GIT_VERSION=${VERSION} -a BUILD_DATE=${DATE}
 
 # ENVTEST_K8S_VERSION refers to the version of kubebuilder assets to be downloaded by envtest binary.
-ENVTEST_K8S_VERSION = 1.24
+ENVTEST_K8S_VERSION = 1.25
 
 # Setting SHELL to bash allows bash commands to be executed by recipes.
 # This is a requirement for 'setup-envtest.sh' in the test target.
@@ -147,10 +147,12 @@ proto-gen: protoc-gen ## Generate Liiklus, ExternalScaler and MetricsService pro
 	PATH="$(LOCALBIN):$(PATH)" protoc -I vendor --proto_path=pkg/metricsservice/api metrics.proto --go_out=pkg/metricsservice/api --go-grpc_out=pkg/metricsservice/api
 
 .PHONY: mockgen-gen
-mockgen-gen: mockgen pkg/mock/mock_scaling/mock_interface.go pkg/mock/mock_scaler/mock_scaler.go pkg/mock/mock_scale/mock_interfaces.go pkg/mock/mock_client/mock_interfaces.go pkg/scalers/liiklus/mocks/mock_liiklus.go
+mockgen-gen: mockgen pkg/mock/mock_scaling/mock_interface.go pkg/mock/mock_scaling/mock_executor/mock_interface.go pkg/mock/mock_scaler/mock_scaler.go pkg/mock/mock_scale/mock_interfaces.go pkg/mock/mock_client/mock_interfaces.go pkg/scalers/liiklus/mocks/mock_liiklus.go
 
 pkg/mock/mock_scaling/mock_interface.go: pkg/scaling/scale_handler.go
 	$(MOCKGEN) -destination=$@ -package=mock_scaling -source=$^
+pkg/mock/mock_scaling/mock_executor/mock_interface.go: pkg/scaling/executor/scale_executor.go
+	$(MOCKGEN) -destination=$@ -package=mock_executor -source=$^
 pkg/mock/mock_scaler/mock_scaler.go: pkg/scalers/scaler.go
 	$(MOCKGEN) -destination=$@ -package=mock_scalers -source=$^
 pkg/mock/mock_scale/mock_interfaces.go: vendor/k8s.io/client-go/scale/interfaces.go

apis/keda/v1alpha1/identifier_test.go

@@ -0,0 +1,81 @@
+package v1alpha1
+
+import (
+	"testing"
+
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+)
+
+type testData struct {
+	name               string
+	expectedIdentifier string
+	soName             string
+	soNamespace        string
+	soKind             string
+}
+
+var tests = []testData{
+	{
+		name:               "all lowercase",
+		expectedIdentifier: "scaledobject.namespace.name",
+		soName:             "name",
+		soNamespace:        "namespace",
+		soKind:             "scaledobject",
+	},
+	{
+		name:               "all uppercase",
+		expectedIdentifier: "scaledobject.namespace.name",
+		soName:             "NAME",
+		soNamespace:        "NAMESPACE",
+		soKind:             "SCALEDOBJECT",
+	},
+	{
+		name:               "camel case",
+		expectedIdentifier: "scaledobject.namespace.name",
+		soName:             "name",
+		soNamespace:        "namespace",
+		soKind:             "scaledobject",
+	},
+	{
+		name:               "missing namespace",
+		expectedIdentifier: "scaledobject..name",
+		soName:             "name",
+		soKind:             "scaledobject",
+	},
+}
+
+func TestGeneratedIdentifierForScaledObject(t *testing.T) {
+	for _, test := range tests {
+		test := test
+		t.Run(test.name, func(t *testing.T) {
+			expectedIdentifier := test.expectedIdentifier
+			genericIdentifier := GenerateIdentifier(test.soKind, test.soNamespace, test.soName)
+
+			scaledObject := &ScaledObject{
+				ObjectMeta: metav1.ObjectMeta{
+					Name:      test.soName,
+					Namespace: test.soNamespace,
+				},
+			}
+			scaledObjectIdentifier := scaledObject.GenerateIdentifier()
+
+			withTriggers, err := AsDuckWithTriggers(scaledObject)
+			if err != nil {
+				t.Errorf("got error while converting to WithTriggers object: %s", err)
+			}
+			withTriggersIdentifier := withTriggers.GenerateIdentifier()
+
+			if expectedIdentifier != genericIdentifier {
+				t.Errorf("genericIdentifier=%q doesn't equal the expectedIdentifier=%q", genericIdentifier, expectedIdentifier)
+			}
+
+			if expectedIdentifier != scaledObjectIdentifier {
+				t.Errorf("scaledObjectIdentifier=%q doesn't equal the expectedIdentifier=%q", scaledObjectIdentifier, expectedIdentifier)
+			}
+
+			if expectedIdentifier != withTriggersIdentifier {
+				t.Errorf("withTriggersIdentifier=%q doesn't equal the expectedIdentifier=%q", withTriggersIdentifier, expectedIdentifier)
+			}
+		})
+	}
+}

apis/keda/v1alpha1/scaledobject_types.go

@@ -123,7 +123,10 @@ type ScaleTarget struct {
 type ScaleTriggers struct {
 	Type string `json:"type"`
 	// +optional
-	Name     string            `json:"name,omitempty"`
+	Name string `json:"name,omitempty"`
+
+	UseCachedMetrics bool `json:"useCachedMetrics,omitempty"`
+
 	Metadata map[string]string `json:"metadata"`
 	// +optional
 	AuthenticationRef *ScaledObjectAuthRef `json:"authenticationRef,omitempty"`
@@ -179,3 +182,8 @@ type ScaledObjectAuthRef struct {
 func init() {
 	SchemeBuilder.Register(&ScaledObject{}, &ScaledObjectList{})
 }
+
+// GenerateIdentifier returns identifier for the object in for "kind.namespace.name"
+func (s *ScaledObject) GenerateIdentifier() string {
+	return GenerateIdentifier("ScaledObject", s.Namespace, s.Name)
+}

apis/keda/v1alpha1/withtriggers_types.go

@@ -17,6 +17,7 @@ limitations under the License.
 package v1alpha1
 
 import (
+	"fmt"
 	"time"
 
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -37,7 +38,8 @@ type WithTriggers struct {
 	metav1.TypeMeta   `json:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty"`
 
-	Spec WithTriggersSpec `json:"spec"`
+	InternalKind string           `json:"internalKind"`
+	Spec         WithTriggersSpec `json:"spec"`
 }
 
 // WithTriggersSpec is the spec for a an object with triggers resource
@@ -89,5 +91,35 @@ func (t *WithTriggers) GetPollingInterval() time.Duration {
 
 // GenerateIdentifier returns identifier for the object in for "kind.namespace.name"
 func (t *WithTriggers) GenerateIdentifier() string {
-	return GenerateIdentifier(t.Kind, t.Namespace, t.Name)
+	return GenerateIdentifier(t.InternalKind, t.Namespace, t.Name)
+}
+
+// AsDuckWithTriggers tries to generates WithTriggers object for input object
+// returns error if input object is unknown
+func AsDuckWithTriggers(scalableObject interface{}) (*WithTriggers, error) {
+	switch obj := scalableObject.(type) {
+	case *ScaledObject:
+		return &WithTriggers{
+			TypeMeta:     obj.TypeMeta,
+			ObjectMeta:   obj.ObjectMeta,
+			InternalKind: "ScaledObject",
+			Spec: WithTriggersSpec{
+				PollingInterval: obj.Spec.PollingInterval,
+				Triggers:        obj.Spec.Triggers,
+			},
+		}, nil
+	case *ScaledJob:
+		return &WithTriggers{
+			TypeMeta:     obj.TypeMeta,
+			ObjectMeta:   obj.ObjectMeta,
+			InternalKind: "ScaledJob",
+			Spec: WithTriggersSpec{
+				PollingInterval: obj.Spec.PollingInterval,
+				Triggers:        obj.Spec.Triggers,
+			},
+		}, nil
+	default:
+		// here could be the conversion from unknown Duck type potentially in the future
+		return nil, fmt.Errorf("unknown scalable object type %v", scalableObject)
+	}
 }

config/crd/bases/keda.sh_scaledjobs.yaml

@@ -7874,6 +7874,8 @@ spec:
                       type: string
                     type:
                       type: string
+                    useCachedMetrics:
+                      type: boolean
                   required:
                   - metadata
                   - type

config/crd/bases/keda.sh_scaledobjects.yaml

@@ -276,6 +276,8 @@ spec:
                       type: string
                     type:
                       type: string
+                    useCachedMetrics:
+                      type: boolean
                   required:
                   - metadata
                   - type

controllers/keda/hpa_test.go

@@ -137,7 +137,6 @@ func setupTest(health map[string]v1alpha1.HealthStatus, scaler *mock_scalers.Moc
 				return scaler, &scalers.ScalerConfig{}, nil
 			},
 		}},
-		Logger:   logr.Discard(),
 		Recorder: nil,
 	}
 	metricSpec := v2.MetricSpec{

controllers/keda/scaledjob_controller.go

@@ -173,6 +173,9 @@ func (r *ScaledJobReconciler) reconcileScaledJob(ctx context.Context, logger log
 	}
 
 	for _, trigger := range scaledJob.Spec.Triggers {
+		if trigger.UseCachedMetrics {
+			logger.Info("Warning: property useCachedMetrics is not supported for ScaledJobs.")
+		}
 		if trigger.MetricType != "" {
 			err := fmt.Errorf("metricType is set in one of the ScaledJob scaler")
 			logger.Error(err, "metricType cannot be set in ScaledJob triggers")

controllers/keda/scaledobject_controller.go

@@ -221,7 +221,7 @@ func (r *ScaledObjectReconciler) reconcileScaledObject(ctx context.Context, logg
 		return "ScaledObject doesn't have correct Idle/Min/Max Replica Counts specification", err
 	}
 
-	err = r.checkTriggerNamesAreUnique(scaledObject)
+	err = r.checkTriggers(scaledObject)
 	if err != nil {
 		return "ScaledObject doesn't have correct triggers specification", err
 	}
@@ -337,14 +337,24 @@ func (r *ScaledObjectReconciler) checkTargetResourceIsScalable(ctx context.Conte
 	return gvkr, nil
 }
 
-// checkTriggerNamesAreUnique checks that all triggerNames in ScaledObject are unique
-func (r *ScaledObjectReconciler) checkTriggerNamesAreUnique(scaledObject *kedav1alpha1.ScaledObject) error {
+// checkTriggers checks that general trigger metadata are valid, it checks:
+// - triggerNames in ScaledObject are unique
+// - useCachedMetrics is defined only for a supported triggers
+func (r *ScaledObjectReconciler) checkTriggers(scaledObject *kedav1alpha1.ScaledObject) error {
 	triggersCount := len(scaledObject.Spec.Triggers)
 
 	if triggersCount > 1 {
 		triggerNames := make(map[string]bool, triggersCount)
 		for i := 0; i < triggersCount; i++ {
-			name := scaledObject.Spec.Triggers[i].Name
+			trigger := scaledObject.Spec.Triggers[i]
+
+			if trigger.UseCachedMetrics {
+				if trigger.Type == "cpu" || trigger.Type == "memory" || trigger.Type == "cron" {
+					return fmt.Errorf("property \"useCachedMetrics\" is not supported for %q scaler", trigger.Type)
+				}
+			}
+
+			name := trigger.Name
 			if name != "" {
 				if _, found := triggerNames[name]; found {
 					// found duplicate name

go.mod

@@ -35,7 +35,6 @@ require (
 	github.com/gobwas/glob v0.2.3
 	github.com/gocql/gocql v1.3.0
 	github.com/golang/mock v1.6.0
-	github.com/golang/protobuf v1.5.2
 	github.com/google/go-cmp v0.5.9
 	github.com/google/uuid v1.3.0
 	github.com/gophercloud/gophercloud v1.1.0
@@ -64,6 +63,7 @@ require (
 	github.com/xdg/scram v1.0.5
 	github.com/xhit/go-str2duration/v2 v2.0.0
 	github.com/youmark/pkcs8 v0.0.0-20201027041543-1326539a0a0a
+	go.etcd.io/etcd/client/v3 v3.5.4
 	go.mongodb.org/mongo-driver v1.11.0
 	golang.org/x/oauth2 v0.2.0
 	google.golang.org/api v0.103.0
@@ -161,6 +161,7 @@ require (
 	github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
 	github.com/golang-sql/sqlexp v0.1.0 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
 	github.com/golang/snappy v0.0.4 // indirect
 	github.com/google/gnostic v0.6.9 // indirect
 	github.com/google/go-querystring v1.1.0 // indirect
@@ -251,7 +252,6 @@ require (
 	github.com/xlab/treeprint v1.1.0 // indirect
 	go.etcd.io/etcd/api/v3 v3.5.4 // indirect
 	go.etcd.io/etcd/client/pkg/v3 v3.5.4 // indirect
-	go.etcd.io/etcd/client/v3 v3.5.4 // indirect
 	go.opencensus.io v0.24.0 // indirect
 	go.opentelemetry.io/contrib v0.20.0 // indirect
 	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.20.0 // indirect