Keda Metric APIServer failing to connect with Keda Operator with connection refused issue

[sinanub18]

After deploying Keda v2.9.0 helm chart the logs of Keda Metric APIServer shows "failed to connect with Keda Operator"

Expected Behavior

The Metric APIServer should establish connection

Actual Behavior

W0110 10:32:24.748466 1 logging.go:59] [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {
"Addr": "keda-operator.operator.svc.cluster.local:9666",
"ServerName": "keda-operator.operator.svc.cluster.local:9666",
"Attributes": null,
"BalancerAttributes": null,
"Type": 0,
"Metadata": null
}. Err: connection error: desc = "transport: Error while dialing dial tcp xx.xxx.xxx.xxx:9666: connect: connection refused"

Steps to Reproduce the Problem

1. Deploy Keda 2.9.0 Helm chart on kubernetes cluster
2. check logs of metric apiserver pod.

Specifications

• KEDA Version: 2.9.0
• Platform & Version: Minikube & GCP
• Kubernetes Version: v1.23.5
• Scaler(s): Any Scaler

After deployment if the Metric APIServer Pod is deleted then new pod doesn't show this issue.

Logs of new metric apiserver pod

I0110 10:42:28.973759 1 listener.go:44] keda_metrics_adapter/controller-runtime/metrics "msg"="Metrics server is starting to listen" "addr"=":8080"
I0110 10:42:28.974730 1 main.go:162] keda_metrics_adapter "msg"="Connecting Metrics Service gRPC client to the server" "address"="keda-operator.operator.svc.cluster.local:9666"
2023/01/10 10:42:28 Starting metrics server at :9022

[zroubalik]

It is an internal gRPC message when it tries to connect. It appears because Metrics Server Pod is Ready while Operator Pod is still in the creation phase. It automatically tries to reconnect, unfortunately it doesn't print a message in case of successful connection.

This is just not so nice warning message, scaling is working correctly.

We can try to make something about this though.

[sinanub18]

Yes the successful connection establish log is missing, I even increased the initialDelay of Metric Server to resolve this but unfortunately no luck, but if scaling works correctly then all good.

I would recommend to add a log once the connection gets established.

[zroubalik]

Let's keep this open to track that we should improve the log messages.

[heavenwing]

I believe that reason is operator deploy don't expose 9666 port, but operator service and Keda Metric APIServer use 9666 port to connect it.