-
Notifications
You must be signed in to change notification settings - Fork 5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[kube-prometheus-stack] TLS handshake error: client sent an HTTP request to an HTTPS server #4898
Comments
You'd have to determine which pod that IP belongs to assuming it is a client on the pod network. Prometheus operator gets regularly accessed by two client groups only: Prometheus when scraping its metrics endpoint and kube-api-server when communicating with the webhook. If you enable TLS in Prometheus operator, its service monitor gets adjusted for TLS so that Prometheus scrapes over TLS with https client. As to the webhook, kube-api-server refuses not to communicate over TLS, so that it always is a https client. See whether you can find that client's IP address amongst pods' IP addresses, e.g. with a command like this: kubectl get pod \
-o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,IP:status.podIP' Depending on your permissions, you can apply it on your monitoring namespace or cluster wide ( |
Describe the bug a clear and concise description of what the bug is.
I did a fresh checkout of "kube-prometheus-stack" helm chart and setup on AWS
EKS
cluster. All pods are running fine. I set agent mode asagentMode: true
invalues.yaml
file.It can scrape pods metrics to Grafana. Everything works as expected except one error I observed in operator pod logs as following.
This tls handshake error keeps coming. Not sure what that
ip:port
is?Another tls error was also there before this tls error. So, to fix that one I added below change in values.yaml file under the
kubEtcd
ServiceMonitor component and worked.serviceMonitor: tlsConfig: insecureSkipVerify: true
However, this tls error stated in the screenshot above is still clueless. It would be immense help if someone could provide any input. Thank you!
Do let me know if the information is not sufficient. Please excuse me for the format, posting for the first time.
What's your helm version?
version.BuildInfo{Version:"v3.15.2", GitCommit:"1a500d5625419a524fdae4b33de351cc4f58ec35", GitTreeState:"clean", GoVersion:"go1.22.4"}
What's your kubectl version?
Client Version: v1.29.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.16-eks-a737599
Which chart?
kube-prometheus-stack in agent mode
What's the chart version?
63.1.0
What happened?
I did a fresh checkout of "kube-prometheus-stack" helm chart and setup on AWS
EKS
cluster. All pods are running fine. I set agent mode asagentMode: true
invalues.yaml
file.It can scrape pods metrics to Grafana. Everything works as expected except one error I observed in operator pod logs as following.
This tls handshake error keeps coming. Not sure what that
ip:port
is?Another tls error was also there before this tls error. So, to fix that one I added below change in values.yaml file under the
kubEtcd
ServiceMonitor component and worked.serviceMonitor: tldConfig: insecureSkipVerify: true
However, this tls error stated in the screenshot above is still clueless. It would be immense help if someone could provide any input. Thank you!
Do let me know if information is not sufficient. Please excuse me for the format, posting for the first time.
What you expected to happen?
No response
How to reproduce it?
No response
Enter the changed values of values.yaml?
No response
Enter the command that you execute and failing/misfunctioning.
helm install kube-prometheus-stack <chart.yaml directory>
Anything else we need to know?
No response
The text was updated successfully, but these errors were encountered: