feat: add monitoring stack

[sxd]

Every cluster now has the podMonitor enabled and the script scripts/setup_monitoring.sh was created to deploy Prometheus Operator and Grafana Operator, using these operators we create:

• Prometheus instance
• Grafana Deployment
• Grafana Datasource (connecting to Prometheus instance)
• Grafana Dashboard (from the project cloudnative-pg/grafana-dashboard)

After the cluster are updated the Grafana dashboard can be accessed redirecting the 3000 port of the Grafana deployment (namespace grafana) on any of the two kind clusters.

The dashboard is loaded from the URL of the json file in the repository, this means that if the content is changed also the dashboard will be updated.

[smiyc]

I tested the setup_monitoring.sh and it failed because my laptop didn't had kustomize installed.
Therefor I would add kustomize to the prereqs in the setup.sh script or add a prereq check into setup_monitoring.sh

[gbartolini]

I would like to keep the requirements to a minimal level, without requiring development tools. We shouldn't forget that the audience are not developers.

[smiyc]

so instead of kustomize, kubectl apply -f < the directory >?

[sxd]

@smiyc I did some minor changes and removed the needed of kustomize, can you check again please! :D

[smiyc]

@sxd the monitoring stack is set up correctly without kustomize.
The cloudnativepg-dashboard GrafanaDashboard is applied, but I don't see it in grafana

[sxd]

@smiyc it should be in the playlists section, can you check please? :D

[smiyc]

@smiyc it should be in the playlists section, can you check please? :D

nope

[sxd]

@smiyc will create a fresh environment to test, and check, but in the meantime, can you check the status of the grafana operator ? looking for errors or something.
None of the command failed right?

[smiyc]

@sxd

the operator wants to download the dashboard from my external IP address...

│ 2024-10-07T20:37:58Z    ERROR    GrafanaDashboardReconciler    error fetching dashboard    {"controller": "grafanadashboard", "controllerGroup": "grafana.integreatly.org", "controllerKind": "GrafanaDashboard", "GrafanaDashboard": {"name":"cloudnativepg-dashboard","namespace":"grafana"}, "namespace": "grafana", "na │
│ me": "cloudnativepg-dashboard", "reconcileID": "818bb948-ce37-4bdb-8507-9b5730b23cb8", "dashboard": "cloudnativepg-dashboard", "error": "dial tcp XXX:443: i/o timeout"}   

a nslookup raw.githubusercontent.com from inside the pod as well outside docker is looking good

...after connecting via hotspot, the dashboard was downloaded...
so the problem is in my setup, nevermind

[gbartolini]

@sxd can you please review this? please do not merge as we need to add Leonardo as co author.

[sxd]

@gbartolini on it!! will modify the commit message too ;) BTW nice changes!