Monitoring and alerting for private environments

[weatherhog]

Context

Our customer Talanx wants to have an environment which uses a central HTTP(S) Proxy so that they have a central point to block traffic if they need to.

Access between MCs and WCs will be done via VPC endpoints. No peering or public endpoints. Access to or from the outside will only be possible via their Transit Gateway (network and routing will be setup by the kaas team). Egress must go through a proxy.

see https://github.com/giantswarm/talanx/issues/26 and #1424

Goal

We need to make sure our current monitoring and alerting setup works as expected for this environment. This implies both that our monitoring works on this new CAPA provider, and that our components support using a proxy for outgoing traffic.

1. Ensure monitoring works on CAPA
   • Alerts are being sent out #1604
   • Alerts can be silenced #1605
   • Job have targets #1606
   • Dashboards have data
   • Data are available in Grafana Cloud
2. Ensure monitoring works behind a proxy

TODOs

• CAPA (private environments) - Check Prometheus down #1485
• CAPA (private environments) - Listing all Network Calls we have #1460
• CAPA (private environment) - Prometheus to Grafana Cloud #1461
• CAPA (private environment) - Alertmanager to Opsgenie #1462
• CAPA (private environement)- Auth0 check if the callback works #1463
• CAPA (private environment)- Alertmanager to slack #1464
• CAPA - Workloadcluster needs to call ingress on Managementcluster #1465
• CAPA (private environment) - Silences need to be pulled via cronjob from Github (also Prometheus Rules) #1466
• https://github.com/giantswarm/giantswarm/issues/24321
• https://github.com/giantswarm/giantswarm/issues/24355
• CAPA (private environments) - prometheus-meta-operator #1703
• CAPA (private environment) - prometheus-agent #1831
• Update network call list at https://docs.giantswarm.io/security/domain-allowlist/

CAPA Test installations are on Golem

[QuentinBisson]

@TheoBrigitte can you duplicate the issues so we have them for capa and capa private envs?

[TheoBrigitte]

Testing environment

goat MC for CAPA private environment is ready for testing, but ⚠️ storage is not ready yet ⚠️

region: eu-north-1
installation name: goat
Its currently uses CUSTOM BRANCHES for most of the main components (capa-app-collection,config , management-cluster-fleet so if you need to change something you need to use these branches.)
Its using self-signed private certificates for ingress endpoints like dex, athena so if you wanna use opsctl login goat you need to add custom CA to trusted CA on your OS, for now this is manual work - see attached CA cert in the thread
The DNS is "public" so you should be able to resolve it anywhere but the IPs are private so you need a VPN in order to interact with the cluster.
SSH is not working yet
ALL pods running on MC should have injected ENVs variables with proxy configuration, but this means the app needs to support this (edited)

custom CA

-----BEGIN CERTIFICATE-----
MIIBeDCCAR6gAwIBAgIQcOnvOyWdUuUqpLsMNL5tiDAKBggqhkjOPQQDAjAcMRow
GAYDVQQDExFnaWdhbnRpYy5pbnRlcm5hbDAeFw0yMjExMjUxMzI5MzBaFw0yMzAy
MjMxMzI5MzBaMBwxGjAYBgNVBAMTEWdpZ2FudGljLmludGVybmFsMFkwEwYHKoZI
zj0CAQYIKoZIzj0DAQcDQgAEOl9V9jyNQGa/sU3u1u/+aJzwd3QsqcX+6yRX+x5T
JCJu42JGr5SMS/ADKZ4r1jJlR1VaOVQc44qFcr32uukxGKNCMEAwDgYDVR0PAQH/
BAQDAgKkMA8GA1UdEwEB/wQFMAMBAf8wHQYDVR0OBBYEFOuLlRMgSfzNiufdQobp
dvsOy3pYMAoGCCqGSM49BAMCA0gAMEUCIQCMrhseZjoIBbgrWfQN9zmyZLgNj16H
9jxviHyTKRt0wwIgDSuGg5D7k2ui24RB7cOqCn+XURTuHGeWfmq2nnLLrPQ=
-----END CERTIFICATE-----

source: https://gigantic.slack.com/archives/C04AJ5FJHEK/p1669392073238199

Proxy interface

When running behind a proxy, it is required for outgoing traffic to go through the proxy to reach out to the public internet. This is achieved by injecting proxy environment variables into pods. Application then need to make sure those are supported where needed.

Overall concept comes from Golang httpproxy library
RFC (still a pull request): https://github.com/giantswarm/rfc/pull/50/files
Example PR (how Proxy environment variables are used): https://github.com/giantswarm/kiam-app/pull/127/files
App require secret to be enabled, example: https://github.com/giantswarm/default-apps-aws/blob/master/helm/default-apps-aws/values.yaml#L129

Addition

For any additional info reach out to : #wg-private-capa-testing

[QuentinBisson]

@TheoBrigitte are we done here?

[TheoBrigitte]

Yes we are done here.