E2E Testing for Prometheus Stack/Monitoring

[mjnagel]

E2E testing for Prometheus should only be run on the all package test only since it focuses on interactions between components - see #211 for some thoughts on this.

Initial thoughts on testing areas:

• Alertmanager alerts firing: At minimum there is a watchdog alert that should be firing when the connection between prometheus and alertmanager is functional (this may require a UI interaction or setting up a receiver in CI for alerts?)
• Prometheus metrics endpoints healthy: This will touch on other apps being properly configured as well, but we should validate that all metrics targets are healthy in prometheus (likely easy to do via API call)
• Validate UI accessibility: Both prometheus and alertmanager have UIs that are not exposed via Istio, but can be useful for debugging. We should validate that these are accessible in cluster (via svc address/port-forward)