Extend tests for Observability stack

[jsmolar]

Summary

Observability automation should include tracing, metrics, dashboards, and alerts, for both Authorino/Limitador standalone and Kuadrant.
Currently, the test suite implements Authorino tracing (supports the usage of Jaeger and Tempo as tracing backends) and metrics.

Documentation

Observability docs: https://docs.kuadrant.io/kuadrant-operator/doc/observability/examples/

Test coverage

Authorino standalone

• Metrics
• Tracing

Kuadrant

• Metrics
• Tracing - Waiting for Add some global way (in the Kuadrant CR?) to configure the underlying components for them to push spans to the OTLP collector kuadrant-operator#731 PR ready Add Kuadrant tracing tests #458
• Dashboards
• Alerts

Authorino

• Metrics - Closed by Removed standalone-only for authornino metrics tests #499
• Tracing - This will be tested in Kuadrant when a global way of config is completed

Limitador

• Metrics - Close by Add test for limitador metrics #493
• Tracing - This will be tested in Kuadrant when a global way of config is completed

[david-martin]

• Investigate and implement new tests for dashboards using Grafana

@jsmolar I wonder if it would be sufficient as an initial test of dashboards to deploy them with kustomize (mounted via configmap volumes), then use the dashboard api to verify they exist in grafana after it is ready?

A similar approach could be used for the example alert rules with the prometheus http api.

That, at the very least, should verify they are valid and are imported ok.