Micrometer-bridge breaks Spring Actuator metrics #12719
Comments
santerivairio
added
bug
|
maybe we need to add also I noticed that the Spring Boot Starter uses a different |
|
I don't know why |
|
here's the definition of the prometheus auto config: https://github.com/spring-projects/spring-boot/blob/main/spring-boot-project/spring-boot-actuator-autoconfigure/src/main/java/org/springframework/boot/actuate/autoconfigure/metrics/export/prometheus/PrometheusMetricsExportAutoConfiguration.java @AutoConfiguration(
before = { CompositeMeterRegistryAutoConfiguration.class, SimpleMetricsExportAutoConfiguration.class },
after = MetricsAutoConfiguration.class)
@ConditionalOnBean(Clock.class)
@ConditionalOnClass(PrometheusMeterRegistry.class)
@ConditionalOnEnabledMetricsExport("prometheus")
@EnableConfigurationProperties(PrometheusProperties.class)
public class PrometheusMetricsExportAutoConfiguration {(all registries are defined this way) compared to what we use @Configuration
// CompositeMeterRegistryAutoConfiguration configures the "final" composite registry
@AutoConfigureBefore(CompositeMeterRegistryAutoConfiguration.class)
// configure after the SimpleMeterRegistry has initialized; it is normally the last MeterRegistry
// implementation to be configured, as it's used as a fallback
// the OTel registry should be added in addition to that fallback and not replace it
@AutoConfigureAfter(SimpleMetricsExportAutoConfiguration.class)
@ConditionalOnBean(Clock.class)
@ConditionalOnClass(MeterRegistry.class)
public class OpenTelemetryMeterRegistryAutoConfiguration {I think the issue is that SimpleMetricsExportAutoConfiguration is absent when prometheus is configured - which makes the order upredictable |
Describe the bug
Using javaagent and spring actuator with micrometer bridge enabled breaks sometimes the /metrics -endpoint.
In those cases it seems like the OpenTelemetryMetricsRegistry is created before PrometheusMetricsRegistry.
In those cases an error stating
io.opentelemetry.javaagent.shaded.instrumentation.micrometer.v1_5.OpenTelemetryMeterRegistry - OpenTelemetry metrics bridge does not support reading measurementsis shown.But it some cases the same exact implementation works after restarting the application. In those cases the PrometheusMetricsRegistry is before the OpenTelemetry one in the registries-Set
We're using custom micrometer metrics that need to be bridged for the OpenTelemetry agent too.
Seems like the issue is that spring-actuator's metrics endpoint implementation uses the first registry that has the wanted metric in it. But because OpenTelemetryMeterRegistry is unreadable, the response is empty (no measurement value). Here's the cause of this issue.
We managed to fix this by overriding the MetricsEndpoint-implementation with a one that excludes OpenTelemetryMetricsRegistry. But in my opinion it would be nicer to have it fixed somewhere else, here perhaps?
Steps to reproduce
spring-boot-starter-actuator 3.2.5
opentelemetry-javaagent 2.9.0
micrometer bridge enabled
query metrics through actuator
/actuator/metrics/<metric name>Expected behavior
No errors, wouldn't require booting until the registries are in correct order
Actual behavior
io.opentelemetry.javaagent.shaded.instrumentation.micrometer.v1_5.OpenTelemetryMeterRegistry - OpenTelemetry metrics bridge does not support reading measurementsis shown.But it some cases the same exact implementation works after restarting the application. In those cases the PrometheusMetricsRegistry is before the OpenTelemetry one in the registries-Set
Javaagent or library instrumentation version
v2.9.0
Environment
JDK: Temurin 17.0.12
OS: eclipse-temurin:17-jre-alpine
Spring Boot 3.2.5
Additional context
No response
The text was updated successfully, but these errors were encountered: