WG - Observability.Next

[brunobat]

Working group to define the strategy around the Micrometer Observability API and OTEL bridge.

Overview

Observability.Next is a strategic initiative to modernize the observability stack within the Quarkus framework, aiming to streamline and enhance its ability to monitor and manage distributed systems. Currently, Quarkus (3.16.x) leverages the following components for observability:
Metrics: Managed through Micrometer, which supports multiple registries (e.g., Prometheus) to expose application metrics.
Tracing: Handled via the OpenTelemetry (OTEL) SDK, enabling distributed tracing data to be exported to an OTEL collector.
Logging: JBoss's LogManager with a handler forwarding logs with OTEL.

While OpenTelemetry is the industry standard for observability, both in protocol and format, the OTEL SDK has proven challenging to adapt due to its design for agent-based observability. In a framework like Quarkus, more granular and direct control over observability is required to monitor application performance, reliability, and user experience effectively.

Micrometer, the de-facto Java standard for metrics, has recently proposed an enhanced, global approach to observability with the new Observation API. Expanding Micrometer’s role in Quarkus’ observability stack represents an opportunity to simplify implementation and maintenance while preserving compatibility with leading observability tools.

Objectives

The Observability.Next initiative seeks to strengthen the integration of Micrometer’s evolving Observability API within Quarkus, aligning it with OpenTelemetry’s protocol standards. Key objectives include:

• Enhanced Observability Control: Implement a framework-native, Micrometer-centric approach to metrics, traces, and logs, ensuring developers have more refined control over observability components without the limitations of the OTEL SDK.
• Broader Metrics and Tracing Compatibility: Continue exposing metrics to various registries, including Prometheus, while maintaining OTEL compatibility for metrics and tracing protocols.
• Improved Performance and Reliability: Define and measure performance baselines of the current observability setup and compare them against the next-to-be Micrometer Observation API and OTEL bridge integration to ensure tangible performance and reliability improvements.
• Continue to support components using Micrometer metrics API or Open Telemetry SDK (like Pulsar)

Scope

Observability.Next will unfold across multiple phases, each handled by specialised working groups. The scope of the initial working group will include:

• Micrometer Observation API Integration: Introduce and integrate Micrometer’s observability API as a foundational layer
• Transformation to OTEL Protocols: Develop a mechanism to translate Micrometer-collected observations into OTEL-compatible metrics, traces, and logs, ensuring smooth interoperability with OTEL-based tools and collectors.
• Performance Benchmarking: Establish a performance baseline for Quarkus’ current observability model, comparing it against the new Micrometer-based observability API and OTEL bridge to quantify gains and identify areas for further optimisation.

Expected Outcomes

• Improved Developer Experience: More intuitive, refined control over observability for Quarkus developers, reducing complexity and overhead associated with setting up and maintaining observability.
• Ecosystem Compatibility: Continued compatibility with existing observability tools such as Prometheus and OTEL collectors, ensuring interoperability with common enterprise monitoring solutions.
• Less breakage due to the OTEL SDK
• More control on the semantic convention in span and metric tags
• Performance Gains: Potentially lower latency and resource consumption in observability operations, contributing to Quarkus’ goal of high-performance, cloud-native applications.

Some concrete features we would get at the framework level:

• Instrument once, plug different frameworks later
• Better support of scopes than the one we have today. See:
  • OpenTelemetry Scope not closed correctly #25102
  • Duplicated OTel spans on the new grpc server implementation. #37140
• Auto-generate metrics documentation from source code instrumented with it.
• A way to manage semantic conventions

Users would benefit from:

• A way to manage their metrics schemas.
• Yet a new framework support, if they want to use it.

---

Point of contact: @brunobat

Related Reading:

• Micrometer, Observation API and OpenTelemetry roadmap: https://groups.google.com/g/quarkus-dev/c/y5-ojIVsa_M/m/4wquJi4bBQAJ

[brunobat]

CC @ebullient @radcortez @alesj @cescoffier @maxandersen @geoand @gsmet

[ppalaga]

cc @jamesnetherton @zhfeng

[brunobat]

CC @jponge @roberttoyonaga @tsegismont @ahus1 @michalvavrik @loicmathieu @mcruzdev @danielsoro

[geoand]

both in protocol and format

Can we elaborate on what exactly this means?

has proven challenging to adapt due to its design for agent-based observability

What are the exact challenges we have faced and how does the Micrometer Observability API aleviate our current pain?

strengthen the integration of Micrometer’s evolving Observability API within Quarkus, aligning it with OpenTelemetry’s protocol standards

What does this mean in practice?

Define and measure performance baselines of the current observability setup and compare them against the next-to-be Micrometer Observation API and OTEL bridge integration to ensure tangible performance and reliability improvements.

Definitely +1 - I have no long argued that we need performance metrics / monitoring for a realilstic / production oriented stack, not just benchmark oriented demos.

Improved Developer Experience: More intuitive, refined control over observability for Quarkus developers, reducing complexity and overhead associated with setting up and maintaining observability.

How so?

Performance Gains: Potentially lower latency and resource consumption in observability operations, contributing to Quarkus’ goal of high-performance, cloud-native applications.

Why do we expect this to be an outcome?

[geoand]

Instrumenting Quarkus and libraries code with it reduces the exposure to OpenTelemetry APIs.

But opens up exposure to Micrometer Observability API

Provides an abstraction layer to semantic conventions. Many can be used for the same instrument. This allows us to control when we change those conventions. OTel has no abstraction for this.

Very interesting indeed.

Instrument once with the Observation API and produce telemetry for multiple frameworks sounds a better aproach.

Sure, but that assumes that Observation API is (or will become) the defacto standard - something which does not seem to be the case today.

Please note, that users would be able to use whatever they like: Micrometer, OpenTelemetry or the Observation API.

Of course, it's very important to make this distinction - the ask here is that Quarkus itself will have Micrometer observation all over its code.

If we do performance tests and measure, it will give us the tools for improvement, however, there is no guarantee.
We need to define a process where we can measure things, make improvements and eventually define a threshold to a performance penalty we are willing to take for this, if needed.

Right. The reason I brought it up is because the way it was written, it sounded like the proposed move to the Observation would improve performance, when in practice we don't know - there are probably the same chances of hurting performance as improving it.

[brunobat]

Instrument once with the Observation API and produce telemetry for multiple frameworks sounds a better aproach.

Sure, but that assumes that Observation API is (or will become) the defacto standard - something which does not seem to be the case today.

I don't think we should care if it becomes a de-facto standard or not. It's "just" a utility we can use and has the features we need. What we should really care is if it's stable and maintained. I think it is, but other people should look into it as well.
The fact that is being used all over Spring to instrument their code, provides some guarantees.

[geoand]

I don't think we should care if it becomes a de-facto standard or not

It does if you consider the part the argument above that stated:

Right now we have a proliferation of custom instrumentation APIs to deal with many frameworks. Think LangChain4J that created a custom API to produce telemetry for Spring and Quarkus. Think SR Fault Tolerance that needs to deal with MP Metrics, Micrometer and now OpenTelemetry Metrics. Think of all the new instrumentation we need to implement OTel Metrics in Quarkus.
What will happen if a new framework shown up?

I am not saying the tool isn't good - I am saying that unless everyone starts using it, I don't see it reducing the mental burden of us producing instrumentation in the places we need.

[brunobat]

Right, but it still prevent's us from having multiple concrete instrumentations spread all over the codebase.

[geoand]

Definitely

[geoand]

I think one thing we need to keep in mind about this whole discussion is that the proposed move will make it easier for us to maintain observability features - it won't on its own provide any new features for users

[brunobat]

Correct. These are some of the features we would get at the framework level:

• Instrument once, plug different frameworks later
• Better support of scopes than the one we have today. See:
  • OpenTelemetry Scope not closed correctly
  • Duplicated OTel spans on the new grpc server implementation
• Auto-generate metrics documentation from source code instrumented with it.
• A way to manage semantic conventions

Users would benefit from:

• A way to manage their metrics schemas.
• Yet a new framework support, if they want to use it.

[cescoffier]

@brunobat, Can you include this in the first post?

[brunobat]

Sure. Done.

[ahus1]

Thank you @brunobat for summoning me here. Here some of my thoughts from a Keycloak perspective:

• As Keycloak, we want to provide Observability to our users with metrics, logs and traces.
• Incoming traffic is HTTP, and outgoing traffic JPA/JDBC, HTTP and LDAP.
• For tracing, we are shipping in Keycloak 26 a preview support with incoming HTTP and JPA/JDBC. We have a custom HTTP client that we still need to instrument, and for LDAP we need to wrap it with custom traces. Same for internal workings of Keycloak that we would need to manually instrument. I agree the OTEL wire protocol for tracing simplifies things in that area. At the same time, we have the first people asking for direct support of the GCP Quarkiverse explorter to be included, which we are hesitant to provide - see OTEL: Quarkiverse exporter for GCP not working keycloak/keycloak#34799.
• We have one library (Infinispan HotRod) that auto-detected OTEL tracing, and adds its spans to that. We would like that to continue working when the Observability.Next continues its work. In other words, different libraries would used different APIs, still things should work when it comes to tracing.
• For our internal instrumentation, we're about to leverage the OTEL SDK APIs. The current approach attempts to wrap things in a central place, and I hope we don't need to change much if Quarkus chooses to promote different APIs. Again, it would be good to use those different APIs in parallel at runtime.
• I'm a big fan of exposing metrics via Prometheus/OpenMetrics, as the one scraping the metrics can (as a trusted party) add the necessary context about where it scraped them from. I haven't figured out how that can work with OTEL metrics that are then pushed. I assume you would need to leverage workload identity mechanism to identify the source of the traffic (aka authenticate the source to the sink) and then add those information at the sink. I assume this applies for traces and logs as well.
• For any push based system IMHO one would need to implement some means of congestion control to prevent an unbounded sender of either metrics logs or traces to overwhelm the receiver. With a pull-based metrics like Prometheus, that comes for free.
• While logging to the console in containers using JSON has been a thing for a while, and I like its asynchronous nature that the information is captured even if the remote system is currently unavailable. I know that people want to use remote logging, possibly with a more structured way. We started to deprecate GELF as the logstash-gelf library was deprecated. We then started to promote remote Syslog. And I've seen the first people asking for OTEL logging as a remote logging option.
• For custom metrics, we're now leveraging the Micrometer API which - as Bruno mentioned - is IMHO a defacto standard in Java. So I assume we're good with whatever route you take here.
• In terms of API stability, we want to provide users of Keycloak non-breaking upgrades between minor releases of Keycloak. We started to align Keycloak minor releases with Quarkus minor releases, at least on its external interfaces. This would be metric names as people would use them in dashboards, and we don't want them to break in minor releases. I wouldn't consider changes in the traces and their labels to be breaking in any kind, as I would use them to explore issues in a specific release. Happy to receive feedback on that. With logging, I could imagine people setting up auditing and alerting on events as they appear in the log, so people might depend on stable labels here. For those people writing extensions for Keycloak, they might also want stable APIs. While we might be reluctant to promise stable APIs while Opentelementry.Next unfolds, we would need to manage expectations in some kind of way.

So TL;DR:

• I'm curious how this will evolve, and I'm happy to provide feedback on implementations.
• I would be interested to learn more how any push-based observability can be combined with workload identity. I didn't find much on this in my Google search, and at least one questions is unanswered here.

cc: @vmuzikar, @mabartos

[brunobat]

I'm finishing a PoC that was forgotten for a while. Will reply afterwards.

[cescoffier]

@brunobat do you have a list of issues that would be associated with this working group?

[brunobat]

Not yet, @cescoffier. Should I create a tag for these types of issues?

[cescoffier]

No label or specific tag, we will associate them with the project once created (so keep a list ;-))

[loicmathieu]

The observation API will not become a standard, so developers should still be able to create traces using the standard OTEL SDK (the OpenTelemetry object).

[brunobat]

Correct

[tsegismont]

@brunobat This proposal sounds good to me.

I don't believe it will be necessary but do you foresee any change required in the Vert.x metrics and tracing SPIs?