Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs]OSPP-Add more observability in apollo config client #5229

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
224 changes: 224 additions & 0 deletions docs/en/client/java-sdk-user-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -414,6 +414,26 @@ The configuration methods, in descending order of priority, are
3. Via the `app.properties` configuration file
* You can specify `apollo.override-system-properties=true` in `classpath:/META-INF/app.properties`

### 1.2.4.9 Monitor-related Configuration

> Applicable for version 2.4.0 and above
Starting from version 2.4.0, the observability of the client has been enhanced. Users can obtain the client status information directly through ConfigService by accessing ConfigMonitor and report the status as metrics to monitoring systems. Below are some related configurations.

`apollo.client.monitor.enabled`: Enables the Monitor mechanism, i.e., whether ConfigMonitor is activated. The default is false.

`apollo.client.monitor.jmx.enabled`: Exposes Monitor data in JMX format. If enabled, tools like J-console and Jprofiler can be used to view the relevant information. The default is false.

![Monitor Configuration](https://cdn.jsdelivr.net/gh/Rawven/image@main/2024-08-24-14-59-01-image.png)

`apollo.client.monitor.exception-queue-size`: Sets the maximum number of exceptions that the Monitor can store. The default value is 25.

`apollo.client.monitor.external.type`: **Non-standard configuration item**, used to activate the corresponding monitoring system's Exporter when exporting metric data. For example, if the apollo-plugin-client-prometheus is introduced, "prometheus" can be specified to enable it. The values available for configuration depend on the MetricsExporter SPI introduced (official or custom implementations). This design allows for easy extensibility. If multiple, incorrect, or no values are set, no Exporter will be enabled.

For specific usage, see the section on Extension Development - Java Client Access to Different Monitoring Systems.

`apollo.client.monitor.external.export-period`: The Exporter exports status information (e.g., thread pools) from the Monitor and converts it into metric data through scheduled tasks. The export-period controls the frequency of these scheduled tasks. The default is 10 seconds.


# II. Maven Dependency

Apollo's client jar package has been uploaded to the central repository, the application only needs to be introduced in the following way when it is actually used.
Expand Down Expand Up @@ -450,6 +470,8 @@ Apollo supports API approach and Spring integration approach, how to choose whic

* For more interesting practical usage scenarios and sample code, please refer to [apollo-use-cases](https://github.com/ctripcorp/apollo-use-cases)



## 3.1 API Usage

The API approach is the easiest and most efficient way to use Apollo configuration without relying on the Spring Framework to use it.
Expand Down Expand Up @@ -519,6 +541,197 @@ String someNamespace = "test";
ConfigFile configFile = ConfigService.getConfigFile("test", ConfigFileFormat.XML);
String content = configFile.getContent();
```
### 3.1.5 Using the Monitor Feature

In version 2.4.0, apollo-client significantly enhanced observability by providing the ConfigMonitor API and metrics export options for JMX and Prometheus.

To enable the monitor feature, you need to configure apollo.client.monitor.enabled to true.


```yaml
apollo:
client:
monitor:
enabled: true
```

#### 3.1.5.1 Exposing Status Information via JMX

Enable apollo.client.monitor.jmx.enabled in the configuration.

```yaml
apollo:
client:
monitor:
enabled: true
jmx:
enabled: true
```

After starting the application, you can view it using J-console or J-profiler; here, we use J-profiler as an example.

![](https://raw.githubusercontent.com/Rawven/image/main/20240828003803.png)

#### 3.1.5.2 Exporting Metrics via Prometheus

Configure apollo.client.monitor.external.type to prometheus in the configuration (make sure to include apollo-client-plugin-prometheus and necessary dependencies!).
```yaml
apollo:
client:
monitor:
enabled: true
external:
type: prometheus
```

You can retrieve ExporterData via ConfigMonitor (the format depends on your configured monitoring system) and then expose the endpoint to Prometheus.
示例代码

```java
@RestController
@ResponseBody
public class TestController {

@GetMapping("/metrics")
public String metrics() {
ConfigMonitor configMonitor = ConfigService.getConfigMonitor();
return configMonitor.getExporterData();
}
}
```

After starting the application, let Prometheus listen to this interface, and you will see request logs with a similar format.
```
# TYPE apollo_client_thread_pool_queue_remaining_capacity gauge
# HELP apollo_client_thread_pool_queue_remaining_capacity apollo gauge metrics
apollo_client_thread_pool_queue_remaining_capacity{thread_pool_name="RemoteConfigRepository"} 2.147483647E9
apollo_client_thread_pool_queue_remaining_capacity{thread_pool_name="AbstractApolloClientMetricsExporter"} 2.147483647E9
apollo_client_thread_pool_queue_remaining_capacity{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_queue_remaining_capacity{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_thread_pool_core_pool_size gauge
# HELP apollo_client_thread_pool_core_pool_size apollo gauge metrics
apollo_client_thread_pool_core_pool_size{thread_pool_name="RemoteConfigRepository"} 1.0
apollo_client_thread_pool_core_pool_size{thread_pool_name="AbstractApolloClientMetricsExporter"} 1.0
apollo_client_thread_pool_core_pool_size{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_core_pool_size{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_thread_pool_largest_pool_size gauge
# HELP apollo_client_thread_pool_largest_pool_size apollo gauge metrics
apollo_client_thread_pool_largest_pool_size{thread_pool_name="RemoteConfigRepository"} 1.0
apollo_client_thread_pool_largest_pool_size{thread_pool_name="AbstractApolloClientMetricsExporter"} 1.0
apollo_client_thread_pool_largest_pool_size{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_largest_pool_size{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_thread_pool_queue_size gauge
# HELP apollo_client_thread_pool_queue_size apollo gauge metrics
apollo_client_thread_pool_queue_size{thread_pool_name="RemoteConfigRepository"} 2.0
apollo_client_thread_pool_queue_size{thread_pool_name="AbstractApolloClientMetricsExporter"} 0.0
apollo_client_thread_pool_queue_size{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_queue_size{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_thread_pool_pool_size gauge
# HELP apollo_client_thread_pool_pool_size apollo gauge metrics
apollo_client_thread_pool_pool_size{thread_pool_name="RemoteConfigRepository"} 1.0
apollo_client_thread_pool_pool_size{thread_pool_name="AbstractApolloClientMetricsExporter"} 1.0
apollo_client_thread_pool_pool_size{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_pool_size{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_namespace_item_num gauge
# HELP apollo_client_namespace_item_num apollo gauge metrics
apollo_client_namespace_item_num{namespace="application"} 8.0
apollo_client_namespace_item_num{namespace="application1"} 2.0
# TYPE apollo_client_thread_pool_completed_task_count gauge
# HELP apollo_client_thread_pool_completed_task_count apollo gauge metrics
apollo_client_thread_pool_completed_task_count{thread_pool_name="RemoteConfigRepository"} 2.0
apollo_client_thread_pool_completed_task_count{thread_pool_name="AbstractApolloClientMetricsExporter"} 0.0
apollo_client_thread_pool_completed_task_count{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_completed_task_count{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_namespace_not_found gauge
# HELP apollo_client_namespace_not_found apollo gauge metrics
apollo_client_namespace_not_found 0.0
# TYPE apollo_client_thread_pool_total_task_count gauge
# HELP apollo_client_thread_pool_total_task_count apollo gauge metrics
apollo_client_thread_pool_total_task_count{thread_pool_name="RemoteConfigRepository"} 4.0
apollo_client_thread_pool_total_task_count{thread_pool_name="AbstractApolloClientMetricsExporter"} 1.0
apollo_client_thread_pool_total_task_count{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_total_task_count{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_namespace_usage counter
# HELP apollo_client_namespace_usage apollo counter metrics
apollo_client_namespace_usage_total{namespace="application"} 1.0
apollo_client_namespace_usage_created{namespace="application"} 1.725899226271E9
apollo_client_namespace_usage_total{namespace="application1"} 1.0
apollo_client_namespace_usage_created{namespace="application1"} 1.72589922627E9
# TYPE apollo_client_thread_pool_maximum_pool_size gauge
# HELP apollo_client_thread_pool_maximum_pool_size apollo gauge metrics
apollo_client_thread_pool_maximum_pool_size{thread_pool_name="RemoteConfigRepository"} 2.147483647E9
apollo_client_thread_pool_maximum_pool_size{thread_pool_name="AbstractApolloClientMetricsExporter"} 2.147483647E9
apollo_client_thread_pool_maximum_pool_size{thread_pool_name="AbstractConfigFile"} 2.147483647E9
apollo_client_thread_pool_maximum_pool_size{thread_pool_name="AbstractConfig"} 2.147483647E9
# TYPE apollo_client_namespace_first_load_time_spend_in_ms gauge
# HELP apollo_client_namespace_first_load_time_spend_in_ms apollo gauge metrics
apollo_client_namespace_first_load_time_spend_in_ms{namespace="application"} 99.0
apollo_client_namespace_first_load_time_spend_in_ms{namespace="application1"} 40.0
# TYPE apollo_client_thread_pool_active_task_count gauge
# HELP apollo_client_thread_pool_active_task_count apollo gauge metrics
apollo_client_thread_pool_active_task_count{thread_pool_name="RemoteConfigRepository"} 0.0
apollo_client_thread_pool_active_task_count{thread_pool_name="AbstractApolloClientMetricsExporter"} 1.0
apollo_client_thread_pool_active_task_count{thread_pool_name="AbstractConfigFile"} 0.0
apollo_client_thread_pool_active_task_count{thread_pool_name="AbstractConfig"} 0.0
# TYPE apollo_client_namespace_timeout gauge
# HELP apollo_client_namespace_timeout apollo gauge metrics
apollo_client_namespace_timeout 0.0
# EOF
```

You can also see similar information on the Prometheus console.

![](https://raw.githubusercontent.com/Rawven/image/main/20240922125033.png)

#### 3.1.5.3 Manually Calling the ConfigMonitor API to Retrieve Related Data (for example, when users need to manually process data to report to the monitoring system)

```java
ConfigMonitor configMonitor = ConfigService.getConfigMonitor();
// Error related monitoring API
ApolloClientExceptionMonitorApi exceptionMonitorApi = configMonitor.getExceptionMonitorApi();
List<Exception> apolloConfigExceptionList = exceptionMonitorApi.getApolloConfigExceptionList();
// Namespace related monitoring API
ApolloClientNamespaceMonitorApi namespaceMonitorApi = configMonitor.getNamespaceMonitorApi();
List<String> namespace404 = namespaceMonitorApi.getNotFoundNamespaces();
// Startup parameter related monitoring API
ApolloClientBootstrapArgsMonitorApi runningParamsMonitorApi = configMonitor.getRunningParamsMonitorApi();
String bootstrapNamespaces = runningParamsMonitorApi.getBootstrapNamespaces();
// Thread pool related monitoring API
ApolloClientThreadPoolMonitorApi threadPoolMonitorApi = configMonitor.getThreadPoolMonitorApi();
ApolloThreadPoolInfo remoteConfigRepositoryThreadPoolInfo = threadPoolMonitorApi.getRemoteConfigRepositoryThreadPoolInfo();
```

#### 3.1.5.4 Metrics Data Table

## Namespace Metrics

| Metric Name | Tags |
|-----------------------------------------------------|-----------|
| apollo_client_namespace_usage_total | namespace |
| apollo_client_namespace_item_num | namespace |
| apollo_client_namespace_not_found | |
| apollo_client_namespace_timeout | |
| apollo_client_namespace_first_load_time_spend_in_ms | namespace |

## Thread Pool Metrics

| Metric Name | Tags |
| -------------------------------------------------- | ---------------- |
| apollo_client_thread_pool_pool_size | thread_pool_name |
| apollo_client_thread_pool_maximum_pool_size | thread_pool_name |
| apollo_client_thread_pool_largest_pool_size | thread_pool_name |
| apollo_client_thread_pool_completed_task_count | thread_pool_name |
| apollo_client_thread_pool_queue_remaining_capacity | thread_pool_name |
| apollo_client_thread_pool_total_task_count | thread_pool_name |
| apollo_client_thread_pool_active_task_count | thread_pool_name |
| apollo_client_thread_pool_core_pool_size | thread_pool_name |
| apollo_client_thread_pool_queue_size | thread_pool_name |

## Exception Metrics

| Metric Name | Tags |
| --------------------------------- | ---- |
| apollo_client_exception_num_total | |

## 3.2 Spring integration approach

Expand Down Expand Up @@ -1282,3 +1495,14 @@ The interface is `com.ctrip.framework.apollo.spi.ConfigServiceLoadBalancerClient
The Input is multiple ConfigServices returned by meta server, and the output is a ConfigService selected.

The default service provider is `com.ctrip.framework.apollo.spi.RandomConfigServiceLoadBalancerClient`, which chooses one ConfigService from multiple ConfigServices using random strategy .

## 7.2 MetricsExporter Extension

> from version 2.4.0
To meet users' varying system requirements for metrics exporting (e.g., Prometheus, Skywalking) when using apollo-client, we have provided an SPI in the enhanced observability of version 2.4.0.

The interface is com.ctrip.framework.apollo.monitor.internal.exporter.ApolloClientMetricsExporter.

We offer an abstract class for a general metrics export framework, com.ctrip.framework.apollo.monitor.internal.exporter.AbstractApolloClientMetricsExporter. You just need to extend this class and implement the relevant methods to customize integration with other monitoring systems. You can refer to the implementation of apollo-plugin-client-prometheus.

By default, there are no services provided, which means no metrics data will be exported.
141 changes: 141 additions & 0 deletions docs/en/extension/java-client-how-to-use-custom-monitor-system.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
In version 2.4.0 and above of the Java client, support for metrics collection and export has been added, allowing users to extend and integrate with different monitoring systems.

## Taking Prometheus Integration as an Example

Create the `PrometheusApolloClientMetricsExporter` class, which extends `AbstractApolloClientMetricsExporter` (the generic metrics export framework).

The code after extending is roughly as follows:

```java

public class PrometheusApolloClientMetricsExporter extends
AbstractApolloClientMetricsExporter implements ApolloClientMetricsExporter {

@Override
public void doInit() {

}

@Override
public boolean isSupport(String form) {

}


@Override
public void registerOrUpdateCounterSample(String name, Map<String, String> tags, double incrValue) {

}


@Override
public void registerOrUpdateGaugeSample(String name, Map<String, String> tags, double value) {

}

@Override
public String response() {

}
}

```

The doInit method is provided for users to extend during initialization and will be called in the init method of AbstractApolloClientMetricsExporter.

```java
@Override
public void init(List<ApolloClientMonitorEventListener> collectors, long collectPeriod) {
log.info("Initializing metrics exporter with {} collectors and collect period of {} seconds.",
collectors.size(), collectPeriod);
doInit();
this.collectors = collectors;
initScheduleMetricsCollectSync(collectPeriod);
log.info("Metrics collection scheduled with a period of {} seconds.", collectPeriod);
}
```

Here, the Prometheus Java client is introduced, and the CollectorRegistry and cache map need to be initialized.

```java
private CollectorRegistry registry;
private Map<String, Collector.Describable> map;

@Override
public void doInit() {
registry = new CollectorRegistry();
map = new HashMap<>();
}
```

The isSupport method will be called in DefaultApolloClientMetricsExporterFactory via SPI to check which MetricsExporter to enable, allowing accurate activation of the configured exporter when multiple SPI implementations exist.

For example, if you want to enable Prometheus and specify the value as "prometheus," it should synchronize here:

```java
@Override
public boolean isSupport(String form) {
return PROMETHEUS.equals(form);
}
```

The methods registerOrUpdateCounterSample and registerOrUpdateGaugeSample are used to register Counter and Gauge type metrics, simply registering based on the provided parameters.

```java
@Override
public void registerOrUpdateCounterSample(String name, Map<String, String> tags,
double incrValue) {
Counter counter = (Counter) map.get(name);
if (counter == null) {
counter = createCounter(name, tags);
map.put(name, counter);
}
counter.labels(tags.values().toArray(new String[0])).inc(incrValue);
}

private Counter createCounter(String name, Map<String, String> tags) {
return Counter.build()
.name(name)
.help("apollo")
.labelNames(tags.keySet().toArray(new String[0]))
.register(registry);
}

@Override
public void registerOrUpdateGaugeSample(String name, Map<String, String> tags, double value) {
Gauge gauge = (Gauge) map.get(name);
if (gauge == null) {
gauge = createGauge(name, tags);
map.put(name, gauge);
}
gauge.labels(tags.values().toArray(new String[0])).set(value);
}

private Gauge createGauge(String name, Map<String, String> tags) {
return Gauge.build()
.name(name)
.help("apollo")
.labelNames(tags.keySet().toArray(new String[0]))
.register(registry);
}
```

Finally, you need to implement the response method, which is used to export data in the format of the integrated monitoring system. It will ultimately be obtained in the getExporterData method of ConfigMonitor, allowing users to expose an endpoint for monitoring systems to pull data.

```java
@Override
public String response() {
try (StringWriter writer = new StringWriter()) {
TextFormat.writeFormat(TextFormat.CONTENT_TYPE_OPENMETRICS_100, writer,
registry.metricFamilySamples());
return writer.toString();
} catch (IOException e) {
logger.error("Write metrics to Prometheus format failed", e);
return "";
}
}
```

At this point, the client's metric data has been integrated with Prometheus.

Full code:https://github.com/apolloconfig/apollo-java/main/master/apollo-plugin/apollo-plugin-client-prometheus/src/main/java/com/ctrip/framework/apollo/monitor/internal/exporter/impl/PrometheusApolloClientMetricsExporter.java
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix the bare URL.

The bare URL at the end of the documentation should be wrapped in angle brackets to make it a valid Markdown link.

Apply this diff to fix the bare URL:

-Full code:https://github.com/apolloconfig/apollo-java/main/master/apollo-plugin/apollo-plugin-client-prometheus/src/main/java/com/ctrip/framework/apollo/monitor/internal/exporter/impl/PrometheusApolloClientMetricsExporter.java
+Full code:<https://github.com/apolloconfig/apollo-java/main/master/apollo-plugin/apollo-plugin-client-prometheus/src/main/java/com/ctrip/framework/apollo/monitor/internal/exporter/impl/PrometheusApolloClientMetricsExporter.java>
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
Full code:https://github.com/apolloconfig/apollo-java/main/master/apollo-plugin/apollo-plugin-client-prometheus/src/main/java/com/ctrip/framework/apollo/monitor/internal/exporter/impl/PrometheusApolloClientMetricsExporter.java
Full code:<https://github.com/apolloconfig/apollo-java/main/master/apollo-plugin/apollo-plugin-client-prometheus/src/main/java/com/ctrip/framework/apollo/monitor/internal/exporter/impl/PrometheusApolloClientMetricsExporter.java>
Tools
Markdownlint

141-141: null
Bare URL used

(MD034, no-bare-urls)

Loading
Loading