Add OpenSearch metrics

[noCharger]

Description

This PR is to

• add a rest high level client wrapper for operations around the original rest high level client using the Decorator Pattern, considering RestHighLevelClient itself is not extensible.
• add spark metrics source for flint
• integrate the client wrapper to be the aspect for collecting opensearch metrics

Metrics included in this PR:

OpenSearch Status Count, Dimension by default [applicationId, jobId, instanceRole]

opensearch.read.2xx.count
opensearch.read.4xx.count
opensearch.read.5xx.count
opensearch.read.403.count

opensearch.write.2xx.count
opensearch.write.4xx.count
opensearch.write.5xx.count
opensearch.write.403.count

Approaches

There are three approaches to achieving aspect-oriented programming (AOP) to provide metrics within Flint: AspectJ library, dynamic proxies, and wrapper (decorator).

AspectJ

• AOP Implementation: AspectJ is a robust AOP solution that enables the seamless and enhanced integration of cross-cutting concerns without altering the existing code base. It employs bytecode weaving (either at compile-time or load-time) to provide a wide range of AOP capabilities such as method interception, behavior modification, and even class structure change. PoC: [POC]Flint metrics AOP #219.
• Performance: Because AspectJ alters the bytecode directly, method calls perform virtually natively. This makes it ideal for high-performance and high-throughput applications.
• Use Cases: Ideally suited for complicated applications with several cross-cutting concerns. AspectJ is especially handy when requiring advanced AOP features that extend beyond simple method interception.
• Challenge: The official sbt-aspectj plugin, https://github.com/sbt/sbt-aspectj, is less maintained. More effort is needed to adapt the AspectJ configuration for the Flint repository.

Java Dynamic Proxies

AOP Implementation: Dynamic proxies make it easier to implement AOP by intercepting methods via reflection. They are limited to interfaces and cannot directly change class behavior or attributes.

Code example:

import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.util.logging.Logger;

public class RestClientInvocationHandler implements InvocationHandler {
    private static final Logger LOG = Logger.getLogger(RestClientInvocationHandler.class.getName());
    private final Object target;

    public RestClientInvocationHandler(Object target) {
        this.target = target;
    }

    @Override
    public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
        long startTime = System.currentTimeMillis();
        try {
            Object result = method.invoke(target, args);
            logMetric("success", System.currentTimeMillis() - startTime);
            return result;
        } catch (Throwable t) {
            logMetric("failure", System.currentTimeMillis() - startTime);
            throw t;
        }
    }

    private void logMetric(String status, long duration) {
        LOG.info("Method " + status + ": " + duration + "ms");
    }
}

import org.opensearch.client.RestHighLevelClient;
import java.lang.reflect.Proxy;

public class RestClientProxyFactory {

    public static RestHighLevelClient createProxy(RestHighLevelClient client) {
        return (RestHighLevelClient) Proxy.newProxyInstance(
            client.getClass().getClassLoader(),
            new Class<?>[]{RestHighLevelClient.class},
            new RestClientInvocationHandler(client)
        );
    }
}


RestHighLevelClient realClient = new RestHighLevelClient(...);
RestHighLevelClient client = RestClientProxyFactory.createProxy(realClient);

Performance: Because of the usage of reflection, they have a larger overhead than AspectJ approach. However, for many common applications, this expense is insignificant.
Use cases: Suitable for applications in which the AOP requirements are confined to method interception and interfaces are the norm. It's a suitable alternative for basic AOP scenarios and projects where introducing a dependency like AspectJ isn't ideal.

Wrapper (Decorator) Approach implemented in this PR

AOP Implementation: This approach doesn't use AOP in the traditional sense but can mimic some aspects of it, like method interception, by manually wrapping classes. It's more of a design pattern than an AOP solution and requires explicit coding for each method and class.
Performance: The wrapper approach introduces additional method calls, but these are usually direct and thus have less overhead than dynamic proxies. The performance impact is generally minimal, but it's more than AspectJ.
Use Cases: Best for situations where you need to wrap a limited number of classes or methods, and the interface of the classes is stable. It's straightforward and easy to understand, making it a good choice for smaller projects or when simplicity is key.

Overall Comparison
Performance Hierarchy: AspectJ > Wrapper Approach > Dynamic Proxies. AspectJ is typically the fastest due to direct bytecode manipulation, followed by the wrapper approach with its direct method calls, and finally dynamic proxies with the overhead of reflection.
Complexity and Flexibility: AspectJ is the most complex but also the most flexible and powerful. Dynamic proxies offer a balance between simplicity and flexibility, suitable for medium complexity needs. The wrapper approach is the simplest but the least flexible, especially for large-scale or changing codebases.
Suitability: AspectJ is ideal for large, complex applications with extensive cross-cutting concerns. Dynamic proxies are good for medium-scale applications or where interface-based programming is predominant. The wrapper approach is best for smaller applications or when you need a straightforward solution without extra dependencies or complexities.

Issues Resolved

#220

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[vmmusings]

Thanks for the changes.

[opensearch-trigger-bot]

The backport to 0.1 failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/opensearch-spark/backport-0.1 0.1
# Navigate to the new working tree
pushd ../.worktrees/opensearch-spark/backport-0.1
# Create a new branch
git switch --create backport/backport-229-to-0.1
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 6f708847c5b91d5d702b5d37d7afc8edc6c13e15
# Push it to GitHub
git push --set-upstream origin backport/backport-229-to-0.1
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/opensearch-spark/backport-0.1

Then, create a pull request where the base branch is 0.1 and the compare/head branch is backport/backport-229-to-0.1.