Add dimensions to cloudwatch metric and metric filter option for CWSink.

[vmmusings]

Description

• Added option to include metric filter in CWSink.
• CloudWatchSDK is not available during runtime, so adding the dependency explicitly.
• Example Usage :--conf spark.metrics.conf.*.sink.cloudwatch.regex=.*Flint.* property would publish only metrics with the regex Given.
• Streamlined implementation of adding dimensions to metricNames.
• No need to review DimensionedName Class: It is picked up from https://github.com/azagniotov/codahale-aggregated-metrics-cloudwatch-reporter.

Usage of Dimensioned Name for below image:

final DimensionedName dimensionedName = DimensionedName
        .withName("FlintOpensearchErrorCount")
        .withDimension("domainIdent", "88888")
        .build();

metricRegistry.counter(dimensionedName.encode()).inc();
metricRegistry.counter(derivedDimensionedName.encode()).inc();

Tested on emr-serverless by publishing dummy metrics.

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[penghuo]

what is the relationship of metrics dimension and screenshot?

final DimensionedName dimensionedName = DimensionedName
        .withName("test")
        .withDimension("key1", "val1")
        .withDimension("key2", "val2")
        .withDimension("key3", "val3")
        .build();

[vmmusings]

what is the relationship of metrics dimension and screenshot?

final DimensionedName dimensionedName = DimensionedName
        .withName("test")
        .withDimension("key1", "val1")
        .withDimension("key2", "val2")
        .withDimension("key3", "val3")
        .build();

Let me change the PR description.
Spark Dropwizard metrics doesn't support dimensions, we encode dimensions in metric name and add them when pushing to CW.
For the screenshot.

DimensionName dimensionedName = DimensionedName
.withName("FlintOpensearchErrorCount")
.withDimension("domainIdent", "88888")
.build()

[dai-chen]

Any chance this can be provided or not?

meanwhile I'm just thinking any impact on open source user who use Flint as library. Does all classes added have to be in Flint core module?

[vmmusings]

• Provided is not working...cloudwatch is not available.
• I still didn't get the idea behind flint-core,[I only see OS Client] should I one more package as flint-common-utils?

[vmmusings]

There won't be any impact to opensource user. We emit metrics but nothing will be pushed unless one configures a sink.

[dai-chen]

• Provided is not working...cloudwatch is not available.
• I still didn't get the idea behind flint-core,[I only see OS Client] should I one more package as flint-common-utils?

For now we only have FlintOpenSearchClient. We may add FlintFileSystemClient later (for storing Flint index data on S3). I was thinking for user who use Flint library in their own application.

[vmmusings]

Even for them we are actually providing benefit by giving out metrics similar to driver and executor metrics. CWSink they can leverage if they wish, otherwise they can ignore. We can discuss offline on this and will post the summary over here.