stats/opentelemetry: Remove OpenTelemetry module and add RLS Metrics e2e tests

[zasweq]

Redid #7655 as dependencies were updated on master.

This PR adds e2e tests for RLS Metrics, with verifications of the OpenTelemetry metrics atoms emitted.

RELEASE NOTES:

• stats/opentelemetry: Stabilize OpenTelemetry and CSM Observability packages

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.73%. Comparing base (ada6787) to head (965d55f).
Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7759      +/-   ##
==========================================
+ Coverage   81.43%   81.73%   +0.30%     
==========================================
  Files         368      374       +6     
  Lines       36799    38136    +1337     
==========================================
+ Hits        29967    31171    +1204     
- Misses       5606     5691      +85     
- Partials     1226     1274      +48     

see 31 files with indirect coverage changes

[ash2k]

It would be great to avoid adding any cloud-specific dependencies. Why not let users inject detectors?

Hm, actually, looking at constructMetadataFromEnv() - it's not the job of gRPC (or any other) library to get those labels, right? resource.New() should be called by the application to centrally construct the resource (which represents the application). Other metrics, traces, etc all need the resource too:

// Resource describes an entity about which identifying information
// and metadata is exposed.  Resource is an immutable object,
// equivalent to a map from key to unique value.
//
// Resources should be passed and stored as pointers
// (`*resource.Resource`).  The `nil` value is equivalent to an empty
// Resource.
type Resource struct {

[ash2k]

If I use gcp detector in my app, I'll have duplicate attributes. OTEL will probably filter them out, but that's a waste of resources.

[zasweq]

Adding this dependency doesn't affect your detectors right? This dependency is part of the CSM Observability feature, and doesn't get invoked unless that feature is explicitly configured.

From my understanding, the detectors are entirely orthogonal unless you chain results together, and are based off environment variables and properties. Thus, this doesn't set any properties of the environment and only optionally reads it if CSM Observability is turned on, and thus shouldn't affect your own detectors.

[ash2k]

Ok, I see CSM is Google Cloud Service Mesh. It makes sense to hard-code the GCP detector then.

Still, would be better for your consumers to have this dependency as an opt-in (most people don't need it). Perhaps the csm package can be made a separate module? grpc-go is used in so many projects. I think it makes sense to minimize the number of dependencies that are brought in.

[zasweq]

The decision was made to release these CSM Observability/OpenTelemetry API's as stable. Rather than have OpenTelemetry as it's own module, we decided to release these features as simply part of our top level go module, as C-Core does. So I think this will be left here. Note that this is simply listing the dependencies. It does not affect binary size or performance unless the feature that uses the dependency is enabled (in this case, CSM Observability).

[ash2k]

Ok, thank you!

[dfawley]

Note that we also put in some safeguards to help protect this from accidentally getting imported into any of the main grpc packages in #7766. There are some nasty ways they could sneak in by moving common functionality to an internal package and then adding an internal dependency on that from somewhere else. We have similar concerns about other things affecting binary size and dependency trees like xds.

[ash2k]

That's great, thank you!