Skip to content

Commit

Permalink
32669 wireframe sum connector (open-telemetry#33759)
Browse files Browse the repository at this point in the history
**Description:** Initial wireframe PR for a new sum connector (as
described in open-telemetry#32669)
Provides bare minimum to start developing a new connector (config,
factory, integration test stubs for connector lifecycle, etc)

Sum Connector takes in logs, metrics, or traces and matches an attribute
then allows summing numerical values present in that attribute and
sending those sums as a time series metric along with any attributes
defined within the connector config.

E.G. Log contains a field for `total_price` of a payment. Matching on
`total_price` as the `source_attribute` will take numerical values from
this attributte and emit a time series metric (`checkout.total`) of the
sums along with any other attributes defined in the connector config.


Example config:
```
receivers:
  foo:
connectors:
  sum:
    logs:
      checkout.total:
        source_attribute: 
          - attributes["total_price"]
        conditions:
          - attributes["total_price"] != "NULL"
        attributes:
          - key: payment.processor
            default_value: unspecified_processor
          - key: env
            default_value: no_env
exporters:
  bar:

service:
  pipelines:
    metrics/sum:
       receivers: [sum]
       exporters: [bar]
    logs:
       receivers: [foo]
       exporters: [sum]
```      

**Link to tracking Issue:** 32669

**Testing:** Generated component and package tests

**Documentation:** README and issue open-telemetry#32669 provide current documentation
of projected future state.
  • Loading branch information
greatestusername authored Jul 3, 2024
1 parent 9d7fec7 commit 94d47eb
Show file tree
Hide file tree
Showing 18 changed files with 755 additions and 0 deletions.
27 changes: 27 additions & 0 deletions .chloggen/32669-wireframe-sum.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: new_component

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: sumconnector

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: "creates a wireframe and initial pr to develop from"

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [32669]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: []
1 change: 1 addition & 0 deletions .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ connector/roundrobinconnector/ @open-teleme
connector/routingconnector/ @open-telemetry/collector-contrib-approvers @jpkrohling @mwear
connector/servicegraphconnector/ @open-telemetry/collector-contrib-approvers @jpkrohling @mapno
connector/spanmetricsconnector/ @open-telemetry/collector-contrib-approvers @portertech @Frapschen
connector/sumconnector/ @open-telemetry/collector-contrib-approvers @greatestusername @shalper2 @crobert-1

examples/demo/ @open-telemetry/collector-contrib-approvers @open-telemetry/collector-approvers

Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/bug_report.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ body:
- connector/routing
- connector/servicegraph
- connector/spanmetrics
- connector/sum
- examples/demo
- exporter/alertmanager
- exporter/alibabacloudlogservice
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/feature_request.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ body:
- connector/routing
- connector/servicegraph
- connector/spanmetrics
- connector/sum
- examples/demo
- exporter/alertmanager
- exporter/alibabacloudlogservice
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/other.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ body:
- connector/routing
- connector/servicegraph
- connector/spanmetrics
- connector/sum
- examples/demo
- exporter/alertmanager
- exporter/alibabacloudlogservice
Expand Down
1 change: 1 addition & 0 deletions .github/ISSUE_TEMPLATE/unmaintained.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ body:
- connector/routing
- connector/servicegraph
- connector/spanmetrics
- connector/sum
- examples/demo
- exporter/alertmanager
- exporter/alibabacloudlogservice
Expand Down
1 change: 1 addition & 0 deletions connector/sumconnector/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
include ../../Makefile.Common
105 changes: 105 additions & 0 deletions connector/sumconnector/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# Sum Connector
<!-- status autogenerated section -->
| Status | |
| ------------- |-----------|
| Distributions | [] |
| Issues | [![Open issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aopen%20label%3Aconnector%2Fsum%20&label=open&color=orange&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aopen+is%3Aissue+label%3Aconnector%2Fsum) [![Closed issues](https://img.shields.io/github/issues-search/open-telemetry/opentelemetry-collector-contrib?query=is%3Aissue%20is%3Aclosed%20label%3Aconnector%2Fsum%20&label=closed&color=blue&logo=opentelemetry)](https://github.com/open-telemetry/opentelemetry-collector-contrib/issues?q=is%3Aclosed+is%3Aissue+label%3Aconnector%2Fsum) |
| [Code Owners](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/CONTRIBUTING.md#becoming-a-code-owner) | [@greatestusername](https://www.github.com/greatestusername), [@shalper2](https://www.github.com/shalper2), [@crobert-1](https://www.github.com/crobert-1) |

[development]: https://github.com/open-telemetry/opentelemetry-collector#development

## Supported Pipeline Types

| [Exporter Pipeline Type] | [Receiver Pipeline Type] | [Stability Level] |
| ------------------------ | ------------------------ | ----------------- |
| traces | metrics | [development] |
| metrics | metrics | [development] |
| logs | metrics | [development] |

[Exporter Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#exporter-pipeline-type
[Receiver Pipeline Type]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md#receiver-pipeline-type
[Stability Level]: https://github.com/open-telemetry/opentelemetry-collector#stability-levels
<!-- end autogenerated section -->

The `sum` connector can be used to sum attribute values from spans, span events, metrics, data points, and log records.

## Configuration

If you are not already familiar with connectors, you may find it helpful to first visit the [Connectors README](https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md).

### Configuration

#### Basic configuration

This configuration will sum numerical values found within the attribute `attribute.with.numerical.value` of any log telemetry routed to the connector. It will then output a metric time series with the name `my.example.metric.name` with those summed values.

```yaml
receivers:
foo:
connectors:
sum:
logs:
my.example.metric.name:
source_attribute: attribute.with.numerical.value
exporters:
bar:

service:
pipelines:
metrics/sum:
receivers: [sum]
exporters: [bar]
logs:
receivers: [foo]
exporters: [sum]
```
#### Required Settings
The sum connector has three required configuration settings and numerous optional settings
- Telemetry type: Nested below the `sum:` connector declaration. Declared as `logs:` in the [Basic Example](#basic-configuration).
- Can be any of `spans`, `spanevents`, `metrics`, `datapoints`, or `logs`.
- Metric name: Nested below the telemetry type; this is the metric name the sum connector will output summed values to. Declared as `my.example.metric.name` in the [Basic Example](#basic-configuration)
- `source_attribute`: A specific attribute to search for within the source telemetry being fed to the connector. This attribute is where the connector will look for numerical values to sum into the output metric value. Declared as `attribute.with.numerical.value` in the [Basic Example](#basic-configuration)

#### Optional Settings

- `conditions`: [OTTL syntax](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/pkg/ottl/LANGUAGE.md) can be used to provide conditions for processing incoming telemetry. Conditions are ORed together, so if any condition is met the attribute's value will be included in the resulting sum.
- `attributes`: Declaration of attributes to include. Any of these attributes found will generate a separate sum for each set of unique combination of attribute values and output as its own datapoint in the metric time series.
- `key`: (required for `attributes`) the attribute name to match against
- `default_value`: (optional for `attributes`) a default value for the attribute when no matches are found. The `default_value` value can be of type string, integer, or float.

#### Detailed Example Configuration

This example declares that the `sum` connector is going to be ingesting `logs` and creating an output metric named `checkout.total` with numerical values found in the `source_attribute` `total.payment`.

It provides a condition to check that the attribute `total.payment` is not `NULL`. It also checks any incoming log telemetry for values present in the attribute `payment.processor` and creates a datapoint within the metric time series for each unique value. Any logs without values in `payment.processor` will be included in a datapoint with the `default_value` of `unspecified_processor`.

```yaml
receivers:
foo:
connectors:
sum:
logs:
checkout.total:
source_attribute: total.payment
conditions:
- attributes["total.payment"] != "NULL"
attributes:
- key: payment.processor
default_value: unspecified_processor
exporters:
bar:
service:
pipelines:
metrics/sum:
receivers: [sum]
exporters: [bar]
logs:
receivers: [foo]
exporters: [sum]
```

[Connectors README]: https://github.com/open-telemetry/opentelemetry-collector/blob/main/connector/README.md
26 changes: 26 additions & 0 deletions connector/sumconnector/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package sumconnector // import "github.com/open-telemetry/opentelemetry-collector-contrib/connector/sumconnector"

// Config for the connector
type Config struct {
Spans map[string]MetricInfo `mapstructure:"spans"`
SpanEvents map[string]MetricInfo `mapstructure:"spanevents"`
Metrics map[string]MetricInfo `mapstructure:"metrics"`
DataPoints map[string]MetricInfo `mapstructure:"datapoints"`
Logs map[string]MetricInfo `mapstructure:"logs"`
}

// MetricInfo for a data type
type MetricInfo struct {
Description string `mapstructure:"description"`
Conditions []string `mapstructure:"conditions"`
Attributes []AttributeConfig `mapstructure:"attributes"`
SourceAttribute string `mapstructure:"source_attribute"`
}

type AttributeConfig struct {
Key string `mapstructure:"key"`
DefaultValue any `mapstructure:"default_value"`
}
59 changes: 59 additions & 0 deletions connector/sumconnector/connector.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package sumconnector // import "github.com/open-telemetry/opentelemetry-collector-contrib/connector/sumconnector"

import (
"context"

"go.opentelemetry.io/collector/component"
"go.opentelemetry.io/collector/consumer"
"go.opentelemetry.io/collector/pdata/plog"
"go.opentelemetry.io/collector/pdata/pmetric"
"go.opentelemetry.io/collector/pdata/ptrace"

"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottldatapoint"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottllog"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlmetric"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspan"
"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/contexts/ottlspanevent"
)

// sum can sum attribute values from spans, span event, metrics, data points, or log records
// and emit the sums onto a metrics pipeline.
type sum struct {
metricsConsumer consumer.Metrics
component.StartFunc
component.ShutdownFunc

spansMetricDefs map[string]metricDef[ottlspan.TransformContext]
spanEventsMetricDefs map[string]metricDef[ottlspanevent.TransformContext]
metricsMetricDefs map[string]metricDef[ottlmetric.TransformContext]
dataPointsMetricDefs map[string]metricDef[ottldatapoint.TransformContext]
logsMetricDefs map[string]metricDef[ottllog.TransformContext]
}

func (c *sum) Capabilities() consumer.Capabilities {
return consumer.Capabilities{MutatesData: false}
}

func (c *sum) ConsumeTraces(ctx context.Context, td ptrace.Traces) error {
sumMetrics := pmetric.NewMetrics()
sumMetrics.ResourceMetrics().EnsureCapacity(td.ResourceSpans().Len())

return c.metricsConsumer.ConsumeMetrics(ctx, sumMetrics)
}

func (c *sum) ConsumeMetrics(ctx context.Context, md pmetric.Metrics) error {
sumMetrics := pmetric.NewMetrics()
sumMetrics.ResourceMetrics().EnsureCapacity(md.ResourceMetrics().Len())

return c.metricsConsumer.ConsumeMetrics(ctx, sumMetrics)
}

func (c *sum) ConsumeLogs(ctx context.Context, ld plog.Logs) error {
sumMetrics := pmetric.NewMetrics()
sumMetrics.ResourceMetrics().EnsureCapacity(ld.ResourceLogs().Len())

return c.metricsConsumer.ConsumeMetrics(ctx, sumMetrics)
}
Loading

0 comments on commit 94d47eb

Please sign in to comment.