Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pkg/ottl] Add contains function #33078

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 27 additions & 0 deletions .chloggen/lkwronski.issue-30420-contains.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: pkg/ottl

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add new Contains function to check item in a slice.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [30420]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user]
27 changes: 27 additions & 0 deletions .chloggen/lkwronski.issue-30420-p-slice-getter.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: pkg/ottl

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Add `PSliceGetter`, a typed getter for `pcommon.Slice`

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [30420]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext:

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [api]
67 changes: 54 additions & 13 deletions pkg/ottl/e2e/e2e_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ package e2e

import (
"context"
"strings"
"testing"
"time"

Expand Down Expand Up @@ -733,80 +734,120 @@ func Test_e2e_converters(t *testing.T) {
func Test_e2e_ottl_features(t *testing.T) {
tests := []struct {
name string
statement string
statement []string
want func(tCtx ottllog.TransformContext)
}{
{
name: "where clause",
statement: `set(attributes["test"], "pass") where body == "operationB"`,
statement: []string{`set(attributes["test"], "pass") where body == "operationB"`},
want: func(_ ottllog.TransformContext) {},
},
{
name: "reach upwards",
statement: `set(attributes["test"], "pass") where resource.attributes["host.name"] == "localhost"`,
statement: []string{`set(attributes["test"], "pass") where resource.attributes["host.name"] == "localhost"`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: "Using enums",
statement: `set(severity_number, SEVERITY_NUMBER_TRACE2) where severity_number == SEVERITY_NUMBER_TRACE`,
statement: []string{`set(severity_number, SEVERITY_NUMBER_TRACE2) where severity_number == SEVERITY_NUMBER_TRACE`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().SetSeverityNumber(2)
},
},
{
name: "Using hex",
statement: `set(attributes["test"], "pass") where trace_id == TraceID(0x0102030405060708090a0b0c0d0e0f10)`,
statement: []string{`set(attributes["test"], "pass") where trace_id == TraceID(0x0102030405060708090a0b0c0d0e0f10)`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: "where clause without comparator",
statement: `set(attributes["test"], "pass") where IsMatch(body, "operation[AC]")`,
statement: []string{`set(attributes["test"], "pass") where IsMatch(body, "operation[AC]")`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: "where clause with Converter return value",
statement: `set(attributes["test"], "pass") where body == Concat(["operation", "A"], "")`,
statement: []string{`set(attributes["test"], "pass") where body == Concat(["operation", "A"], "")`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: "composing functions",
statement: `merge_maps(attributes, ParseJSON("{\"json_test\":\"pass\"}"), "insert") where body == "operationA"`,
statement: []string{`merge_maps(attributes, ParseJSON("{\"json_test\":\"pass\"}"), "insert") where body == "operationA"`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("json_test", "pass")
},
},
{
name: "where clause with Contains return value",
statement: []string{`set(attributes["test"], "pass") where Contains(["hello", "world"], "hello")`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: "where clause with Contains ints return value",
statement: []string{`set(attributes["test"], "pass") where Contains([1, 2, 3, 4], 4)`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: "where clause with Contains floats return value",
statement: []string{`set(attributes["test"], "pass") where Contains([1.1, 2.2, 3.3, 4.4], 4.4)`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: `set attribute when tag "staging" is in tags attributes slice using Contains`,
statement: []string{
`set(attributes["tags"], ["staging", "hello", "world", "work"])`,
`set(attributes["staging"], "true") where Contains(attributes["tags"], "staging")`,
},
want: func(tCtx ottllog.TransformContext) {
var tags = tCtx.GetLogRecord().Attributes().PutEmptySlice("tags")
tags.AppendEmpty().SetStr("staging")
tags.AppendEmpty().SetStr("hello")
tags.AppendEmpty().SetStr("world")
tags.AppendEmpty().SetStr("work")

tCtx.GetLogRecord().Attributes().PutStr("staging", "true")
},
},
{
name: "complex indexing found",
statement: `set(attributes["test"], attributes["foo"]["bar"])`,
statement: []string{`set(attributes["test"], attributes["foo"]["bar"])`},
want: func(tCtx ottllog.TransformContext) {
tCtx.GetLogRecord().Attributes().PutStr("test", "pass")
},
},
{
name: "complex indexing not found",
statement: `set(attributes["test"], attributes["metadata"]["uid"])`,
statement: []string{`set(attributes["test"], attributes["metadata"]["uid"])`},
want: func(_ ottllog.TransformContext) {},
},
}

for _, tt := range tests {
t.Run(tt.statement, func(t *testing.T) {
t.Run(strings.Join(tt.statement, "\n"), func(t *testing.T) {
settings := componenttest.NewNopTelemetrySettings()
logParser, err := ottllog.NewParser(ottlfuncs.StandardFuncs[ottllog.TransformContext](), settings)
assert.NoError(t, err)
logStatements, err := logParser.ParseStatement(tt.statement)
logStatements, err := logParser.ParseStatements(tt.statement)
assert.NoError(t, err)

tCtx := constructLogTransformContext()
_, _, _ = logStatements.Execute(context.Background(), tCtx)

for i := range logStatements {
_, _, _ = logStatements[i].Execute(context.Background(), tCtx)
}

exTCtx := constructLogTransformContext()
tt.want(exTCtx)
Expand Down
41 changes: 41 additions & 0 deletions pkg/ottl/expression.go
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,47 @@ func (l *listGetter[K]) Get(ctx context.Context, tCtx K) (any, error) {
return evaluated, nil
}

// PSliceGetter is a Getter that must return a pcommon.Slice.
type PSliceGetter[K any] interface {
evan-bradley marked this conversation as resolved.
Show resolved Hide resolved
Get(ctx context.Context, tCtx K) (pcommon.Slice, error)
}

// PStandardSliceGetter is a basic implementation of PSliceGetter
type StandardPSliceGetter[K any] struct {
Getter func(ctx context.Context, tCtx K) (any, error)
}

// Get retrieves a pcommon.Slice value.
// If the value is not a pcommon.Slice a new TypeError is returned.
// If there is an error getting the value it will be returned.
func (g StandardPSliceGetter[K]) Get(ctx context.Context, tCtx K) (pcommon.Slice, error) {
val, err := g.Getter(ctx, tCtx)
if err != nil {
return pcommon.Slice{}, fmt.Errorf("error getting value in %T: %w", g, err)
}
if val == nil {
return pcommon.Slice{}, TypeError("expected pcommon.Slice but got nil")
}
switch v := val.(type) {
case pcommon.Slice:
return v, nil
case pcommon.Value:
if v.Type() == pcommon.ValueTypeSlice {
return v.Slice(), nil
}
return pcommon.Slice{}, TypeError(fmt.Sprintf("expected pcommon.Slice but got %v", v.Type()))
evan-bradley marked this conversation as resolved.
Show resolved Hide resolved
case []any:
s := pcommon.NewSlice()
err := s.FromRaw(v)
if err != nil {
return pcommon.Slice{}, err
}
return s, nil
default:
return pcommon.Slice{}, TypeError(fmt.Sprintf("expected pcommon.Slice but got %T", val))
}
}

// TypeError represents that a value was not an expected type.
type TypeError string

Expand Down
6 changes: 6 additions & 0 deletions pkg/ottl/functions.go
Original file line number Diff line number Diff line change
Expand Up @@ -469,6 +469,12 @@ func (p *Parser[K]) buildArg(argVal value, argType reflect.Type) (any, error) {
return nil, err
}
return StandardPMapGetter[K]{Getter: arg.Get}, nil
case strings.HasPrefix(name, "PSliceGetter"):
arg, err := p.newGetter(argVal)
if err != nil {
return nil, err
}
return StandardPSliceGetter[K]{Getter: arg.Get}, nil
Comment on lines +472 to +477
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding the new Getter requires some new tests in functions_test.go that hit this case

case strings.HasPrefix(name, "DurationGetter"):
arg, err := p.newGetter(argVal)
if err != nil {
Expand Down
19 changes: 19 additions & 0 deletions pkg/ottl/ottlfuncs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -411,6 +411,7 @@ Available Converters:

- [Base64Decode](#base64decode)
- [Concat](#concat)
- [Contains](#contains)
- [ConvertCase](#convertcase)
- [Day](#day)
- [ExtractPatterns](#extractpatterns)
Expand Down Expand Up @@ -493,6 +494,24 @@ Examples:

- `Concat(["HTTP method is: ", attributes["http.method"]], "")`


### Contains

`Contains(target, item)`

The Contains function checks if an item is present in a given slice `target`. It returns true if the `item` is found, and false otherwise.

`target` is a slice of primitive-typed values.

`item` is the primitive-typed value to check for in the `target`.

Examples:

- `Contains(attributes["tags"], "staging")`
- `Contains([1, 2, 3, 4, 5], 3)`
- `Contains([1.1, 2.2, 3.3, 4.4], 4.4)`
- `Contains(["GET", "PUT", "POST"], "GET")`

### ConvertCase

`ConvertCase(target, toCase)`
Expand Down
51 changes: 51 additions & 0 deletions pkg/ottl/ottlfuncs/func_contains.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
// Copyright The OpenTelemetry Authors
// SPDX-License-Identifier: Apache-2.0

package ottlfuncs // import "github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl/ottlfuncs"

import (
"context"
"fmt"

"github.com/open-telemetry/opentelemetry-collector-contrib/pkg/ottl"
)

type ContainsArguments[K any] struct {
Target ottl.PSliceGetter[K]
Item ottl.Getter[K]
}

func NewContainsFactory[K any]() ottl.Factory[K] {
return ottl.NewFactory("Contains", &ContainsArguments[K]{}, createContainsFunction[K])
}

func createContainsFunction[K any](_ ottl.FunctionContext, oArgs ottl.Arguments) (ottl.ExprFunc[K], error) {
args, ok := oArgs.(*ContainsArguments[K])

if !ok {
return nil, fmt.Errorf("ContainsFactory args must be of type *ContainsArguments[K]")
}

return contains(args.Target, args.Item), nil
}

func contains[K any](target ottl.PSliceGetter[K], itemGetter ottl.Getter[K]) ottl.ExprFunc[K] {
return func(ctx context.Context, tCtx K) (any, error) {
slice, sliceErr := target.Get(ctx, tCtx)
if sliceErr != nil {
return nil, sliceErr
}
item, itemErr := itemGetter.Get(ctx, tCtx)
if itemErr != nil {
return nil, itemErr
}

for i := 0; i < slice.Len(); i++ {
val := slice.At(i).AsRaw()
if val == item {
return true, nil
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although functionally correct and generic enough to cover multiple data types, this approach can become prohibitively expensive for performance sensitive applications under load. For example, if this approach was used for supporting routing-connector in open-telemetry collector to inspect 100+ string values while processing 100,000 spans per second kind of workload. A map would be much more efficient for such kind of conditions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just to clarify, when i say multiple data types.... I meant to appreciate that it can handle integer, strings, booleans alike.... However I am concerned that outer type has to be a slice only... which may not be the optimal choice for heavy workloads of say, simple string based look up.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand your concern, but I'm not sure how to do it with a map. If we want to handle Contains that accept a slice as input, I don't see how to convert it to a map in linear time.

If we accepted map as input in Contains, it would be easy to get an efficient search, but I'm not sure how to do it with slice.

return false, nil
}
}
Loading