[processor/logdedup] feat: add ottl condition to logdedup processor #35443
Conversation
github-actions
bot
requested review from
BinaryFissionGames and
MikeGoldsmith
There was a problem hiding this comment.
Overall looks good, thanks @colelaven. I've left a couple of suggestions. The main one being breaking out condition into a set of conditions.
colelaven
force-pushed
the
feat/add-condition-to-logdedup-processor
branch
from
36b0a50
to
a2135f1
Compare
| consumedLogs := allSinkLogs[0] | ||
| require.Equal(t, 2, consumedLogs.LogRecordCount()) | ||
|
|
||
| require.Equal(t, 1, consumedLogs.ResourceLogs().Len()) | ||
| consumedRl := consumedLogs.ResourceLogs().At(0) | ||
| require.Equal(t, 1, consumedRl.ScopeLogs().Len()) | ||
| consumedSl := consumedRl.ScopeLogs().At(0) | ||
| require.Equal(t, 2, consumedSl.LogRecords().Len()) | ||
|
|
||
| for i := 0; i < consumedSl.LogRecords().Len(); i++ { | ||
| consumedLogRecord := consumedSl.LogRecords().At(i) | ||
| ID, ok := consumedLogRecord.Attributes().Get("ID") | ||
| require.True(t, ok) | ||
| require.Equal(t, int64(2), ID.Int()) | ||
| } | ||
|
|
||
| dedupedLogs := allSinkLogs[1] | ||
| require.Equal(t, 1, dedupedLogs.LogRecordCount()) | ||
|
|
||
| require.Equal(t, 1, dedupedLogs.ResourceLogs().Len()) | ||
| dedupedRl := dedupedLogs.ResourceLogs().At(0) | ||
| require.Equal(t, 1, dedupedRl.ScopeLogs().Len()) | ||
| dedupedSl := dedupedRl.ScopeLogs().At(0) | ||
| require.Equal(t, 1, dedupedSl.LogRecords().Len()) | ||
| dedupedLogRecord := dedupedSl.LogRecords().At(0) | ||
|
|
||
| countVal, ok := dedupedLogRecord.Attributes().Get(cfg.LogCountAttribute) | ||
| require.True(t, ok) | ||
| require.Equal(t, int64(2), countVal.Int()) |
There was a problem hiding this comment.
When comparing actual vs expected telemetry, the code can be very complex and still the comparison can be very incomplete. We added some packages to facilitate more precise validation of expected results.
Specifically, use the golden package capture and reload expected results. Use golden.WriteLogs to write these objects once. Then delete the WriteLogs call and use ReadLogs in the actual test.
Then use pdatatest/plogtest.CompareLogs (and make sure to assert no error). You may need to pass in some options to ignore volatile parts of the logs.
There was a problem hiding this comment.
Good suggestion, just pushed a commit to use golden for the tests and it looks much cleaner now.
| require.NoError(t, err) | ||
| } | ||
|
|
||
| func removeTimestamps(plogs ...plog.Logs) { |
There was a problem hiding this comment.
There should be an option in plogtest to ignore these attributes values. This still allows validation that the attributes are present.
There was a problem hiding this comment.
There wasn't an existing option to ignore log record attribute values, only one for ignoring resource attributes. I added one for ignoring log record attribute values and it is working great.
Let me know if adding that option is not something I should be doing in this PR.
|
Thanks @colelaven |
…pen-telemetry#35443) **Description:** Adds OTTL Condition field to Deduplicate Logs Processor **Link to tracking Issue:** Closes open-telemetry#35440 **Testing:** - Tested functionality with BindPlane - Added unit tests for the condition logic **Documentation:** Added documentation to the logdedup processor README about the condition field and an example configuration with a condition. --------- Co-authored-by: Mike Goldsmith <[email protected]>
Description:
Adds OTTL Condition field to Deduplicate Logs Processor
Link to tracking Issue: Closes #35440
Testing:
Documentation: Added documentation to the logdedup processor README about the condition field and an example configuration with a condition.