Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(observability): OpenTelemetry logs #13291

Merged
merged 5 commits into from
Jul 5, 2024
Merged

Conversation

samugi
Copy link
Member

@samugi samugi commented Jun 24, 2024

Note for reviewers: commits are split and commented appropriately, I would recommend checking them one by one.

Summary

This PR adds OpenTelemetry formatted logs as a new
signal supported by the OpenTelemetry plugin.

It patches the ngx.log function (and adapts kong PDK log) to collect the logged lines and
enrich them with additional context info:

  • severity
  • introspection details
  • timestamp

And for request-scoped logs also:

  • Request ID
  • Trace ID
  • Span ID
  • Trace flags

The log entries are protobuf-serialized and sent to the endpoint
configured in the OpenTelemetry plugin's config.logs_endpoint field.

Checklist

Issue reference

KAG-4712

@samugi samugi changed the title Feat/otel formatted logs feat(observability): OpenTelemetry-formatted logs Jun 24, 2024
@samugi samugi changed the title feat(observability): OpenTelemetry-formatted logs feat(observability): OpenTelemetry logs Jun 24, 2024
@samugi samugi requested review from hanshuebner and ADD-SP June 24, 2024 07:36
@samugi samugi force-pushed the feat/otel-formatted-logs branch 2 times, most recently from 724a84a to a775404 Compare June 24, 2024 09:05
@samugi samugi added this to the 3.8.0 milestone Jun 24, 2024
@samugi samugi force-pushed the feat/otel-formatted-logs branch 3 times, most recently from 5e5257d to 3621b8f Compare June 24, 2024 17:22
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Show resolved Hide resolved
kong/plugins/opentelemetry/handler.lua Outdated Show resolved Hide resolved
Copy link
Contributor

@ADD-SP ADD-SP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And also I suggest not rebase commits when applying reviewing suggestions, we can use the following command for better view for the reviewer.

git commit --fixup <commit_id>

@samugi
Copy link
Member Author

samugi commented Jun 25, 2024

And also I suggest not rebase commits when applying reviewing suggestions, we can use the following command for better view for the reviewer.

git commit --fixup <commit_id>

yes absolutely. I wasn't planning to rebase (for now), which will make syncing (via cherry-pick) with the existing EE PR easier as well.

@samugi samugi force-pushed the feat/otel-formatted-logs branch from b0ab33b to b5c2ece Compare July 3, 2024 09:56
@samugi samugi requested a review from ADD-SP July 3, 2024 09:57
@samugi samugi force-pushed the feat/otel-formatted-logs branch from b5c2ece to ad0b93c Compare July 3, 2024 10:40
@samugi samugi mentioned this pull request Jul 3, 2024
3 tasks
kong/globalpatches.lua Show resolved Hide resolved
kong/plugins/opentelemetry/logs.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/otlp.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/otlp.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/traces.lua Show resolved Hide resolved
kong/plugins/opentelemetry/logs.lua Show resolved Hide resolved
kong/plugins/opentelemetry/logs.lua Show resolved Hide resolved
kong/plugins/opentelemetry/traces.lua Show resolved Hide resolved
kong/plugins/opentelemetry/schema.lua Show resolved Hide resolved
kong/plugins/opentelemetry/utils.lua Outdated Show resolved Hide resolved
kong/plugins/opentelemetry/utils.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Outdated Show resolved Hide resolved
@samugi samugi force-pushed the feat/otel-formatted-logs branch from fc44de7 to be23116 Compare July 4, 2024 08:20
@samugi samugi requested a review from ADD-SP July 4, 2024 08:51
kong/observability/logs.lua Outdated Show resolved Hide resolved
kong/observability/logs.lua Show resolved Hide resolved
@samugi samugi requested a review from ADD-SP July 4, 2024 10:44
@samugi samugi force-pushed the feat/otel-formatted-logs branch 2 times, most recently from 2cd7338 to 6f94ca4 Compare July 4, 2024 15:20
Spans were not set as "active" during their lifetime.
This commit addresses the issue by setting each span as the current
"active" span in instrumentations. This is necessary in order to have
the correct span_id information in the OTel-formatted log entries.
@samugi samugi force-pushed the feat/otel-formatted-logs branch from 7e5d680 to f17eb6e Compare July 4, 2024 20:35
Copy link
Contributor

@ADD-SP ADD-SP left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, please rebase commits before the merge.

@samugi samugi force-pushed the feat/otel-formatted-logs branch from d4d9110 to 1ff59b7 Compare July 5, 2024 07:11
samugi added 4 commits July 5, 2024 11:20
When a disabled and non-request scoped hook was called outside of
a request scope, it was producing an error while trying to index ngx.ctx

This commit adds a check to early return from is_group_enabled when
outside of the request scope

This allows for hooks to still work when used in other contexts where
they can be enabled via the always_enable() function

This commit also adds support for disabling hooks.
This commit adds OpenTelemetry formatted logs as a new
signal supported by the OpenTelemetry plugin.

It patches the ngx.log function to collect the logged line and
enrich it with additional context info:

* severity
* introspection details
* timestamp

And for request-scoped logs also:

* Request ID
* Trace ID
* Span ID
* Trace flags

The log entries are protobuf-serialized and sent to the endpoint
configured in the OpenTelemetry plugin's config.logs_endpoint field.

feat(opentelemetry): add traceflags to otel logs

This commit collects an additional tracing information (trace flags) in
the internal context and adds it to produced OTel-formatted log entries.
This commit moves the tracing and logging core code in the observability
folder.
This commit adds unit and integration tests for the OTel-formatted logs
feature.
@samugi samugi force-pushed the feat/otel-formatted-logs branch from a06b405 to c1f99a9 Compare July 5, 2024 09:24
@samugi samugi merged commit 8aea938 into master Jul 5, 2024
29 checks passed
@samugi samugi deleted the feat/otel-formatted-logs branch July 5, 2024 09:55
@team-gateway-bot
Copy link
Collaborator

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13291-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13291-to-master-to-upstream
git checkout -b cherry-pick-13291-to-master-to-upstream
ancref=$(git merge-base 8f9b82d870985cdd5fb417d589ad2f7ddec967e4 c1f99a9ff2a5986dbd72e85d9897167df2124b77)
git cherry-pick -x $ancref..c1f99a9ff2a5986dbd72e85d9897167df2124b77

@github-actions github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 5, 2024
@samugi samugi removed cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention labels Jul 5, 2024
samugi added a commit to Kong/docs.konghq.com that referenced this pull request Jul 5, 2024
docs for Kong/kong#13291

Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: Hans Hübner <[email protected]>
lena-larionova added a commit to Kong/docs.konghq.com that referenced this pull request Jul 23, 2024
* feat(OpenTelemetry): support opentelemetry logs

docs for Kong/kong#13291

Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: Hans Hübner <[email protected]>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <[email protected]>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <[email protected]>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <[email protected]>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <[email protected]>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <[email protected]>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <[email protected]>

* Update app/_hub/kong-inc/opentelemetry/overview/_index.md

Co-authored-by: lena-larionova <[email protected]>

---------

Co-authored-by: Hans Hübner <[email protected]>
Co-authored-by: lena-larionova <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants