Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

processor_labels: Process operations for output purposed contexts of metrics instead of the original one #9632

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

processor_labels: Process operations for output purposed contexts of metrics instead of the original one #9632

wants to merge 1 commit into from
+13 −16

Conversation

cosmo0920
Copy link
Contributor

@cosmo0920 cosmo0920 commented Nov 22, 2024 •
edited
Loading

In the current implementation, Fluent Bit somewhat doesn't update the internal values and timestamps for processing metrics contexts.
Instead, we need to create the target of cmetrics context at first, and then we process with the newly created metrics contexts.
This resolves the glitches of not updating the values and the timestamps for each of targeted contexts of cmetrics.

Closes #9565.

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
service:
  flush: 5
  daemon: off
  log_level: info

pipeline:
  inputs:
    - name: node_exporter_metrics
      tag: metrics.node_metrics
      scrape_interval: 10s
      metrics: netdev
      processors:
        metrics:
          - name: metrics_selector
            metric_name: /node_network_(transmit|receive)_bytes_total/
            action: include
          - name: labels
            insert: foo bar

  outputs:
    - name: stdout
      match: '*'
  • Debug log output from testing the change
Fluent Bit v3.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/11/22 18:10:45] [ info] Configuration:
[2024/11/22 18:10:45] [ info]  flush time     | 5.000000 seconds
[2024/11/22 18:10:45] [ info]  grace          | 5 seconds
[2024/11/22 18:10:45] [ info]  daemon         | 0
[2024/11/22 18:10:45] [ info] ___________
[2024/11/22 18:10:45] [ info]  inputs:
[2024/11/22 18:10:45] [ info]      node_exporter_metrics
[2024/11/22 18:10:45] [ info] ___________
[2024/11/22 18:10:45] [ info]  filters:
[2024/11/22 18:10:45] [ info] ___________
[2024/11/22 18:10:45] [ info]  outputs:
[2024/11/22 18:10:45] [ info]      stdout.0
[2024/11/22 18:10:45] [ info] ___________
[2024/11/22 18:10:45] [ info]  collectors:
[2024/11/22 18:10:45] [ info] [fluent bit] version=3.2.2, commit=8994bfcde5, pid=355057
[2024/11/22 18:10:45] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2024/11/22 18:10:45] [ info] [storage] ver=1.1.6, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/11/22 18:10:45] [ info] [simd    ] SSE2
[2024/11/22 18:10:45] [ info] [cmetrics] version=0.9.9
[2024/11/22 18:10:45] [ info] [ctraces ] version=0.5.7
[2024/11/22 18:10:45] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] initializing
[2024/11/22 18:10:45] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.procfs = /proc
[2024/11/22 18:10:45] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] storage_strategy='memory' (memory only)
[2024/11/22 18:10:45] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] path.sysfs  = /sys
[2024/11/22 18:10:45] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] enabled metrics netdev
[2024/11/22 18:10:45] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] [thread init] initialization OK
[2024/11/22 18:10:45] [ info] [input:node_exporter_metrics:node_exporter_metrics.0] thread instance initialized
[2024/11/22 18:10:45] [debug] [node_exporter_metrics:node_exporter_metrics.0] created event channels: read=38 write=39
[2024/11/22 18:10:45] [debug] [processor:metrics_selector:metrics_selector.0] action type INCLUDE
[2024/11/22 18:10:45] [debug] [stdout:stdout.0] created event channels: read=42 write=43
[2024/11/22 18:10:45] [ info] [output:stdout:stdout.0] worker #0 started
[2024/11/22 18:10:45] [ info] [sp] stream processor started
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="lo",foo="bar"} = 263913270
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="enp5s0",foo="bar"} = 17196790
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="wlo1",foo="bar"} = 65780869
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="lxcbr0",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_net[2024/11/22 18:10:54] [debug] [task] created task=0x5fcdca0 id=0 OK
work_transmit_bytes_total{device="lo",foo="bar"} = 263913270
[2024/11/22 18:10:54] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="enp5s0",foo="bar"} = 5634155
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="wlo1",foo="bar"} = 1881891
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="lxcbr0",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:10:54.755905270Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
[2024/11/22 18:10:54] [debug] [output:stdout:stdout.0] cmt decode msgpack returned : 1
[2024/11/22 18:10:54] [debug] [out flush] cb_destroy coro_id=0
[2024/11/22 18:10:54] [debug] [task] destroy task=0x5fcdca0 (task_id=0)
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="lo",foo="bar"} = 263954840
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="enp5s0",foo="bar"} = 17204342
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="wlo1",foo="bar"} = 65784300
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="lxcbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 263954840
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="enp5s0",foo="bar"} = 5634316
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="wlo1",foo="bar"} = 1881891
[2024/11/22 18:11:09] [debug] [task] created task=0x60c8040 id=0 OK
[2024/11/22 18:11:09] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="lxcbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
[2024/11/22 18:11:09] [debug] [output:stdout:stdout.0] cmt decode msgpack returned : 1
[2024/11/22 18:11:09] [debug] [out flush] cb_destroy coro_id=1
[2024/11/22 18:11:09] [debug] [task] destroy task=0x60c8040 (task_id=0)
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="lo",foo="bar"} = 263954840
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="enp5s0",foo="bar"} = 17204342
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="wlo1",foo="bar"} = 65784300
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="lxcbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 263954840
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="enp5s0",foo="bar"} = 5634316
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="wlo1",foo="bar"} = 1881891
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="lxcbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:11:04.754084279Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
[2024/11/22 18:11:14] [debug] [task] created task=0x6177ab0 id=0 OK
[2024/11/22 18:11:14] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/11/22 18:11:14] [debug] [output:stdout:stdout.0] cmt decode msgpack returned : 1
[2024/11/22 18:11:14] [debug] [out flush] cb_destroy coro_id=2
[2024/11/22 18:11:14] [debug] [task] destroy task=0x6177ab0 (task_id=0)
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="lo",foo="bar"} = 263996412
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="enp5s0",foo="bar"} = 17209987
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="wlo1",foo="bar"} = 65787152
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="lxcbr0",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_receive_bytes_total{device="docker0",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="lo",foo="bar"} = 263996412
[2024/11/22 18:11:29] [debug] [task] created task=0x6225300 id=0 OK
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="enp5s0",foo="bar"} = 5634619
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="wlo1",foo="bar"} = 1881891
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="lxcbr0",foo="bar"} = 0
[2024/11/22 18:11:29] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="virbr1",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="virbr0",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="virbr2",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="br-4acfacd9185e",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="br-71a6ef278695",foo="bar"} = 0
2024-11-22T09:11:14.762178528Z node_network_transmit_bytes_total{device="docker0",foo="bar"} = 0
[2024/11/22 18:11:29] [debug] [output:stdout:stdout.0] cmt decode msgpack returned : 1
[2024/11/22 18:11:29] [debug] [out flush] cb_destroy coro_id=3
[2024/11/22 18:11:29] [debug] [task] destroy task=0x6225300 (task_id=0)
^C[2024/11/22 18:11:30] [engine] caught signal (SIGINT)
[2024/11/22 18:11:30] [ warn] [engine] service will shutdown in max 5 seconds
[2024/11/22 18:11:30] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread pause instance
[2024/11/22 18:11:30] [ info] [engine] service has stopped (0 pending tasks)
[2024/11/22 18:11:30] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread pause instance
[2024/11/22 18:11:30] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/11/22 18:11:30] [ info] [output:stdout:stdout.0] thread worker #0 stopped
[2024/11/22 18:11:30] [debug] [input:node_exporter_metrics:node_exporter_metrics.0] thread exit instance
  • Attached Valgrind output that shows no leaks or memory corruption was found
==355057== 
==355057== HEAP SUMMARY:
==355057==     in use at exit: 0 bytes in 0 blocks
==355057==   total heap usage: 11,495 allocs, 11,495 frees, 4,019,325 bytes allocated
==355057== 
==355057== All heap blocks were freed -- no leaks are possible
==355057== 
==355057== For lists of detected and suppressed errors, rerun with: -s
==355057== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

@cosmo0920
processor_labels: Process operations for output purposed contexts of …
7114b97 
…metrics instead of the original one

Signed-off-by: Hiroshi Hatake <[email protected]>
@cosmo0920 cosmo0920 added this to the Fluent Bit v3.2.2 milestone Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@edsiper edsiper Awaiting requested review from edsiper edsiper is a code owner

@leonardo-albertovich leonardo-albertovich Awaiting requested review from leonardo-albertovich leonardo-albertovich is a code owner

@fujimotos fujimotos Awaiting requested review from fujimotos fujimotos is a code owner

@koleini koleini Awaiting requested review from koleini koleini is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
docs-required
Projects
None yet
Milestone
Fluent Bit v3.2.2
Development

Successfully merging this pull request may close these issues.

node_exporter_metrics' netdev timestamp doesn't update when inserting label in processor
1 participant
@cosmo0920