fleet: correctly register fleet agent.

[pwhelan]

Summary

This PR passes the fleet_id parameter to the calyptia output plugin when joining a fleet so it can be correctly registered as an agent.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• Run local packaging test showing all targets (including any new ones) build.
• Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[edsiper]

the PR touches two plugins:

• in_calyptia_fleet
• custom_calyptia

Please split the single commit in two and use the component as prefix as listed above.

[pwhelan]

I rewrote the history in several commits. I also went ahead and included a bunch of commits which resolve problems with the machine-id being included in the directory name where the fleet configuration is stored and loaded from.

[pwhelan]

Here is an example of me running it with a personal fleet:

/bin/fluent-bit -C calyptia -p api_key=***REDACTED*** -p fleet_name=new-fleet-test-4
Fluent Bit v2.1.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/08/21 22:10:04] [ info] [fluent bit] version=2.1.9, commit=86f68d95b3, pid=251590
[2023/08/21 22:10:04] [ info] [custom:calyptia:calyptia.0] custom initialized!
[2023/08/21 22:10:04] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/08/21 22:10:04] [ info] [cmetrics] version=0.6.3
[2023/08/21 22:10:04] [ info] [ctraces ] version=0.3.1
[2023/08/21 22:10:04] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] initializing
[2023/08/21 22:10:04] [ info] [input:fluentbit_metrics:fluentbit_metrics.0] storage_strategy='memory' (memory only)
[2023/08/21 22:10:04] [ info] [input:calyptia_fleet:calyptia_fleet.1] initializing
[2023/08/21 22:10:04] [ info] [input:calyptia_fleet:calyptia_fleet.1] storage_strategy='memory' (memory only)
[2023/08/21 22:10:04] [ info] [input:calyptia_fleet:calyptia_fleet.1] initializing calyptia fleet input.
[2023/08/21 22:10:04] [ info] [sp] stream processor started
[2023/08/21 22:10:06] [ info] [input:calyptia_fleet:calyptia_fleet.1] force the reloading of the configuration file=-1.
[2023/08/21 22:10:06] [ info] [input:calyptia_fleet:calyptia_fleet.1] loading configuration from /tmp/calyptia-fleet/***REDACTED***/new-fleet-test-4/new.ini.
[2023/08/21 22:10:11] [engine] caught signal (SIGHUP)
[2023/08/21 22:10:11] [ info] reloading instance pid=251590 tid=0x7ffff6da1ac0
[2023/08/21 22:10:11] [ info] [reload] stop everything of the old context
[2023/08/21 22:10:11] [ warn] [engine] service will shutdown when all remaining tasks are flushed
[2023/08/21 22:10:11] [ info] [input] pausing fluentbit_metrics.0
[2023/08/21 22:10:11] [ info] [input] pausing calyptia_fleet.1
[2023/08/21 22:10:12] [ info] [engine] service has stopped (0 pending tasks)
[2023/08/21 22:10:12] [ info] [input] pausing fluentbit_metrics.0
[2023/08/21 22:10:12] [ info] [input] pausing calyptia_fleet.1
[2023/08/21 22:10:12] [ info] [reload] start everything
[2023/08/21 22:10:12] [ info] [fluent bit] version=2.1.9, commit=86f68d95b3, pid=251590
[2023/08/21 22:10:12] [ info] [custom:calyptia:calyptia.0] custom initialized!
[2023/08/21 22:10:12] [ info] [custom:calyptia:calyptia.1] custom initialized!
[2023/08/21 22:10:12] [ info] [storage] ver=1.4.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/08/21 22:10:12] [ info] [cmetrics] version=0.6.3
[2023/08/21 22:10:12] [ info] [ctraces ] version=0.3.1
[2023/08/21 22:10:12] [ info] [input:dummy:dummy.0] initializing
[2023/08/21 22:10:12] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/08/21 22:10:12] [ info] [input:fluentbit_metrics:fluentbit_metrics.1] initializing
[2023/08/21 22:10:12] [ info] [input:fluentbit_metrics:fluentbit_metrics.1] storage_strategy='memory' (memory only)
[2023/08/21 22:10:12] [ info] [input:calyptia_fleet:calyptia_fleet.2] initializing
[2023/08/21 22:10:12] [ info] [input:calyptia_fleet:calyptia_fleet.2] storage_strategy='memory' (memory only)
[2023/08/21 22:10:12] [ info] [input:calyptia_fleet:calyptia_fleet.2] initializing calyptia fleet input.
[2023/08/21 22:10:12] [ info] [input:fluentbit_metrics:fluentbit_metrics.3] initializing
[2023/08/21 22:10:12] [ info] [input:fluentbit_metrics:fluentbit_metrics.3] storage_strategy='memory' (memory only)
[2023/08/21 22:10:12] [ info] [input:calyptia_fleet:calyptia_fleet.4] initializing
[2023/08/21 22:10:12] [ info] [input:calyptia_fleet:calyptia_fleet.4] storage_strategy='memory' (memory only)
[2023/08/21 22:10:12] [ info] [input:calyptia_fleet:calyptia_fleet.4] initializing calyptia fleet input.
[2023/08/21 22:10:12] [ info] [output:stdout:stdout.0] worker #0 started
[2023/08/21 22:10:12] [ info] [output:calyptia:calyptia.1] connected to Calyptia, agent_id='***REDACTED***'
[2023/08/21 22:10:12] [ info] [sp] stream processor started
{"date":1692670213.389681,"message":"dummy2"}
{"date":1692670214.389678,"message":"dummy2"}
{"date":1692670215.444314,"message":"dummy2"}
{"date":1692670216.389703,"message":"dummy2"}
{"date":1692670217.390269,"message":"dummy2"}
....
{"date":1692670259.389681,"message":"dummy2"}
^C[2023/08/21 22:11:00] [engine] caught signal (SIGINT)
{"date":1692670260.392524,"message":"dummy2"}
[2023/08/21 22:11:00] [ warn] [engine] service will shutdown in max 5 seconds
[2023/08/21 22:11:00] [ info] [input] pausing dummy.0
[2023/08/21 22:11:00] [ info] [input] pausing fluentbit_metrics.1
[2023/08/21 22:11:00] [ info] [input] pausing calyptia_fleet.2
[2023/08/21 22:11:00] [ info] [input] pausing fluentbit_metrics.3
[2023/08/21 22:11:00] [ info] [input] pausing calyptia_fleet.4
[2023/08/21 22:11:01] [ info] [engine] service has stopped (0 pending tasks)
[2023/08/21 22:11:01] [ info] [input] pausing dummy.0
[2023/08/21 22:11:01] [ info] [input] pausing fluentbit_metrics.1
[2023/08/21 22:11:01] [ info] [input] pausing calyptia_fleet.2
[2023/08/21 22:11:01] [ info] [input] pausing fluentbit_metrics.3
[2023/08/21 22:11:01] [ info] [input] pausing calyptia_fleet.4
[2023/08/21 22:11:01] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2023/08/21 22:11:01] [ info] [output:stdout:stdout.0] thread worker #0 stopped

[edsiper]

@pwhelan it seems a newer PR generated some conflicts here, would you please fix them ?

[pwhelan]

@pwhelan it seems a newer PR generated some conflicts here, would you please fix them ?

Done. I also did some style fixes as well.

[edsiper]

thanks for splitting the content... but there are 13 commits that touches 2 plugins; maybe 2 commits are good enough since there are small changes.

[edsiper]

@pwhelan ping

[pwhelan]

thanks for splitting the content... but there are 13 commits that touches 2 plugins; maybe

I've decided to split out the two main changes in this pull request into the changes to custom_calyptia to correctly set the fleet_id for out_calyptia and a new pull request that will contain the work that allows a single machine to register to multiple fleets.

That way this PR is contained in a single commit.