the Upstream configuration file doesn't support environmental variables

[AzureLeMoon]

Bug Report

Describe the bug
the upstream config file used by the forward plugin does not support use of environmental variables and assigns empty values instead of using the env var.
this is problematic when trying to for example assign the shared_key or port parameters, since it means the parameters need to be set in plaintext inside the upstream config which would raise security concerns in case of shared_key.

To Reproduce

in case of using an env var for share_key:

[2024/07/13 14:18:34] [error] [output:forward:forward.0] failed authorization: shared_key mismatch

if the password is set in plain text however the connection succeeds.

in case of using an env var for port:

[2024/07/13 14:18:25] [debug] [upstream_ha] opening file /fluent-bit/etc/configs/upstream.conf
.
.
.
[2024/07/13 13:57:08] [debug] [net] TCP connection timed out: fluentd-03.example.com:0
[2024/07/13 13:57:08] [debug] [net] TCP connection timed out: fluentd-01.example.com:0
[2024/07/13 13:57:08] [debug] [net] could not connect to fluentd-03.example.com:0
[2024/07/13 13:57:08] [debug] [net] could not connect to fluentd-01.example.com:0
[2024/07/13 13:57:08] [debug] [upstream] connection #113 failed to fluentd-03.example.com:0
[2024/07/13 13:57:08] [debug] [upstream] connection #108 failed to fluentd-01.example.com:0

Steps to reproduce the problem:

• add env variables to the upstream config file:

[UPSTREAM]
    name       forward-balancing

[NODE]
    name       node-1
    host       fluentd-01.example.com
    port       ${FLUENTD_PORT}
    Shared_Key    ${FLUENTD_PASSWORD}
    Self_Hostname flb.local
    tls           on
    tls.verify    off

[NODE]
    name       node-2
    host       fluentd-02.example.com
    port       ${FLUENTD_PORT}
    Shared_Key    ${FLUENTD_PASSWORD}
    Self_Hostname flb.local
    tls           on
    tls.verify    off

[NODE]
    name       node-3
    host       fluentd-03.example.com
    port       ${FLUENTD_PORT}
    Shared_Key    ${FLUENTD_PASSWORD}
    Self_Hostname flb.local
    tls           on
    tls.verify    off

• set the env variables whether inside a docker container or in linux environment.

docker run -d --name fluent-bit \
-e FLUENTD_PORT=24224 \
-e FLUENTD_PASSWORD=password  \
-v /opt/fluent-bit/configs:/fluent-bit/etc/configs \
-v /opt/fluent-bit/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml \
--net=host \
cr.fluentbit.io/fluent/fluent-bit -c fluent-bit/etc/fluent-bit.yaml

• address the config file in the output plugin:

pipeline:
   outputs:
    - name: forward
      match: '*'
      upstream: configs/upstream.conf

• run fluent-bit

Expected behavior
the upstream configuration file should respect the env variables and connect on the correct port and use the correct password.

Your Environment

• Version used: v3.0.7
• Configuration: samples provided
• Environment name and version (e.g. Kubernetes? What version?): docker engine v26.1.4
• Server type and version: OpenStack VM
• Operating System and version: ubuntu 20.04LTS
• Filters and plugins: forward plugin, tail input, tcp input

Additional context
the main reason for this issue is the security concern about using plaintext passwords in a production environment and supporting env variables would be great help for security.

[AzureLeMoon]

hello @edsiper ,
i saw that this was added to the 3.1.3 milestone, are there any updates?
is there extra info you'd need that i could provide ?

[leonardo-albertovich]

Addressed in PR #9255

[AzureLeMoon]

Addressed in PR #9255

Hey there,
I saw that you changed the base parameters(keys like host,port, etc..) to be translated from env variables if applicable,

however parameters used by the caller plugin( in this case out_forward) are still stored in a hashmap which means they wont be translated from env variables, the most notable parameter is the Shared_Key which means the password for connecting to the endpoint still needs to be in plaintext.

It would be much appreciated if this was handled as well

[leonardo-albertovich]

Sure, I missed that detail and thought the only part of the system that lacked that feature was the ha stuff.
I'll add that as well =)

[leonardo-albertovich]

I think my last commit should cover the missing feature, I'd really appreciate it if you gave me some more feedback.

[AzureLeMoon]

@leonardo-albertovich
hey thanks for the follow up.
I built a docker image using your branch and tested it,
I was able to use env variables to set the Shared_Key and other parameters so everything seems to be in order.
looking forward to this change being merged and released so we can start using it in prod.

[edsiper]

thanks everyone. this will be available on 3.1.7

[edsiper]

fixed in #9255