TLS not working with opentelemetry output plugin [minimal docker-compose reproducer included]

[ajschmidt8]

Bug Report

Describe the bug

The opentelemetry output plugin does not successfully flush traces to the backend when TLS is enabled. Instead, Fluent Bit throws the following error:

[error] [output:opentelemetry:opentelemetry.1] <HOST>:4318, HTTP status=0

To Reproduce
A docker-compose reproducer can be found here: https://github.com/ajschmidt8/fluent-bit-repro.

Expected behavior

The opentelemetry output plugin should successfully flush traces to the backend when TLS is enabled.

Screenshots

Your Environment

• Version used: 3.2.0
• Configuration:

[SERVICE]
    Flush                1
    Log_level            debug

[INPUT]
    name    opentelemetry
    listen  0.0.0.0
    port    4318

[OUTPUT]
    Name                 opentelemetry
    Match                *
    Host                 tempo-mtls
    Port                 4318
    Log_response_payload True
    Tls                  On
    Tls.verify           On
    Tls.crt_file /certs/cert.pem
    Tls.key_file /certs/key.pem
    Tls.ca_file /certs/ca.pem

• Operating System and version: Ubuntu 22.04
• Plugins: opentelemetry (input and output)

cc: @msarahan

[patrick-stephens]

3.2.1 has a fix for TLS, I think this is a duplicate of #9593 but specifically for the OTEL output.

[patrick-stephens]

See #9593 (comment)

[patrick-stephens]

@ajschmidt8 can you double check with 3.2.1?

[ajschmidt8]

@patrick-stephens, thanks for the quick reply. I've just tried the 3.2.1 container, but the issue still persists. See my screenshot below.

[nalcabio-tom]

Same with 3.2.1 (on a raspberrypi)

[patrick-stephens]

Does it work with http2 off set in the output?

[ElectricWeasel]

Does it work with http2 off set in the output?

Yes, indeed it does.

Nov 22 11:45:23 petra7.xxxx.xxx fluent-bit[2322335]: [2024/11/22 11:45:23] [ info] [output:opentelemetry:opentelemetry.1] otel.xxx.xxx:443, HTTP status=200

[ajschmidt8]

Does it work with http2 off set in the output?

Yes, indeed it does.

Nov 22 11:45:23 petra7.xxxx.xxx fluent-bit[2322335]: [2024/11/22 11:45:23] [ info] [output:opentelemetry:opentelemetry.1] otel.xxx.xxx:443, HTTP status=200

I confirmed as well. It works with http2 off:

[2024/11/22 14:14:28] [ info] [output:opentelemetry:opentelemetry.1] tempo-mtls:4318, HTTP status=200

[patrick-stephens]

Yeah so I think it's down to something in the http2 change to using as default but @leonardo-albertovich is investigating.

He did add the option to disable http2 specifically to allow us to revert back to the old system :)