Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fluentd not starting if backend not reachable #2786

Open
sgohl opened this issue Jan 21, 2020 · 3 comments
Open

Fluentd not starting if backend not reachable #2786

sgohl opened this issue Jan 21, 2020 · 3 comments
Assignees
Labels
enhancement Feature request or improve operations

Comments

@sgohl
Copy link

sgohl commented Jan 21, 2020

Perhaps this is connected to #1280 and it seems like a general mistake in design.
A failed backend connection definitely must not prevent fluentd from starting at all.

/usr/lib/ruby/2.5.0/net/http.rb:939:in `rescue in block in connect': Failed to open TCP connection to couch:5984 (getaddrinfo: Try again) (SocketError)
	from /usr/lib/ruby/2.5.0/net/http.rb:936:in `block in connect'

I am using the official Docker image fluent/fluentd:latest

Fluentd is widely used to fire-and-forget messages, often without really caring if some messages get lost.
It is a complete show-stopper if it refuses to start when backends are not available.

There should be a configuration switch to silently ignore outages and just warn on start, at least.

@repeatedly
Copy link
Member

There should be a configuration switch to silently ignore outages and just warn on start, at least.

This seems acceptable idea. Maybe, some plugins don't work when configure/start raises an error.
But if users reviewed the code and no problem, this behaviour is useful.

@repeatedly repeatedly self-assigned this Jan 21, 2020
@repeatedly repeatedly added the enhancement Feature request or improve operations label Jan 21, 2020
@sgohl
Copy link
Author

sgohl commented Jan 21, 2020

I'm actually afraid, this would result in not retrying connecting the backend ever.
If the buffer directive could be enhanced to accept connection errors and a configurable interval for retrying, that would be the best solution, I think.

@repeatedly
Copy link
Member

If the buffer directive could be enhanced to accept connection errors and a configurable interval for retrying,

buffer can't help this situation. buffer is used after pipeline started.
fluentd stops for configure/start phase error because there are unrecoverable errors, e.g. incorrect host/port/authn/authz parameter. fluentd can't judge errors are configuration mistake or expected unreachable situation.
Adding @ignore_startup_error to help latter case.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Feature request or improve operations
Projects
None yet
Development

No branches or pull requests

2 participants