Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert backup chunk specification for suppressing useless chunk backup sets #247

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

bysnupy
Copy link

@bysnupy bysnupy commented Jul 11, 2022

Description:
Fluent::Unrecoverable Error makes the log set backup per a whole chunk unit, not only relevant log message.
It even causes irrelevant logs lost due to not sending whole chunk sets for backup, in other words, unrecoverable error can block sending the whole chunk data set which has any bigger logs than 256kb. Ideally, we should skip only to send the certain bigger log message than 256kb, not all chunk data including multiple log messages.

In addition the chunk backup file would keep accumulating as useless things, because there is no solution to restore the chunk backup set binary files now. Refer fluent/fluentd#3813 for potential risk of this behavior please.

…p sets

"Fluent::Unrecoverable Error" make the log set backup per a whole chunk unit, not only relevant log message.
It cause irrelevant logs lost due to whole chunk backup, in other words, not sending and backup whole chunk data set including the bigger log than 256 kb.

In addition the chunk backup file is remained as useless things, because there is no solution to restore the backup sets now.
Refer fluent/fluentd#3813
Copy link
Member

@cosmo0920 cosmo0920 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change shouldn't be accepted. Because this change introduces malformed requests which contain too large data for ingested AWS CloudWatch Logs. This should cause unnecessary consuming data bandwidths.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants