-
Notifications
You must be signed in to change notification settings - Fork 26
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Empty file check for GZip content-type #55
Labels
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We are evaluating the GZip content-type to reduce the network latency between the Logstash and Google CloudStorage, and we identified one issue related to empty files.
like when there is no content to flush within configured interval it will just create empty files in google cloud storage.
Will check if someone has any workaround for the same. anyway I will propose a change
please approve or suggest a better solution / workaround for the same
Logstash information:
Please include the following information:
bin/logstash --version
) 8.14.3JVM (e.g.
java -version
): temurin-11If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:
java -version
)JAVA_HOME
environment variable if set.OS version (
uname -a
if on a Unix-like system): 23.6.0 DarwinDescription of the problem including expected versus actual behavior: the expected behaviour is once the interval is finished and there is no content to flush to GCS Bucket it should ignore and rotate the temp file. The Actual behaviour is once the interval period is over and there is no content in the temp GZip it just writes the empty file to GCS Bucket
Steps to reproduce:
Please include a minimal but complete recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
use below output plugin configuration
google_cloud_storage {
bucket => "bucket-abc"
temp_directory => "/tmp/"
log_file_prefix => "AnyPrefix"
max_file_size_kbytes => 5120
max_concurrent_uploads => 5
codec => plain { format => "%{message}" }
output_format => "json"
date_pattern => "%Y-%m-%dT%H-%M-00"
flush_interval_secs => 5
gzip => true
gzip_content_encoding => false
uploader_interval_secs => 60
include_uuid => true
include_hostname => true
}
Install the google_cloud_storage plugin
start the logstash with the given conf
wiat for some time, the empty GZips will be created in the GCS bucket
Provide logs (if relevant): none as of now
The text was updated successfully, but these errors were encountered: