Log Loss Due to Compression Errors with Fluent Bit and Logback

[geri100]

Bug Report

I have a Spring Boot application that generates logs into a logs folder. I'm using Logback, which also compresses the logs and deletes older files after a specified number of days. Compression happens every minute, meaning it compresses the file every minute. It keeps a maximum of 3 days' worth of logs.

Fluent Bit only processes .gz files, so it only detects logs after they've been compressed by Logback. I've configured Logback to ensure that Fluent Bit doesn't detect the file until compression is complete. (Logback fix: First, compress the file with a template name, and then use an atomic operation to rename it so Fluent Bit can read it.)

However, every 2-3 days, for about 3-5 minutes, Fluent Bit displays errors like "[error] [input:tail:tail.0] decompression failed for logs/log.2024-11-11-13-14.json.gz" and "[error] [gzip] invalid magic bytes: d473". In other words, it somehow considers the .gz file invalid, even though I’m sure the file is correctly generated. If I re-feed that same file, Fluent Bit processes it without issues.

[error] [input:tail:tail.0] decompression failed for logs/log.2024-11-11-13-14.json.gz
[error] [gzip] invalid magic bytes: d473

Your Environment

• Version used: 3.1.6
• Configuration:
  [SERVICE]
  Daemon Off
  Flush 1
  Log_Level info
  Parsers_File /fluentbit-configuration/custom_parsers.conf
  HTTP_Server On
  HTTP_Listen 0.0.0.0
  HTTP_Port 2020
  Health_Check On
  HC_Errors_Count 1
  HC_Retry_Failure_Count 1
  HC_Period 60
  Hot_Reload On
  scheduler.base 3
  scheduler.cap 30
  storage.path /logs/fluentbit-database/storage/
  storage.sync full
  storage.checksum on
  storage.backlog.mem_limit 128MB
  storage.max_chunks_up 64
  storage.delete_irrecoverable_chunks on

[INPUT]
Name tail
Path /logs/log.*.json
DB /logs/fluentbit-database/fluentbit_database.sqlite
DB.sync FULL
Rotate_Wait 30
Parser json_parser
Buffer_Chunk_Size 128KB
Buffer_Max_Size 256k
Log_Level info
Skip_Long_Lines On
Refresh_Interval 15
Inotify_Watcher false
Static_Batch_Size 512KB
Read_from_Head true
storage.type filesystem
storage.pause_on_chunks_overlimit on

• Environment name and version (e.g. Kubernetes? What version?): Kubernetes

Additional context
A lot of valuable logs are being lost.