[Q&A] ECS agent logs marked as unreadable by Fluentd

[JuliannaBrandao]

What is a problem?

ECS Agent tail logging not working anymore after changing the image to Amazon Linux 2023. The configuration is the same as before.

The logs only show <file.log> unreadable. It is excluded and would be examined next time., no more errors or warnings even when changing the fluentd logging to trace, so it doesn't look like the issue is rotation.

The first source does work as expected, the second one is the one not working, so I know Kinesis is properly configured.

What I have done and did not work:

• Changed permission to the folder and files to 777
• Changed the owner of the folder and files
• Changed the path to the entire directory instead of specific files
• Changed encoding from the log files from ascii to utf-8
• Added the configuration:

    read_from_head true
    follow_inodes true
    rotate_wait 30

I wonder if my issue is related to this request: #4361

Original LS (before fix attempts):

sudo ls -l /var/log/ecs
total 212
-rw-r--r--. 1 root root    674 Nov  7 15:08 audit.log
-rw-r--r--. 1 root root    908 Nov  7 09:57 audit.log.2024-11-07-09
-rw-r--r--. 1 root root   6285 Nov  7 10:56 audit.log.2024-11-07-10
-rw-r--r--. 1 root root   4656 Nov  7 11:59 audit.log.2024-11-07-11
-rw-r--r--. 1 root root   3963 Nov  7 12:57 audit.log.2024-11-07-12
-rw-r--r--. 1 root root   5670 Nov  7 13:58 audit.log.2024-11-07-13
-rw-r--r--. 1 root root   2574 Nov  7 14:52 audit.log.2024-11-07-14
-rw-r--r--. 1 root root    345 Nov  7 15:07 ecs-agent.log
-rw-r--r--. 1 root root 116513 Nov  7 09:58 ecs-agent.log.2024-11-07-09
-rw-r--r--. 1 root root  12917 Nov  7 10:59 ecs-agent.log.2024-11-07-10
-rw-r--r--. 1 root root   3516 Nov  7 11:43 ecs-agent.log.2024-11-07-11
-rw-r--r--. 1 root root   2432 Nov  7 12:50 ecs-agent.log.2024-11-07-12
-rw-r--r--. 1 root root   4686 Nov  7 13:59 ecs-agent.log.2024-11-07-13
-rw-r--r--. 1 root root   1840 Nov  7 14:59 ecs-agent.log.2024-11-07-14
-rw-r--r--. 1 root root   9736 Oct 23 13:08 ecs-init.log
-rw-r--r--. 1 root root    288 Nov  7 09:48 ecs-volume-plugin.log
drw-r--r--. 2 root root      6 Oct 23 13:02 exec

Describe the configuration of Fluentd

`
@type tail
path /var/log/containers/*
pos_file /tmp/containers.log.pos
read_from_head true
tag docker
from_encoding UTF-8
encoding UTF-8
format multiline
format_firstline /\d{4}-\d{2}-\d{2}T/
format1 /^(?\d{4}-\d{2}-\d{2}T.*)/
refresh_interval 60s
multiline_flush_interval 3s

@type tail path /var/log/ecs/ecs-agent.log.* pos_file /tmp/ecs-agent.log.pos read_from_head true tag ecs-agent format none @type monitor_agent bind 127.0.0.1 port 24220 @type record_transformer tags application @type record_transformer tags ecs-agent

<match **>
@type kinesis_streams_aggregated
compression zlib
stream_name <stream_name>
region eu-west-1
flush_interval 1
flush_thread_interval 0.1
flush_thread_burst_interval 0.01
flush_thread_count 15
chunk_limit_size 1m
buffer_type file
buffer_path /tmp/td-agent/buffer/log
flush_at_shutdown true
retry_max_times 15
retry_max_interval 5s

log_level error `

Describe the logs of Fluentd

2024-11-07 14:53:16 +0000 [warn]: #0 fluent/log.rb:383:warn: /var/log/ecs/audit.log unreadable. It is excluded and would be examined next time.
2024-11-07 14:53:16 +0000 [warn]: #0 fluent/log.rb:383:warn: /var/log/ecs/audit.log.2024-11-07-13 unreadable. It is excluded and would be examined next time.
2024-11-07 14:53:16 +0000 [warn]: #0 fluent/log.rb:383:warn: /var/log/ecs/ecs-agent.log unreadable. It is excluded and would be examined next time.
2024-11-07 14:53:16 +0000 [warn]: #0 fluent/log.rb:383:warn: /var/log/ecs/ecs-agent.log.2024-11-07-13 unreadable. It is excluded and would be examined next time.
2024-11-07 14:53:16 +0000 [warn]: #0 fluent/log.rb:383:warn: /var/log/ecs/ecs-init.log unreadable. It is excluded and would be examined next time.
2024-11-07 14:53:16 +0000 [warn]: #0 fluent/log.rb:383:warn: /var/log/ecs/ecs-volume-plugin.log unreadable. It is excluded and would be examined next time.
2024-11-07 14:53:16 +0000 [warn]: #0 fluent/log.rb:383:warn: /var/log/ecs/exec unreadable. It is excluded and would be examined next time.

Environment

- Fluentd version: 1.16.5
- Fluent Package version: 5.0.4 
- Operating system: Amazon Linux 2023.5.20240730 Fedora
- Kernel version: 6.1.97-104.177.amzn2023.x86_64