Fluentd customized docker image for being deployed as log aggregator in Fluentbit/Fluentd forwarder/aggregation architecture pattern.
Based on official fluentd docker image, plugins have been added and default configuration has been included.
The following plugins has been added to the default fluentd image
- fluent-plugin-elasticsearch: ES as backend for routing the logs
elasticsearch-xpack gem need to be installaed as a dependency when using elasticsearch ILM policies.
- fluent-plugin-prometheus: Enabling prometheus monitoring
- fluent-plugin-record-modifier: record_modifier filter faster and lightweight than embedded transform_record filter.
- fluent-plugin-grafana-loki: Loki as backend for routing the logs
When configuring fluent-plugin-elasticsearch, a specific sniffer class need to be configured for implementing reconnection logic to ES(sniffer_class_name Fluent::Plugin::ElasticsearchSimpleSniffer). See plugin documentation fluent-plugin-elasticsearh: Sniffer Class Name.
The path to the sniffer class need to be passed as parameter to fluentd command (-r option). Docker's entrypoint.sh has been updated to automatically provide the path to the sniffer class.
Fluentd is configured:
-
to export Prometheus metrics. See [fluentd documentation] (https://docs.fluentd.org/monitoring-fluentd/monitoring-prometheus)
-
to collect logs from forwarders (Fluentbit/Fluentd in forwarding mode)
-
to route all logs to Elasticsearch backend
Github actions have been configured to automatically build amd64 and arm64 docker images.
This images are available in docker hub https://hub.docker.com/r/ricsanfre/fluentd-aggregator
To create an aggregator that collects logs from other forwarders
docker run -d -p 24224:24224 -p 24224:24224/udp ricsanfre/fluentd-aggregator:latest