es_out: support Upstream Servers

[claralp]

This should enable Upstream feature (https://docs.fluentbit.io/manual/configuration/upstream_servers) support for the Elasticsearch Output Plugin.

It was tested in a local setup with two Elasticsearch instances (es-1 & es-2).
Here is some example output when only es-1 is available:

Fluent Bit v1.3.0
Copyright (C) Treasure Data

[2019/09/03 10:40:44] [debug] [storage] [cio stream] new stream registered: stdin.0
[2019/09/03 10:40:44] [ info] [storage] initializing...
[2019/09/03 10:40:44] [ info] [storage] in-memory
[2019/09/03 10:40:44] [ info] [storage] normal synchronization mode, checksum disabled, max_chunks_up=128
[2019/09/03 10:40:44] [ info] [engine] started (pid=32277)
[2019/09/03 10:40:44] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2019/09/03 10:40:44] [debug] [upstream_ha] opening file ****/es-balancing
[2019/09/03 10:40:44] [debug] [router] default match rule stdin.0:es.0
[2019/09/03 10:40:44] [ info] [sp] stream processor started
9
[2019/09/03 10:41:13] [debug] [task] created task=0x1fdfb00 id=0 OK
[2019/09/03 10:41:13] [ info] [out_es] trying node es-1
[2019/09/03 10:41:14] [debug] [out_es] HTTP Status=200 URI=/_bulk
[2019/09/03 10:41:14] [debug] [out_es Elasticsearch response
{"took":26,"errors":false,"items":[{"index":{"_index":"fluent-bit","_type":"flb_type","_id":"cDpI9mwBjwEjY-Ep_CwT","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":1,"_primary_term":1,"status":201}}]}
[2019/09/03 10:41:14] [debug] [task] destroy task=0x1fdfb00 (task_id=0)
8
[2019/09/03 10:41:38] [debug] [task] created task=0x1f41880 id=0 OK
[2019/09/03 10:41:38] [ info] [out_es] trying node es-2
[2019/09/03 10:41:42] [error] [io] TCP connection failed: x.x.x.x:9200 (No route to host)
[2019/09/03 10:41:42] [error] [out_es] no upstream connections available
[2019/09/03 10:41:42] [debug] [retry] new retry created for task_id=0 attemps=1
[2019/09/03 10:41:42] [debug] [sched] retry=0x1f36580 0 in 6 seconds
[2019/09/03 10:41:47] [ info] [out_es] trying node es-1
[2019/09/03 10:41:48] [debug] [out_es] HTTP Status=200 URI=/_bulk
[2019/09/03 10:41:48] [debug] [out_es Elasticsearch response
{"took":13,"errors":false,"items":[{"index":{"_index":"fluent-bit","_type":"flb_type","_id":"cTpJ9mwBjwEjY-EpgCzi","_version":1,"result":"created","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":2,"_primary_term":1,"status":201}}]}
[2019/09/03 10:41:48] [debug] [task] destroy task=0x1f41880 (task_id=0)
^C[engine] caught signal (SIGINT)
[2019/09/03 11:40:01] [ info] [input] pausing stdin.0
[engine] caught signal (SIGSEGV)

[edsiper]

can you please test it rebasing on top of GIT master ?

[edsiper]

if this represents a target elasticsearch node I will propose struct flb_elasticsearch_node

[edsiper]

can you please test it rebasing on top of GIT master ?

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[mabrarov]

Hi @claralui, @edsiper and @PettitWesley,

This is a really needful feature (without which my team cannot use Fluent Bit). Are there any plans for this pull request? Is there something I can do to help get it merged? Not sure I can help to resolve merge conflicts in this pull request, because source branch doesn't belong to my repository.

Thank you.

[mabrarov]

Hi colleagues,

Just FYI, I'm working on mabrarov/fluent-bit@master...feature/out_es_upstream_support (current state - it builds successfully, but I never tried to run it), which is based (idea, some parts of the code) on this pull request and on Forward output plugin.

The plan is:

1. Implement basic support of Upstream Servers (only parameters known to create_node function, like host and port).
2. Implement support of overriding additional parameters, like index.

The 2nd change may need to go into a dedicated pull request to reduce size of pull request. IMHO, the most of use cases when support of Upstream Servers is needed in Elasticsearch output plugin should be covered by the 1st change.

Thank you.

[claralp]

Hi,
back then we solved this use-case in my team with an internal workaround.
I’m recently a bit short on time, hopefully next weekend I can check on how to rebase this with the current master. Will check out your branch as well @mabrarov

[mabrarov]

Hi @claralui,

FYI, there 2 known to me workarounds:

1. Fluent Bit (running as DaemonSet in K8s in my case) → Fluentd (supports multiple Elasticsearch hosts, can run on multiple hosts, because Fluent Bit Forward plugin supports Upstream Servers) → Elasticsearch cluster
2. Fluent Bit → (HTTP) load balancer (F5 BIG-IP in my case) → Elasticsearch cluster

My team is going with 1st workaround for now (because we already have Fluentd DaemonSet in OpenShift → Fluentd → Elasticsearch cluster), but ideally it would be great to avoid extra hop b/w Fluent Bit DaemonSet and Elasticsearch cluster.

Thank you for update.

[mabrarov]

FYI, I have successfully completed manual testing of happy path on feature/out_es_upstream_support branch for single node and for 3 node Elasticsearch clusters. Refer to https://github.com/mabrarov/elastic-stack for Docker Compose projects for Elasticsearch clusters and for Fluent Bit configurations.

[mabrarov]

FYI part 2 from that comment is ready in feature/out_es_upstream_support_extended branch. So far tested manually happy path only.

[mabrarov]

Opened pull request #7608 for Elasticsearch plugin changes and fluent/fluent-bit-docs#1143 for respective documentation changes. Please note that pull request #7608 implements part 2 from that comment, i.e. covers more than this pull request and fixes existing tests for Elasticsearch output plugin (which required changes in test supporting part of Fluent Bit engine).

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.