fluent-bit fails to forward logs when using AWS Opensearch 2.3

[gautam-nutalapati]

Describe the question/issue

I have been using AWS-Firelens + AWS-fluent-bit to forward logs from ECS to opensearch.
After upgrading AWS Opensearch from 1.3 to 2.3, firelens log forwarder fails with below error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"}],"type":"illegal_argument_exception","reason":"Action/metadata line [1] contains an unknown parameter [_type]"},"status":400}

Configuration

Based on example from amazon-ecs-firelens-examples and AWS Copilot
we have been using below config for firelens which was working before:

logging:
  image: 651627600016.dkr.ecr.us-east-1.amazonaws.com/prisidio/fluent-bit:0.0.9
  enableMetadata: true
  configFilePath: /fluent-bit/configs/custom-fluent-bit.conf
  destination:
    Name: es
    AWS_REGION: us-east-1
    Host: OPENSEARCH_DOMAIN.us-east-1.es.amazonaws.com
    Port: 443
    Index: INDEX_NAME
    AWS_Auth: On
    tls: On

Custom config:

[SERVICE]
    flush 5
    parsers_file /fluent-bit/parsers/custom-parsers.conf
# Comment below for testing
[FILTER]
    Name                  multiline
    Match                 *
    multiline.key_content log
    multiline.parser      multiline-regex
[FILTER]
    Name                  record_modifier
    Match                 *
    Record svc ${COPILOT_SERVICE_NAME}
    Remove_key            _type
    Remove_key            _score

Fluent Bit Log Output

[2022/12/20 20:16:41] [error] [output:es:es.2] HTTP status=400 URI=/_bulk, response:
[2022/12/20 20:16:37] [ warn] [engine] chunk '1-1671567385.188264483.flb' cannot be retried: task_id=0, input=emitter_for_multiline.1 > output=es.2

Fluent Bit Version Info

2.29.0

Cluster Details

Service is running on AWS ECS Fargate.
AWS Firelens is used for routing logs to AWS Opensearch.

Application Details

Simple Java web service app that prints logs to console, producing about 200 logs per minute.

Steps to reproduce issue

Try out aws-ecs-firelens-opensearch example with 2.3 opensearch domain.

Related Issues

opensearch-project/OpenSearch#5516

[PettitWesley]

I think you just need Suppress_Type_Name On with newer opensearch. Will check on this.

[PettitWesley]

Opensearch team got back to me and confirmed the above is correct:

For the issue you reported, it is because type has been removed in OpenSearch 2.0 (which was a deprecation in 1.0 and 7.10 ES),
Any client connecting to OpenSearch 2.0+ has to suppress the type parameter

[FILTER]
Name record_modifier
Match *
Record svc ${COPILOT_SERVICE_NAME}
Remove_key _type
Remove_key _score

I don't think this is how _type works. This would remove _type if it was a key that your app adds to its log records. Is that the case?

When we talk about removing the type name for opensearch, IIRC, its a key that the es or opensearch outputs would add to the request payload unless you specific Supress_Type_Name On in your config. Adding that option to the output config is the solution here.

[ivan-amity]

This one can be resolved by following this one (OpenSearch 2.0 and above )
I'm also using OpenSearch 2.3.x right now.
https://docs.fluentbit.io/manual/pipeline/outputs/opensearch#action-metadata-contains-an-unknown-parameter-type

Need to add Suppress_Type_Name On in the output configuration.

[Matrix-Zhang]

i have set Suppress_Type_Name On

but have new errors

{"took":5,"errors":true,"items":[{"create":{"_index":"fluent-bit","_id":"ZEr4XoYB8rpviE8U0CG5","status":400,"error":{"type":"mapper_parsing_exception","reason":"object mapping for [kubernetes.labels.app] tried to parse field [app] as object, but found a concrete value"}}}]}
[2023/02/17 10:43:30] [ warn] [engine] failed to flush chunk '1-1676630552.336057817.flb', retry in 49 seconds: task_id=3, input=tail.0 > output=opensearch.0 (out_id=0)

[gautam-nutalapati]

Adding Suppress_Type_Name On in the output configuration fixed my issue.

To answer my question,
Firelens confiration I used (Also changed the param Name: opensearch):

logging:
  image: 651627600016.dkr.ecr.us-east-1.amazonaws.com/prisidio/fluent-bit:0.0.9
  enableMetadata: true
  configFilePath: /fluent-bit/configs/custom-fluent-bit.conf
  destination:
    Name: opensearch
    AWS_REGION: us-east-1
    Host: OPENSEARCH_DOMAIN.us-east-1.es.amazonaws.com
    Port: 443
    Index: INDEX_NAME
    AWS_Auth: On
    tls: On
    Suppress_Type_Name: On

Custom config (Removed Remove_key ):

[SERVICE]
    flush 5
    parsers_file /fluent-bit/parsers/custom-parsers.conf
# Comment below for testing
[FILTER]
    Name                  multiline
    Match                 *
    multiline.key_content log
    multiline.parser      multiline-regex
[FILTER]
    Name                  record_modifier
    Match                 *
    Record svc ${COPILOT_SERVICE_NAME}

@Matrix-Zhang your issue seems specific to your cluster and custom parsing setup.
Closing this issue.