[BUG] AWS openseach source error: ElasticsearchVersionInfo.buildFlavor

[qxmips]

Hi. I am testing data prepper new openseach source functionally with this config:

opensearch-to-s3:
  delay: "10"
  workers: 4
  source:
    opensearch:
      hosts: [ "https://my-aws-opensearch-2.7:443" ]
      username: "***"
      password: "***"
      indices:
        include:
          - index_name_regex: "myindex"
      scheduling:
        interval: "PT1H"
        index_read_count: 5
      search_options:
        search_context_type: "point_in_time"
        batch_size: 10000

  buffer:
    bounded_blocking:
      buffer_size: 12800
      batch_size: 200

  sink:
    - stdout:

I use aws managed opensearch 2.7 .
and I face this error: ElasticsearchVersionInfo.buildFlavor

dataprepper-data-prepper-1  | 2023-11-08T02:42:28,268 [opensearch-to-s3-sink-worker-2-thread-1] ERROR org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor - Pipeline [opensearch-to-s3] process worker encountered a fatal exception, cannot proceed further
dataprepper-data-prepper-1  | java.util.concurrent.ExecutionException: java.lang.RuntimeException: Unable to call info API using the elasticsearch client
dataprepper-data-prepper-1  | at java.util.concurrent.FutureTask.report(Unknown Source) ~[?:?]
dataprepper-data-prepper-1  | at java.util.concurrent.FutureTask.get(Unknown Source) ~[?:?]
dataprepper-data-prepper-1  | at org.opensearch.dataprepper.pipeline.common.PipelineThreadPoolExecutor.afterExecute(PipelineThreadPoolExecutor.java:70) [data-prepper-core-2.5.0.jar:?]
dataprepper-data-prepper-1  | at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [?:?]
dataprepper-data-prepper-1  | at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [?:?]
dataprepper-data-prepper-1  | at java.lang.Thread.run(Unknown Source) [?:?]
dataprepper-data-prepper-1  | Caused by: java.lang.RuntimeException: Unable to call info API using the elasticsearch client
dataprepper-data-prepper-1  | at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.SearchAccessorStrategy.getDistributionAndVersionNumber(SearchAccessorStrategy.java:181) ~[opensearch-source-2.5.0.jar:?]
dataprepper-data-prepper-1  | at org.opensearch.dataprepper.plugins.source.opensearch.worker.client.SearchAccessorStrategy.getSearchAccessor(SearchAccessorStrategy.java:97) ~[opensearch-source-2.5.0.jar:?]
dataprepper-data-prepper-1  | at org.opensearch.dataprepper.plugins.source.opensearch.OpenSearchSource.startProcess(OpenSearchSource.java:69) ~[opensearch-source-2.5.0.jar:?]
dataprepper-data-prepper-1  | at org.opensearch.dataprepper.plugins.source.opensearch.OpenSearchSource.start(OpenSearchSource.java:58) ~[opensearch-source-2.5.0.jar:?]
dataprepper-data-prepper-1  | at org.opensearch.dataprepper.pipeline.Pipeline.startSourceAndProcessors(Pipeline.java:210) ~[data-prepper-core-2.5.0.jar:?]
dataprepper-data-prepper-1  | at org.opensearch.dataprepper.pipeline.Pipeline.lambda$execute$2(Pipeline.java:251) ~[data-prepper-core-2.5.0.jar:?]
dataprepper-data-prepper-1  | at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
dataprepper-data-prepper-1  | at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
dataprepper-data-prepper-1  | ... 3 more
dataprepper-data-prepper-1  | Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'ElasticsearchVersionInfo.buildFlavor'
dataprepper-data-prepper-1  | at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.elasticsearch._types.ElasticsearchVersionInfo.<init>(ElasticsearchVersionInfo.java:74) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.elasticsearch._types.ElasticsearchVersionInfo.<init>(ElasticsearchVersionInfo.java:50) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.elasticsearch._types.ElasticsearchVersionInfo$Builder.build(ElasticsearchVersionInfo.java:300) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.elasticsearch._types.ElasticsearchVersionInfo$Builder.build(ElasticsearchVersionInfo.java:200) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:72) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:176) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:137) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:328) ~[elasticsearch-java-7.17.0.jar:?]
dataprepper-data-prepper-1  | at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:294) ~[elasticsearch-java-7.17.0.jar:?]

It's aws manged opensearch 2.7 (override_main_response_version: true)
why does it try to use elasticsearch-java-7.17.0 instead of
opensearch-java-2.5.0 ?

[graytaylor0]

Do you have compatibility mode enabled on this domain? If so there is a distribution_version that is being added in 2.6 to handle this case (#3636)

[qxmips]

@graytaylor0
yes, the compatibility mode is enabled. I guess I have to wait for 2.6. Thank you.

[dlvenable]

This should be resolved by #3636 and available in 2.6.0.

[kclinden]

Did you figure out how to get this to work?