fix(opensearch-provider): fix working with opensearch 2.x.x #16

serge-r · 2023-01-18T11:58:45Z

Signed-off-by: serge-r [email protected]

Description

Hello!

Some time ago, I tried to use this provider to support our Opensearch 2.4.0 infrastructure in Terraform. But I have found the next problems:

version mismatch problem - Opensearch is using different version numbers, instead of Elasticsearch, but this provider is checking it only for Elastic, which makes this provider not usable for Opensearch.
deprecated API for index templates - elastic7.client.IndexDeleteTemplate and elastic7.client.IndexGetTemplate are deprecated functions and lead to problems during usage, I have changed it to actual versions.
elastic6 support - I'm not sure, that elastic6 lib is actually for Opensearch because Opensearch has been made from Elasticsearch 7 versions. I also didn't find info about API compatibility with Elasticsearch version 6. I have removed the version-checking mechanism in this PR, and it makes elastic6 lib unusable. I preferred to remove elastic6 lib too.

Perhaps there are more problems, but at this time I have tested only Indexes and IndexTemplates resources.

I know also about https://github.com/phillbaker/terraform-provider-elasticsearch, which this provider was forked from, but I think, that is more about Elasticsearch, not Opensearch. Unfortunately, this provider also has the same problems with version detection.

Please say what are you thinking about this PR. And also big thanks for your and Phillbaker's work!

Issues Resolved

version mismatch problem
deprecated API for index templates

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

haarchri · 2023-03-04T14:56:01Z

can we see this PR merged soon ?

prudhvigodithi · 2023-03-09T21:56:34Z

Hey thanks for the PR @serge-r and apologies for the long wait, ready to roll.
@serge-r can you please fix the conflicts and we can merge this PR @phillbaker can you please review ?
Thank you
@bbarani @peterzhuamazon

serge-r · 2023-03-14T05:25:51Z

Thank you guys and sorry for delay, I will check and send a response ASAP

andrejvanderzee · 2023-03-16T11:48:52Z

For us, the resources opensearch_index_template and opensearch_dashboard_object are not working (applies successfully but no result in AWS OpenSearch 2.3):

resource "opensearch_index_template" "index_template" {
  for_each = var.indices
  name     = each.key
  body     = <<EOF
{
  "index_patterns": ["${each.key}*"],
  "template" : {
    "settings": {
      "plugins.index_state_management.rollover_alias": "${each.key}"
   }
  }
}
EOF
}

resource "opensearch_dashboard_object" "index_pattern" {
  for_each = var.indices
  body     = <<EOF
[
  {
    "_id": "index-pattern:${each.key}",
    "_type": "_doc",
    "_source": {
      "type": "index-pattern",
      "index-pattern": {
        "title": "${each.key}-*",
        "timeFieldName": "@timestamp"
      }
    }
  }
]
EOF
}

Happy to test when fixes are implemented.

justinhauer · 2023-03-21T14:48:21Z

@phillbaker @jackson-theisen @prudhvigodithi or @peterzhuamazon can one of you please review this? I also need this feature.

serge-r · 2023-03-23T12:52:31Z

Hello guys!

I have pushed fixed version. Please review.

@andrejvanderzee I tested opensearch_dashboard_object and opensearch_index_template with self-hosted opensearch and dashboard version 2.3, everything working normally. Unfortunately I have no ability to test it with AWS opensearch.

I think that need to use this version carefully, because Opensearch can use different API - at this time this provider using elastic7 library. Perhaps it will work better with native opensearch go lib.

…astic6 lib, remove version check Signed-off-by: Sergei Razukov <[email protected]>

andrejvanderzee · 2023-03-24T16:04:53Z

@serge-r The index templates are correctly created with your fork on AWS Opensearch 2.5. But, it keeps giving me changes after multiple applies:

  # opensearch_index_template.index_template["kube-shared"] will be updated in-place
  ~ resource "opensearch_index_template" "index_template" {
      ~ body = jsonencode(
          ~ {
              + index_patterns  = [
                  + "kube-shared*",
                ]
              - index_templates = [
                  - {
                      - index_template = {
                          - index_patterns = [
                              - "kube-shared*",
                            ]
                          - template       = {
                              - settings = {
                                  - index = {
                                      - number_of_replicas = "1"
                                      - number_of_shards   = "1"
                                      - plugins            = {
                                          - index_state_management = {
                                              - rollover_alias = "kube-shared"
                                            }
                                        }
                                      - refresh_interval   = "60s"
                                    }
                                }
                            }
                        }
                      - name           = "kube-shared"
                    },
                ] -> null
              + template        = {
                  + settings = {
                      + index                                           = {
                          + number_of_replicas = 1
                          + number_of_shards   = 1
                          + refresh_interval   = "60s"
                        }
                      + "plugins.index_state_management.rollover_alias" = "kube-shared"
                    }
                }
            }
        )
        id   = "kube-shared"
        name = "kube-shared"
    }

andrejvanderzee · 2023-03-28T07:36:40Z

@serge-r The problem is with keeping appplying non-changes while it already has been applied, on AWS you soon run into errors such as the below (for opensearch_index, which gives similar problem in the official beta-release):

Error: elastic: Error 429 (Too Many Requests): index [kubernetes-events-dev-000001] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block]; [type=cluster_block_exception]
│ 
│   with opensearch_index.bootstrap["kubernetes-events-dev"],
│   on indices.tf line 19, in resource "opensearch_index" "bootstrap":
│   19: resource "opensearch_index" "bootstrap" {

Would be nice if the above can be fixed.

andrejvanderzee · 2023-04-03T14:52:14Z

@phillbaker @prudhvigodithi @peterzhuamazon @jackson-theisen Can someone please review? Looks good to me at least until the Opensearch Go client is used by this provider.

prudhvigodithi · 2023-04-04T04:15:36Z

Hey @serge-r I see the tests are failing also, I assume the PR is not rebased can you please check? Also @phillbaker can you add your initial thoughts on this?
Thank you
@peterzhuamazon @bbarani

phillbaker · 2023-04-05T02:01:44Z

Thank you for the contribution. Right now, I'd like to focus on getting a v1 of this provider out the door. I'm happy to fully review this once that happens.

At a high level, this PR is pretty large and widely impacting. It should be broken up into separate PRs, one for each major change. E.g. a first PR for dropping elasticsearch 6 support, a second PR for deprecated API for index templates, etc. Those smaller PRs will also make merging individual fixes easier without having to wait for the entire PR to be ready.

Overall, these changes will need to have appropriate updates to the tests.

andrejvanderzee · 2023-04-06T13:25:48Z

@serge-r Do you have an example of opensearch_dashboard_object that works for you? This does not work for me:

resource "opensearch_dashboard_object" "index_pattern" {
  for_each = var.indices
  body     = <<EOF
[
  {
    "_id": "index-pattern:${each.key}",
    "_type": "_doc",
    "_source": {
      "type": "index-pattern",
      "index-pattern": {
        "title": "${each.key}-*",
        "timeFieldName": "@timestamp"
      }
    }
  }
]
EOF
}

It applies, but in AWS Opensearch 2.5 I see now saved object / index pattern.

serge-r · 2023-04-14T06:39:10Z

Hello!

Sorry for delay, unfortunately I can't spend many times on it(

Removing elastic6 it's already huge amount of changes, but I understand the problem and try to divide the PR to small PRs.

@andrejvanderzee about consistent changing errors - I will check, thank you for the info

about problem with opensearch_dashboard_object - check please which index in AWS using for store dashboard objects, perhaps it have different name. Default index in provider is ".dashboard". In AWS it can be ".kibana" or any other name. You can set dashboards index via index field in the resource like

resource "opensearch_dashboard_object" "index_pattern" {
  index = ".kibana"
  for_each = var.indices
  body     = <<EOF
[
  {
    "_id": "index-pattern:${each.key}",
    "_type": "_doc",
    "_source": {
      "type": "index-pattern",
      "index-pattern": {
        "title": "${each.key}-*",
        "timeFieldName": "@timestamp"
      }
    }
  }
]
EOF
}

andrejvanderzee · 2023-04-14T09:35:05Z

Thanks @serge-r adding index = ".kibana" made it work.

xsnrg · 2023-04-21T19:15:50Z

Any thoughts on when this might get merged and a new version made available?

prudhvigodithi · 2023-04-26T16:40:37Z

At a high level, this PR is pretty large and widely impacting. It should be broken up into separate PRs, one for each major change. E.g. a first PR for dropping elasticsearch 6 support, a second PR for deprecated API for index templates, etc. Those smaller PRs will also make merging individual fixes easier without having to wait for the entire PR to be ready.

Overall, these changes will need to have appropriate updates to the tests.

Hey @serge-r based on @phillbaker suggestion, can you please split the changes into different PR's and adding unit tests for each of them? I have created an issue to track the 2.x support #39 please check. Also if you are ok we can close this PR and move forward with smaller PRs that can track 2.x support.
Thank you
@bbarani @peterzhuamazon @rishabh6788

serge-r · 2023-05-02T06:59:09Z

Hello guys, and sorry for waiting. I have made new PR with less amount of changes, please review.

#41

bbarani · 2023-05-04T16:24:14Z

@prudhvigodithi @phillbaker Can we prioritize this PR? It would be beneficial for lots of users using 2.x version.

andrejvanderzee · 2023-05-04T16:26:47Z

+1

elchananmizrachi · 2023-06-28T06:46:41Z

@prudhvigodithi @phillbaker Can someone please take a look at this PR please?

markdboyd · 2023-07-05T20:36:07Z

What is the minimum change necessary to support Opensearch 2.x.x? Is it just this?:

version mismatch problem - Opensearch is using different version numbers, instead of Elasticsearch, but this provider is checking it only for Elastic, which makes this provider not usable for Opensearch.

Or do we need the Elastic 6/7 deprecations as well to support Opensearch 2.x.x?

justinhauer · 2023-07-11T20:04:30Z

totally appreciate everyone's time in the open source project, any updates on this?

shurkus · 2023-07-13T10:51:24Z

Any news?

stevesimpson418 · 2023-07-17T13:37:39Z

This would be huge for those in the community running OpenSearch 2.x.x

prudhvigodithi · 2023-08-05T03:07:10Z

Closing this PR, please check the latest PR that has all the support for 2.x #62.
Thank you

prudhvigodithi · 2023-08-08T19:59:18Z

Hey All, we just released the Beta version of OpenSearch Terraform Provider with 2.x support. We aim to release the non-beta version based on the feedback. Additionally, we encourage and welcome contributions from the community. If you come across any issues or wish to expand the coverage of APIs, we invite you to contribute to the project.
Provider Download link: https://registry.terraform.io/providers/opensearch-project/opensearch/2.0.0-beta.1

Thank you

serge-r force-pushed the opensearch-version branch 2 times, most recently from 732a233 to 4952486 Compare January 19, 2023 11:28

serge-r force-pushed the opensearch-version branch from 4952486 to d09a94a Compare March 23, 2023 12:45

serge-r requested review from peterzhuamazon, prudhvigodithi, jackson-theisen and phillbaker as code owners March 23, 2023 12:45

serge-r force-pushed the opensearch-version branch 3 times, most recently from 3d08113 to 3b8b81d Compare March 23, 2023 13:24

fix(opensearch-provider): fix deprecated indexTemplate API, remove el…

0f50f68

…astic6 lib, remove version check Signed-off-by: Sergei Razukov <[email protected]>

serge-r force-pushed the opensearch-version branch from 3b8b81d to 0f50f68 Compare March 23, 2023 13:27

This was referenced Apr 25, 2023

[BUG] Error: opensearch version 2.5.0 is older than 6.0.0 and is not supported, flavor: 0. #36

Closed

[BUG] Update to use _plugin in request paths #32

Closed

prudhvigodithi mentioned this pull request Apr 26, 2023

[FEATURE] Support 2.x version of OpenSearch #39

Closed

serge-r mentioned this pull request May 2, 2023

fix(provider): fix working with opensearch #41

Closed

prudhvigodithi mentioned this pull request Aug 2, 2023

Provider 2.x support #62

Merged

prudhvigodithi closed this Aug 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(opensearch-provider): fix working with opensearch 2.x.x #16

fix(opensearch-provider): fix working with opensearch 2.x.x #16

serge-r commented Jan 18, 2023 •

edited

Loading

haarchri commented Mar 4, 2023

prudhvigodithi commented Mar 9, 2023

serge-r commented Mar 14, 2023

andrejvanderzee commented Mar 16, 2023 •

edited

Loading

justinhauer commented Mar 21, 2023

serge-r commented Mar 23, 2023

andrejvanderzee commented Mar 24, 2023 •

edited

Loading

andrejvanderzee commented Mar 28, 2023 •

edited

Loading

andrejvanderzee commented Apr 3, 2023

prudhvigodithi commented Apr 4, 2023

phillbaker commented Apr 5, 2023

andrejvanderzee commented Apr 6, 2023

serge-r commented Apr 14, 2023

andrejvanderzee commented Apr 14, 2023

xsnrg commented Apr 21, 2023

prudhvigodithi commented Apr 26, 2023

serge-r commented May 2, 2023

bbarani commented May 4, 2023

andrejvanderzee commented May 4, 2023

elchananmizrachi commented Jun 28, 2023

markdboyd commented Jul 5, 2023

justinhauer commented Jul 11, 2023

shurkus commented Jul 13, 2023

stevesimpson418 commented Jul 17, 2023

prudhvigodithi commented Aug 5, 2023

prudhvigodithi commented Aug 8, 2023

fix(opensearch-provider): fix working with opensearch 2.x.x #16

fix(opensearch-provider): fix working with opensearch 2.x.x #16

Conversation

serge-r commented Jan 18, 2023 • edited Loading

Description

Issues Resolved

haarchri commented Mar 4, 2023

prudhvigodithi commented Mar 9, 2023

serge-r commented Mar 14, 2023

andrejvanderzee commented Mar 16, 2023 • edited Loading

justinhauer commented Mar 21, 2023

serge-r commented Mar 23, 2023

andrejvanderzee commented Mar 24, 2023 • edited Loading

andrejvanderzee commented Mar 28, 2023 • edited Loading

andrejvanderzee commented Apr 3, 2023

prudhvigodithi commented Apr 4, 2023

phillbaker commented Apr 5, 2023

andrejvanderzee commented Apr 6, 2023

serge-r commented Apr 14, 2023

andrejvanderzee commented Apr 14, 2023

xsnrg commented Apr 21, 2023

prudhvigodithi commented Apr 26, 2023

serge-r commented May 2, 2023

bbarani commented May 4, 2023

andrejvanderzee commented May 4, 2023

elchananmizrachi commented Jun 28, 2023

markdboyd commented Jul 5, 2023

justinhauer commented Jul 11, 2023

shurkus commented Jul 13, 2023

stevesimpson418 commented Jul 17, 2023

prudhvigodithi commented Aug 5, 2023

prudhvigodithi commented Aug 8, 2023

serge-r commented Jan 18, 2023 •

edited

Loading

andrejvanderzee commented Mar 16, 2023 •

edited

Loading

andrejvanderzee commented Mar 24, 2023 •

edited

Loading

andrejvanderzee commented Mar 28, 2023 •

edited

Loading