[BUG] Opensearch ISM Policy resource

[dirnikk]

Terraform version: 1.4.6
Opensearch Terraform provider: 2.0.0
Opensearch cluster hosted in AWS: OpenSearch_2_7_R20230928

It seems that there are 2 issues with the resource opensearch_ism_policy:

1. When I run the following terraform code:

resource "opensearch_ism_policy" "studio-retention-policy" {
  policy_id = "studio-retention-policy"
  body      = file("../index-policies/studio-retention-policy.json")
}

If the resource is not created beforehand it gives the following error:

Error: error getting policy: /_plugins/_ism/policies/retention-policy : elastic: Error 404 (Not Found): Policy not found [type=status_exception]

  with opensearch_ism_policy.retention-policy,
  on main.tf line 162, in resource "opensearch_ism_policy" "retention-policy":
 162: resource "opensearch_ism_policy" "retention-policy" {

The correct behaviour should be if there is no such policy let Terraform create it. This is the logic for a resource in Terraform in general. If I want to manager resources created outside of Terraform I should import them.
The resource works like semi-apply-import thing.
I noticed similar behaviour with role_mapping resource the difference is that once the changes from Terraform are applied it does not tries to reapply the same changes on next Terraform run.

2. And if I manually create the resource in Opensearch dashboard it tries to apply some changes:

Terraform will perform the following actions:

  # opensearch_ism_policy.retention-policy will be updated in-place
  ~ resource "opensearch_ism_policy" "retention-policy" {
      ~ body         = jsonencode(
          ~ {
              ~ policy = {
                  + error_notification = null
                  + policy_id          = "retention-policy"
                  + schema_version     = 17
                    # (4 unchanged attributes hidden)
                }
            }
        )
        id           = "retention-policy"
        # (3 unchanged attributes hidden)
    }

The code completes successfully and the modifications are done.
However, if I run the same code again it tries to apply the same changes again and I don't understand why.
The expected behaviour is once the changes are applied to not re-apply the same thing again.
I compared line by line with what I have in the JSON Policy in Opensearch dashboard and what I have in the policy_document.json file. They are the same.

{
    "policy": {
        "policy_id": "retention-policy",
        "description": "Demonstrate a retention-policy.",
        "schema_version": 17,
        "error_notification": null,
        "default_state": "hot_state",
        "states": [
            {
                "name": "hot_state",
                "actions": [],
                "transitions": [
                    {
                        "state_name": "warm_state",
                        "conditions": {
                            "min_index_age": "2d"
                        }
                    },
                    {
                        "state_name": "warm_state",
                        "conditions": {
                            "min_size": "30gb"
                        }
                    }
                ]
            },
            {
                "name": "warm_state",
                "actions": [
                    {
                        "timeout": "24h",
                        "retry": {
                            "count": 5,
                            "backoff": "exponential",
                            "delay": "1h"
                        },
                        "warm_migration": {}
                    }
                ],
                "transitions": [
                    {
                        "state_name": "cold_state",
                        "conditions": {
                            "min_index_age": "28d"
                        }
                    }
                ]
            },
            {
                "name": "cold_state",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "cold_migration": {
                            "start_time": null,
                            "end_time": null,
                            "timestamp_field": "@timestamp",
                            "ignore": "none"
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "30d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "retry": {
                            "count": 3,
                            "backoff": "exponential",
                            "delay": "1m"
                        },
                        "cold_delete": {}
                    }
                ],
                "transitions": []
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "eks_*",
                    "logstash-*",
                    "cwl-*"
                ],
                "priority": 1
            }
        ]
    }
}

Also there is a configuration in the json policy file for all 3 attributes that Terraform wants to update.

Any information will be much appreciated, thank you :)

[prudhvigodithi]

[Untriage]
Adding @rblcoder, can you please take a look and your thoughts?
Thank you

[rblcoder]

The following example works with the attached json. I am able to create the policy and re-apply does not show changes.

OpenSearch v2.11.0
Terraform v1.6.0
OpenSearch Terraform provider v2.0.0

resource "opensearch_ism_policy" "rollover_index_policy" {
  policy_id = "rollover_policy"
  body      = jsonencode(jsondecode(file("./policy.json")))

}

policy.json

The equivalent code with inline json

resource "opensearch_ism_policy" "rollover_index_policy" {
  policy_id = "rollover_policy"
  body = jsonencode(
  {
  "policy": {
    "description": "Example rollover policy.",
    "default_state": "rollover",
    "states": [
      {
        "name": "rollover",
        "actions": [
          {
            "rollover": {
              "min_doc_count": 1
            }
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": {
      "index_patterns": ["log*"],
      "priority": 100
    }
  }
}

  )
}

json objects related issue hashicorp/terraform-provider-awscc#509

[dirnikk]

Hey @rblcoder,
Thanks for checking!
What could be the reason of not creating the resource if does not exists?

[rblcoder]

@dirnikk I am able to create the policy through this code without manual creating the policy through OpenSearch UI or API.

[dirnikk]

@rblcoder thanks again,
It's so confusing why it interact like that.
I can see there are some yellow line in my ide not sure why:

Could you please paste the exact code that you are using to create the resource?

provider config:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.12.0"
    }
    opensearch = {
      source  = "opensearch-project/opensearch"
      version = "2.0.0"
    }
  }
}

provider "opensearch" {
  url         = "https://${var.cluster_name}.${data.aws_route53_zone.opensearch.name}"
  aws_region  = data.aws_region.current.name
  healthcheck = false
}

Thanks!

[rblcoder]

@dirnikk Here is the exact code

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.0.0"
    }
  }
}

provider "opensearch" {
  url = "http://127.0.0.1:9200"
  username = "admin"
  password = "admin"
}

resource "opensearch_ism_policy" "rollover_index_policy" {
  policy_id = "rollover_policy"
  body      = jsonencode(jsondecode(file("./policy.json")))

}

[prudhvigodithi]

Thanks @rblcoder, Closing this issue @dirnikk please re-open if required.

[adelsz]

Running into both of these issues with latest provider version 2.3.0. Can this be reopened?
@prudhvigodithi