[BUG] Cannot authenticate against AWS OpenSearch Service domain using Basic Authentication

[stagha]

What is the bug?

Provider is unable to authenticate against OpenSearch endpoint.

How can one reproduce the bug?

I am using an AWS OpenSearch Service endpoint. Fine-grained access control is enabled using an internal database. I have confirmed that I can make requests to the endpoint directly using Postman using Basic Auth with the same credentials.

I have set the following environment as follows (I am running on Windows.) For some reason, setting the corresponding parameters directly on the provider definition in Terraform has no effect.

> $Env:OPENSEARCH_HEALTH = "false"
> $Env:OPENSEARCH_URL = "https://vpc-***OBFUSCATED***.us-east-1.es.amazonaws.com"
> $Env:OPENSEARCH_USERNAME = "admin"
> $Env:OPENSEARCH_PASSWORD = "***OBFUSCATED***"

Here is my provider:

provider "opensearch" {
  alias = "opensearch"

  sign_aws_requests = false
}

Here are my resource configurations:

resource "opensearch_role" "this" {
  provider = opensearch

  role_name   = local.role_name
  description = "OSS role for ${var.name}"

  cluster_permissions = ["*"]

  index_permissions {
    index_patterns  = ["${var.prefix}-*"]
    allowed_actions = ["write"]
  }

  tenant_permissions {
    tenant_patterns = ["${var.prefix}-*"]
    allowed_actions = ["write"]
  }
}

resource "opensearch_roles_mapping" "this" {
  provider = opensearch

  role_name     = local.role_name
  description   = "OSS role to IAM role mapping for ${var.name}"
  backend_roles = [var.iam_role_arn]
}

Other details that may or may not be relevant:

• I have a .aws/credentials file in my user's home directory, but I would expect this provider to ignore it based on the value of sign_aws_requests above.
• The same state also uses the aws provider to create other resources. But that shouldn't have any bearing here.

This is the output of terraform apply:

module.opensearch_role.opensearch_role.this: Creating...
module.opensearch_role.opensearch_roles_mapping.this: Creating...

Error: HTTP 403 Forbidden: Permission denied. Please ensure that the correct credentials are being used to access the cluster.

  on modules\opensearch_role\main.tf line 5, in resource "opensearch_role" "this":
   5: resource "opensearch_role" "this" {



Error: HTTP 403 Forbidden: Permission denied. Please ensure that the correct credentials are being used to access the cluster.

  on modules\opensearch_role\main.tf line 24, in resource "opensearch_roles_mapping" "this":
  24: resource "opensearch_roles_mapping" "this" {

What is the expected behavior?

Terraform should successfully create my OpenSearch resources.

What is your host/environment?

• OS: Windows 11
• Provider: 2.0.0
• Terraform: 0.14.4
• OpenSearch Domain: OpenSearch 2.7

[jmurillo9]

Yeah, I was getting a similar experience...

@stagha - take a look at this GitHub issue. Although it's specific to Elasticsearch, it seems to have solved my issue.

• Error: HTTP 403 Forbidden: Permission denied. Please ensure that the correct credentials are being used to access the cluster phillbaker/terraform-provider-elasticsearch#318

Testing locally, I was able to create a test user (a very basic task) within my OpenSearch domain (which has SAML enabled).

provider "opensearch" {
  url = "<INSERT_CUSTOM_DOMAIN_URL>"
  sign_aws_requests = false
  username = "REDACTED"
  password = "REDACTED"
}

I'll need to play around with this a little more. Ideally, I can just use an IAM role and/or my current AWS context. A repository secret to be passed in could be acceptable for the time being. So far, I haven't been able to get aws_assume_role_arn or the current AWS context to work.

[peterzhuamazon]

[Triage] @prudhvigodithi.

[prudhvigodithi]

Hey @jmurillo9 thanks, are you open to contribute with a fix for this bug?
Thank you

[prudhvigodithi]

Hey @stagha for the AWS OpenSearch did you try with access policy Only use fine-grained access control ? then it directly works with username and password as per https://linuxhint.com/access-aws-opensearch/.

Adding @bbarani @rishabh6788

[stagha]

@prudhvigodithi yes that option is checked. Note again that I was able to make requests from the cluster endpoint using curl and the master username/password.

[Sovietaced]

I encountered 403s when I had issues with request signing. If the credentials are wrong the provider just blows up. So maybe there is an issue when you have username/password sourced from environment variables and the request signing sourced from the provider config?

Would be worth just trying to configure everything in the provider inline given that you validated that curl works.

[stagha]

That is part of the problem. When I try to configure everything on the provider, I get errors as if the settings are completely missing. E.g. I get "URL missing" errors even if I have url set to a hard-coded URL on the provider. Going the environment variable route was the only way it would take.

Perhaps this is the root of the problem? Perhaps sign_aws_requests is not properly being set to false even though I explicitly set it on the provider? But if so, what is the solution?

[rblcoder]

The following configuration is working for AWS OpenSearch v2.9
Terraform v1.6.0

https://github.com/rblcoder/terraform-opensearch-samples/blob/main/aws_opensearch_basic_auth_manual/main.tf

provider "opensearch" {
  url = "url"
  healthcheck       = "false"
  aws_region        = "region"
  username          = "username"
  password          = "password"
  sign_aws_requests = "false" 
  version_ping_timeout = "10"

}

[joewragg]

I am getting 403s with AWS auth and basic auth.

I have mapped my basic auth user and IAM role to all_access role.

I can see it's authorisation not authentication because if I change the password I get 401 instead

[rblcoder]

@joewragg Please share your terraform version, terraform provider version along with complete terraform code including provider configuration.

[joewragg]

@joewragg Please share your terraform version, terraform provider version along with complete terraform code including provider configuration.

terraform version: 1.5.6

code (redacted)

...
    opensearch = {
      source  = "opensearch-project/opensearch"
      version = "2.2.1"
    }
...

provider "opensearch" {
  url = "https://${data.aws_opensearch_domain.opensearch.endpoint}"
  username = "myuser"
  password = "mypassword"
  sign_aws_requests = false
  aws_region        = "eu-west-1"
  sniff             = false
}

I have also tested locally with curl basic auth creds and get 200 OK

[rblcoder]

@joewragg
The following terraform code works for me.
Terraform v1.7.5

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.2.1"
    }
  }
}

provider "opensearch" {
  url = "url"
  healthcheck       = "false"
  aws_region        = "region"
  username          = "username"
  password          = "password"
  sign_aws_requests = "false" 
  version_ping_timeout = "10"

}

resource "opensearch_index" "index" {
  name = "sample"
  number_of_replicas = "1"
  number_of_shards = "1"
 
}

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # opensearch_index.index will be created
  + resource "opensearch_index" "index" {
      + force_destroy      = false
      + id                 = (known after apply)
      + name               = "sample"
      + number_of_replicas = "1"
      + number_of_shards   = "1"
      + rollover_alias     = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

opensearch_index.index: Creating...
opensearch_index.index: Creation complete after 1s [id=sample]

[joewragg]

@joewragg The following terraform code works for me. Terraform v1.7.5

terraform {
  required_providers {
    opensearch = {
      source = "opensearch-project/opensearch"
      version = "2.2.1"
    }
  }
}

provider "opensearch" {
  url = "url"
  healthcheck       = "false"
  aws_region        = "region"
  username          = "username"
  password          = "password"
  sign_aws_requests = "false" 
  version_ping_timeout = "10"

}

resource "opensearch_index" "index" {
  name = "sample"
  number_of_replicas = "1"
  number_of_shards = "1"
 
}

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # opensearch_index.index will be created
  + resource "opensearch_index" "index" {
      + force_destroy      = false
      + id                 = (known after apply)
      + name               = "sample"
      + number_of_replicas = "1"
      + number_of_shards   = "1"
      + rollover_alias     = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

opensearch_index.index: Creating...
opensearch_index.index: Creation complete after 1s [id=sample]

I have tested these settings and versions and get the same error I wonder if it is related to these settings in AWS? :