Metricbeat is unable to connect to AWS OpenSearch service. ERROR 401 Unauthorized

[nagachinni]

Describe the bug

Metricbeat running on EC2 is failing with error: unable to connect to AWS OpenSearch service. ERROR 401 Unauthorized
My openSearch domain enabled fine-grain access control(FGAC) with IAM role as master user.
I have the access policy attached to EC2 IAM role already and mapped the opensearch role "all_access" so it should reach the domain endpoint. However, this is not working.

Related component

Other

To Reproduce

1. Create AWS OpenSearch domain and enable FGAC.
2. Create an EC2 instance and attach the IAM policy to allow access to opensearch domain created above.

"Effect": "Allow",
"Action": [
    "es:*"
 ],
 "Resource": [
     "<domainarn>"
]

3. Update opensearch domain access policy as below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": [
        "es:*"
      ],
      "Resource": "<domain_arn>"
    }
  ]
}

4. Install metricbeat-oss7.12.1 on EC2 instance and configure to push metrics to opensearch domain endpoint.

You'll receive error as stated above when running metricbeat.

Expected behavior

Metricbeat should be able to push metrics to opensearch endpoint.

Additional Details

Host/Environment (please complete the following information):

• OS: Windows

[wbeckler]

Metricbeat is maintained by Elastic. It looks like someone raised a similar issue for filebeat here: elastic/beats#31613. Maybe you could raise a similar issue?

[peternied]

[Triage - attendees 1 2 3 4 5]
@nagachinni Thanks for creating this issue; however, it isn't being accepted due to being out of scope for this project, please reach out through support channels with your hosting provider to troubleshoot.