Application Log: Permission error for reading ism policy after created a new Application Log Pipeline (S3).

[erlonpinheiro]

Describe the bug

When creating an Application Log Pipeline with Amazon S3 as Log Source the processor lambda is executed once, I think is to create an ISM (Index State Management) Policy. But it is causing a permission error to get information from the ISM. I believe this could be causing issues on creating this policy.

Expected Behavior

Get the information from ISM and finish without errors creating the ISM policy.

Current Behavior

After have created the Application Pipeline, I've checked the Processor Lambda Log and detected the following errors:

[INFO] 2024-06-06T18:19:33.088Z 8d4f07ae-7012-5350-9eb9-59758fe386e8 GET https://vpc-XXX-tc2rqyc2gz5nnjbtqly6c5rwgu.us-east-1.es.amazonaws.com/_plugins/_ism/policies/xx_xxxxx_request_log-ism-policy

[INFO] 2024-06-06T18:19:33.124Z 8d4f07ae-7012-5350-9eb9-59758fe386e8 --> get_ism_policy response code 403

[INFO] 2024-06-06T18:19:33.124Z 8d4f07ae-7012-5350-9eb9-59758fe386e8 --> get_ism_policy response content b'{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:admin/opendistro/ism/policy/get] and User [name=arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7, backend_roles=[arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:admin/opendistro/ism/policy/get] and User [name=arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7, backend_roles=[arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7], requestedTenant=null]"},"status":403}'

[INFO] 2024-06-06T18:19:33.124Z 8d4f07ae-7012-5350-9eb9-59758fe386e8 the last response code is 403, the last response content is b'{"error":{"root_cause":[{"type":"security_exception","reason":"no permissions for [cluster:admin/opendistro/ism/policy/get] and User [name=arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7, backend_roles=[arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7], requestedTenant=null]"}],"type":"security_exception","reason":"no permissions for [cluster:admin/opendistro/ism/policy/get] and User [name=arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7, backend_roles=[arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-daf6fa9c-3311-4bde-8065-0a14785bcdc7], requestedTenant=null]"},"status":403}'

LAMBDA_WARNING: Unhandled exception. The most likely cause is an issue in the function code. However, in rare cases, a Lambda runtime update can cause unexpected function behavior. For functions using managed runtimes, runtime updates can be triggered by a function change, or can be applied automatically. To determine if the runtime has been updated, check the runtime version in the INIT_START log entry. If this error correlates with a change in the runtime version, you may be able to mitigate this error by temporarily rolling back to the previous runtime version. For more information, see https://docs.aws.amazon.com/lambda/latest/dg/runtimes-update.html

[ERROR] APIException: [UNKNOWN_ERROR] error in calling get_ism_policy
 Traceback (most recent call last):
   File "/var/task/lambda_function.py", line 44, in lambda_handler
     raise e
   File "/var/task/lambda_function.py", line 34, in lambda_handler
     idx_svc.init_idx_env()
   File "/var/task/idx/idx_svc.py", line 117, in init_idx_env
     self._init_ism()
   File "/var/task/idx/idx_svc.py", line 171, in _init_ism
     self.run_func_with_retry(   File "/var/task/idx/idx_svc.py", line 88, in run_func_with_retry
     response = func(**kwargs)
   File "/var/task/idx/opensearch_client.py", line 130, in create_ism_policy
     raise APIException(ErrorCode.UNKNOWN_ERROR, "error in calling get_ism_policy")

And the ISM policy is not created.

Reproduction Steps

Create an Application Log Pipeline with Amazon S3 as a Log Source and right after its state is Active, check the logs at the Logs Tab. The error is supposed to be there at the first (and only at this point) Log Stream entry.

Possible Solution

Add necessary permission to role arn:aws:iam::XXXXXXXXXXXX:role/CL-log-processor-*

Additional Information/Context

On my tests I am creating an Application Log Pipeline which I have created and deleted before, also using the same index I've used before at the deleted pipeline. Then I receive an info informing that and I just hit Continue.

Solution Version

v2.1.2

AWS Region. e.g., us-east-1

us-east-1

Other information

No response

[erlonpinheiro]

New information. I realized that after 5 minutes the processor lambda function has started again. This time the ISM policy was created.

Is this an expected behaviour? If yes, the Application Log Pipeline should stay in "pending" state until the ISM is fully created, shouldn't?