Light Engine: LogMerger State Machine: S3ObjectMigration Lambda fails due to lack of KMS permissions on analytics S3 bucket

[cspring86]

Describe the bug

I've deployed the Light Engine version of the solution exclusively.

I've configured log merge after 7 days and log archive after 365 days.

I've been using Athena to query the log data perfectly fine since the solution has been deployed. However, after a couple of months, I tried querying log data older than 7 days and it returned nothing. So I checked the analytics S3 bucket and confirmed that no data existed older than 7 days.

After investigating, I've found the following error in the S3ObjectMigration Lambda logs:

An error occurred (AccessDenied) when calling the CreateMultipartUpload operation: User: arn:aws:sts::[account_id]:assumed-role/clwos-CentralizedLogging-S3ObjectMigrationRole-lmkktgQaNF2r/clwos-CentralizedLogging-S3ObjectMigration-aFZF08trX9Eg is not authorized to perform: kms:GenerateDataKey on this resource because no identity-based policy allows the kms:GenerateDataKey action

The analytics S3 bucket has a customer managed KMS key as the default key, so it can't upload the newly merged object back to the analytics bucket.

Expected Behavior

The solution should support customer managed KMS keys on the analytics S3 bucket such that it accepts the KMS key as a parameter and adds the relevant permissions to the S3ObjectMigration Lambda role.

The solution should also properly fail when the error occurs. Currently, everything looks like it's working perfectly. The state machine all succeed, the Lambdas all succeed, so there's no clear indication something has gone wrong.

Current Behavior

The solution silently fails when uploading merged logs back to the analytics S3 bucket, if that bucket has a customer managed KMS key as its default encryption key.

The solution also reports success when the error occurs in the State Machines and Lambda functions.

Reproduction Steps

• Deploy dependent resources e.g. SNS topics for alarms, S3 bucket for analytics engine and customer managed KMS key for the S3 bucket.
• Deploy the solution.
• Configure a pipeline using Light Engine targeting the S3 bucket created in the first step.
  • To speed things up, set the "Logs merge (days)" setting to 1 day.
• Wait over a day and notice that the logs older than a day start disappearing from the analytics S3 bucket.

Possible Solution

Allow end user to provide a customer managed KMS key to the solution for the analytics S3 bucket so it can update the S3ObjectMigration Lambda with the relevant permissions.

Even better, have the solution check the S3 bucket for default encryption with KMS, and if set, fetch the KMS key information and configure itself accordingly.

Additional Information/Context

No response

Solution Version

2.1.1

AWS Region. e.g., us-east-1

us-east-1

Other information

No response

[AlbertMingXu]

I want to confirm, the centralized bucket being used for the logs is a self-built bucket encrypted with CMK, right?

You can search for "KMSPublicAccessPolicy" in IAM -> Policy and add the corresponding CMK permissions.

[AlbertMingXu]

I will reproduce this issue and fix it in version 2.2.0.

[cspring86]

I want to confirm, the centralized bucket being used for the logs is a self-built bucket encrypted with CMK, right?

You can search for "KMSPublicAccessPolicy" in IAM -> Policy and add the corresponding CMK permissions.

Yes, I think I'm going to manually add the permissions for now.

I will reproduce this issue and fix it in version 2.2.0.

Amazing, thanks!