Feature: Suppress cloudwatch alerts when the state changes from INSUFFICIENT_DATA to OK

[veldotshine]

Is your feature request related to a problem? Please describe.
Cloudwatch alarms are created part of fullset stack. The alarms are configured to trigger for ALARM and OK State. The alarm state is set to INSUFFICIENT_DATA when an cloudwatch alarm is created, and then it moves to OK state once it collects the data point in the evaluation range. During this period, since the state changes from INSUFFICIENT_DATA to OK the alarms triggers SNS notification. This creates a bunch of alerts whenever a new full set is created.

Describe the solution you'd like
Inorder to avoid the alerts being triggered during a new full set creation.,

• A lambda function can be added to alarms and it can trigger the SNS alert for OK state based on the old_state of the alarm.
  • i.e Ignore the alert, if the old_state value is INSUFFICIENT_DATA.
  • Trigger the SNS notification887, if the old_state value is ALARM.

Additional context
This is only an overhead when SNS notifications are integrated with Opsgenie/any other alert system.