From 9c106c74e33b5594e2990961a56c7399e3c9a95f Mon Sep 17 00:00:00 2001 From: Shawn Huckabay Date: Wed, 9 Oct 2024 12:11:27 -0500 Subject: [PATCH] POL-1355 New Policy: AWS Rightsize ElastiCache (#2702) * update * fix * update * fix * update * fix * comimt * update * update * update * update * update * update --- .spellignore | 4 + cost/aws/rightsize_elasticache/CHANGELOG.md | 5 + cost/aws/rightsize_elasticache/README.md | 96 ++ .../aws_rightsize_elasticache.pt | 1469 +++++++++++++++++ .../aws_rightsize_elasticache_meta_parent.pt | 1399 ++++++++++++++++ .../master_policy_permissions_list.json | 53 + .../master_policy_permissions_list.yaml | 31 + .../meta_parent_policy_compiler.rb | 1 + .../validated_policy_templates.yaml | 1 + 9 files changed, 3059 insertions(+) create mode 100644 cost/aws/rightsize_elasticache/CHANGELOG.md create mode 100644 cost/aws/rightsize_elasticache/README.md create mode 100644 cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt create mode 100644 cost/aws/rightsize_elasticache/aws_rightsize_elasticache_meta_parent.pt diff --git a/.spellignore b/.spellignore index 1c67184d4e..0d1e900e33 100644 --- a/.spellignore +++ b/.spellignore @@ -585,3 +585,7 @@ ByteCount PacketCount balancers backfill +ElastiCache +elasticache +oversized +freeable diff --git a/cost/aws/rightsize_elasticache/CHANGELOG.md b/cost/aws/rightsize_elasticache/CHANGELOG.md new file mode 100644 index 0000000000..a1ed544621 --- /dev/null +++ b/cost/aws/rightsize_elasticache/CHANGELOG.md @@ -0,0 +1,5 @@ +# Changelog + +## v0.1.0 + +- Initial release diff --git a/cost/aws/rightsize_elasticache/README.md b/cost/aws/rightsize_elasticache/README.md new file mode 100644 index 0000000000..8db9c2f1f7 --- /dev/null +++ b/cost/aws/rightsize_elasticache/README.md @@ -0,0 +1,96 @@ +# AWS Rightsize ElastiCache + +## What It Does + +This policy template reports any underutilized AWS ElastiCache resources based on CPU and memory usage. Optionally, this report can be emailed and oversized resources can be downsized. + +### Policy Savings Details + +The policy includes the estimated monthly savings. The estimated monthly savings is recognized if the resource is downsized. + +- The `Estimated Monthly Savings` is calculated by multiplying the amortized cost of the resource for 1 day, as found within Flexera CCO, by 30.44, which is the average number of days in a month. +- Savings is estimated at 50% of the current cost of the resource. +- Since the costs of individual resources are obtained from Flexera CCO, they will take into account any Flexera adjustment rules or cloud provider discounts present in the Flexera platform. +- If the resource cannot be found in Flexera CCO, the `Estimated Monthly Savings` is 0. +- The incident message detail includes the sum of each resource `Estimated Monthly Savings` as `Potential Monthly Savings`. +- Both `Estimated Monthly Savings` and `Potential Monthly Savings` will be reported in the currency of the Flexera organization the policy is applied in. + +## Input Parameters + +- *Email Addresses* - Email addresses of the recipients you wish to notify when new incidents are created. +- *Account Number* - The Account number for use with the AWS STS Cross Account Role. Leave blank when using AWS IAM Access key and secret. It only needs to be passed when the desired AWS account is different than the one associated with the Flexera One credential. [More information is available in our documentation.](https://docs.flexera.com/flexera/EN/Automation/ProviderCredentials.htm#automationadmin_1982464505_1123608) +- *Allow/Deny Regions* - Whether to treat Allow/Deny Regions List parameter as allow or deny list. Has no effect if Allow/Deny Regions List is left empty. +- *Allow/Deny Regions List* - A list of regions to allow or deny for an AWS account. Please enter the regions code if SCP is enabled. See [Available Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions) in AWS; otherwise, the policy may fail on regions that are disabled via SCP. Leave blank to consider all the regions. +- *Exclusion Tags* - The policy will filter resources containing the specified tags from the results. The following formats are supported: + - `Key` - Filter all resources with the specified tag key. + - `Key==Value` - Filter all resources with the specified tag key:value pair. + - `Key!=Value` - Filter all resources missing the specified tag key:value pair. This will also filter all resources missing the specified tag key. + - `Key=~/Regex/` - Filter all resources where the value for the specified key matches the specified regex string. + - `Key!~/Regex/` - Filter all resources where the value for the specified key does not match the specified regex string. This will also filter all resources missing the specified tag key. +- *Exclusion Tags: Any / All* - Whether to filter instances containing any of the specified tags or only those that contain all of them. Only applicable if more than one value is entered in the `Exclusion Tags` field. +- *Statistic Lookback Period* - How many days back to look at CPU and memory usage data for clusters. This value cannot be set higher than 90 because AWS does not retain metrics for longer than 90 days. +- *Threshold Statistic* - Statistic to use when determining if a cluster is underutilized. +- *CPU Threshold (%)* - The CPU threshold at which to consider a cluster to be 'underutilized' and therefore be flagged for downsizing. Set to -1 to ignore CPU utilization +- *Freeable Memory Threshold (%)* - The freeable memory threshold at which to consider a cluster to be 'underutilized' and therefore be flagged for downsizing. Set to -1 to ignore memory utilization +- *Automatic Actions* - When this value is set, this policy will automatically take the selected action(s). + +Please note that the "Automatic Actions" parameter contains a list of action(s) that can be performed on the resources. When it is selected, the policy will automatically execute the corresponding action on the data that failed the checks, post incident generation. Please leave this parameter blank for *manual* action. +For example if a user selects the "Downsize ElastiCache Nodes" action while applying the policy, the nodes of all underutilized ElastiCache clusters will be downsized. + +## Policy Actions + +- Sends an email notification +- Downsize nodes of underutilized ElastiCache clusters after approval + +## Prerequisites + +This Policy Template uses [Credentials](https://docs.flexera.com/flexera/EN/Automation/ManagingCredentialsExternal.htm) for authenticating to datasources -- in order to apply this policy you must have a Credential registered in the system that is compatible with this policy. If there are no Credentials listed when you apply the policy, please contact your Flexera Org Admin and ask them to register a Credential that is compatible with this policy. The information below should be consulted when creating the credential(s). + +### Credential configuration + +For administrators [creating and managing credentials](https://docs.flexera.com/flexera/EN/Automation/ManagingCredentialsExternal.htm) to use with this policy, the following information is needed: + +- [**AWS Credential**](https://docs.flexera.com/flexera/EN/Automation/ProviderCredentials.htm#automationadmin_1982464505_1121575) (*provider=aws*) which has the following permissions: + - `sts:GetCallerIdentity` + - `cloudwatch:GetMetricData` + - `ec2:DescribeRegions` + - `elasticache:DescribeCacheClusters` + - `elasticache:ListTagsForResource` + - `elasticache:ModifyCacheCluster`* + + \* Only required for taking action (terminating or downsizing); the policy will still function in a read-only capacity without these permissions. + + Example IAM Permission Policy: + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "sts:GetCallerIdentity", + "cloudwatch:GetMetricData", + "ec2:DescribeRegions", + "elasticache:DescribeCacheClusters", + "elasticache:ListTagsForResource", + "elasticache:ModifyCacheCluster" + ], + "Resource": "*" + } + ] + } + ``` + +- [**Flexera Credential**](https://docs.flexera.com/flexera/EN/Automation/ProviderCredentials.htm) (*provider=flexera*) which has the following roles: + - `billing_center_viewer` + +The [Provider-Specific Credentials](https://docs.flexera.com/flexera/EN/Automation/ProviderCredentials.htm) page in the docs has detailed instructions for setting up Credentials for the most common providers. + +## Supported Clouds + +- AWS + +## Cost + +This policy template does not incur any cloud costs. diff --git a/cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt b/cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt new file mode 100644 index 0000000000..3ee73b2267 --- /dev/null +++ b/cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt @@ -0,0 +1,1469 @@ +name "AWS Rightsize ElastiCache" +rs_pt_ver 20180301 +type "policy" +short_description "Reports any underutilized AWS ElastiCache resources and resizes them after approval. See the [README](https://github.com/flexera-public/policy_templates/tree/master/cost/aws/rightsize_elasticache/) and [docs.flexera.com/flexera/EN/Automation](https://docs.flexera.com/flexera/EN/Automation/AutomationGS.htm) to learn more." +long_description "" +category "Cost" +severity "low" +default_frequency "weekly" +info( + version: "0.1.0", + provider: "AWS", + service: "Database", + policy_set: "Rightsize Database Instances", + recommendation_type: "Usage Reduction" +) + +############################################################################### +# Parameters +############################################################################### + +parameter "param_email" do + type "list" + category "Policy Settings" + label "Email Addresses" + description "Email addresses of the recipients you wish to notify." + default [] +end + +parameter "param_aws_account_number" do + type "string" + category "Policy Settings" + label "Account Number" + description "Leave blank; this is for automated use with Meta Policies. See README for more details." + default "" +end + +parameter "param_min_savings" do + type "number" + category "Policy Settings" + label "Minimum Savings Threshold" + description "Minimum potential savings required to generate a recommendation" + min_value 0 + default 0 +end + +parameter "param_exclusion_tags" do + type "list" + category "Filters" + label "Exclusion Tags" + description "Cloud native tags to ignore clusters that you don't want to produce recommendations for. Enter the Key name to filter clusters with a specific Key, regardless of Value, and enter Key==Value to filter clusters with a specific Key:Value pair. Other operators and regex are supported; please see the README for more details." + default [] +end + +parameter "param_exclusion_tags_boolean" do + type "string" + category "Filters" + label "Exclusion Tags: Any / All" + description "Whether to filter clusters containing any of the specified tags or only those that contain all of them. Only applicable if more than one value is entered in the 'Exclusion Tags' field." + allowed_values "Any", "All" + default "Any" +end + +parameter "param_regions_allow_or_deny" do + type "string" + category "Filters" + label "Allow/Deny Regions" + description "Allow or Deny entered regions. See the README for more details" + allowed_values "Allow", "Deny" + default "Allow" +end + +parameter "param_regions_list" do + type "list" + category "Filters" + label "Allow/Deny Regions List" + description "A list of allowed or denied regions. See the README for more details" + allowed_pattern /^([a-zA-Z-_]+-[a-zA-Z0-9-_]+-[0-9-_]+,*|)+$/ + default [] +end + +parameter "param_stats_lookback" do + type "number" + category "Statistics" + label "Statistic Lookback Period" + description "How many days back to look at CPU and memory usage data for clusters. This value cannot be set higher than 90 because AWS does not retain metrics for longer than 90 days." + min_value 1 + max_value 90 + default 30 +end + +parameter "param_stats_threshold" do + type "string" + category "Statistics" + label "Threshold Statistic" + description "Statistic to use when determining if a cluster is underutilized" + allowed_values "Average", "Maximum", "p99", "p95", "p90" + default "Average" +end + +parameter "param_stats_threshold_cpu_value" do + type "number" + category "Statistics" + label "CPU Threshold (%)" + description "The CPU threshold at which to consider a cluster to be 'underutilized' and therefore be flagged for downsizing. Set to -1 to ignore CPU utilization" + min_value -1 + max_value 100 + default 40 +end + +parameter "param_stats_threshold_mem_value" do + type "number" + category "Statistics" + label "Freeable Memory Threshold (%)" + description "The freeable memory threshold at which to consider a cluster to be 'underutilized' and therefore be flagged for downsizing. Set to -1 to ignore memory utilization" + min_value -1 + max_value 100 + default 60 +end + +parameter "param_automatic_action" do + type "list" + category "Actions" + label "Automatic Actions" + description "When this value is set, this policy will automatically take the selected action(s)" + allowed_values ["Downsize ElastiCache Nodes"] + default [] +end + +############################################################################### +# Authentication +############################################################################### + +credentials "auth_aws" do + schemes "aws", "aws_sts" + label "AWS" + description "Select the AWS Credential from the list" + tags "provider=aws" + aws_account_number $param_aws_account_number +end + +credentials "auth_flexera" do + schemes "oauth2" + label "Flexera" + description "Select Flexera One OAuth2 credentials" + tags "provider=flexera" +end + +############################################################################### +# Pagination +############################################################################### + +pagination "pagination_aws_getmetricdata" do + get_page_marker do + body_path "NextToken" + end + set_page_marker do + body_field "NextToken" + end +end + +############################################################################### +# Datasources & Scripts +############################################################################### + +# Get applied policy metadata for use later +datasource "ds_applied_policy" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/applied_policies/", policy_id]) + header "Api-Version", "1.0" + end +end + +# Get region-specific Flexera API endpoints +datasource "ds_flexera_api_hosts" do + run_script $js_flexera_api_hosts, rs_optima_host +end + +script "js_flexera_api_hosts", type: "javascript" do + parameters "rs_optima_host" + result "result" + code <<-EOS + host_table = { + "api.optima.flexeraeng.com": { + flexera: "api.flexera.com", + fsm: "api.fsm.flexeraeng.com" + }, + "api.optima-eu.flexeraeng.com": { + flexera: "api.flexera.eu", + fsm: "api.fsm-eu.flexeraeng.com" + }, + "api.optima-apac.flexeraeng.com": { + flexera: "api.flexera.au", + fsm: "api.fsm-apac.flexeraeng.com" + } + } + + result = host_table[rs_optima_host] +EOS +end + +# Get AWS account info +datasource "ds_cloud_vendor_accounts" do + request do + auth $auth_flexera + host val($ds_flexera_api_hosts, 'flexera') + path join(["/finops-analytics/v1/orgs/", rs_org_id, "/cloud-vendor-accounts"]) + header "Api-Version", "1.0" + end + result do + encoding "json" + collect jmes_path(response, "values[*]") do + field "id", jmes_path(col_item, "aws.accountId") + field "name", jmes_path(col_item, "name") + field "tags", jmes_path(col_item, "tags") + end + end +end + +datasource "ds_get_caller_identity" do + request do + auth $auth_aws + verb "GET" + host "sts.amazonaws.com" + path "/" + query "Action", "GetCallerIdentity" + query "Version", "2011-06-15" + header "User-Agent", "RS Policies" + end + result do + encoding "xml" + collect xpath(response, "//GetCallerIdentityResponse/GetCallerIdentityResult") do + field "account", xpath(col_item, "Account") + end + end +end + +datasource "ds_aws_account" do + run_script $js_aws_account, $ds_cloud_vendor_accounts, $ds_get_caller_identity +end + +script "js_aws_account", type:"javascript" do + parameters "ds_cloud_vendor_accounts", "ds_get_caller_identity" + result "result" + code <<-EOS + result = _.find(ds_cloud_vendor_accounts, function(account) { + return account['id'] == ds_get_caller_identity[0]['account'] + }) + + // This is in case the API does not return the relevant account info + if (result == undefined) { + result = { + id: ds_get_caller_identity[0]['account'], + name: "", + tags: {} + } + } +EOS +end + +datasource "ds_billing_centers" do + request do + auth $auth_flexera + host rs_optima_host + path join(["/analytics/orgs/", rs_org_id, "/billing_centers"]) + query "view", "allocation_table" + header "Api-Version", "1.0" + header "User-Agent", "RS Policies" + ignore_status [403] + end + result do + encoding "json" + collect jmes_path(response, "[*]") do + field "href", jmes_path(col_item, "href") + field "id", jmes_path(col_item, "id") + field "name", jmes_path(col_item, "name") + field "parent_id", jmes_path(col_item, "parent_id") + end + end +end + +# Gather top level billing center IDs for when we pull cost data +datasource "ds_top_level_bcs" do + run_script $js_top_level_bcs, $ds_billing_centers +end + +script "js_top_level_bcs", type: "javascript" do + parameters "ds_billing_centers" + result "result" + code <<-EOS + filtered_bcs = _.filter(ds_billing_centers, function(bc) { + return bc['parent_id'] == null || bc['parent_id'] == undefined + }) + + result = _.compact(_.pluck(filtered_bcs, 'id')) +EOS +end + +datasource "ds_currency_reference" do + request do + host "raw.githubusercontent.com" + path "/flexera-public/policy_templates/master/data/currency/currency_reference.json" + header "User-Agent", "RS Policies" + end +end + +datasource "ds_currency_code" do + request do + auth $auth_flexera + host rs_optima_host + path join(["/bill-analysis/orgs/", rs_org_id, "/settings/currency_code"]) + header "Api-Version", "0.1" + header "User-Agent", "RS Policies" + ignore_status [403] + end + result do + encoding "json" + field "id", jmes_path(response, "id") + field "value", jmes_path(response, "value") + end +end + +datasource "ds_currency" do + run_script $js_currency, $ds_currency_reference, $ds_currency_code +end + +script "js_currency", type:"javascript" do + parameters "ds_currency_reference", "ds_currency_code" + result "result" + code <<-EOS + symbol = "$" + separator = "," + + if (ds_currency_code['value'] != undefined) { + if (ds_currency_reference[ds_currency_code['value']] != undefined) { + symbol = ds_currency_reference[ds_currency_code['value']]['symbol'] + + if (ds_currency_reference[ds_currency_code['value']]['t_separator'] != undefined) { + separator = ds_currency_reference[ds_currency_code['value']]['t_separator'] + } else { + separator = "" + } + } + } + + result = { + symbol: symbol, + separator: separator + } +EOS +end + +datasource "ds_describe_regions" do + request do + auth $auth_aws + verb "GET" + host "ec2.amazonaws.com" + path "/" + query "Action", "DescribeRegions" + query "Version", "2016-11-15" + query "Filter.1.Name", "opt-in-status" + query "Filter.1.Value.1", "opt-in-not-required" + query "Filter.1.Value.2", "opted-in" + # Header X-Meta-Flexera has no affect on datasource query, but is required for Meta Policies + # Forces `ds_is_deleted` datasource to run first during policy execution + header "Meta-Flexera", val($ds_is_deleted, "path") + end + result do + encoding "xml" + collect xpath(response, "//DescribeRegionsResponse/regionInfo/item", "array") do + field "region", xpath(col_item, "regionName") + end + end +end + +datasource "ds_regions" do + run_script $js_regions, $ds_describe_regions, $param_regions_list, $param_regions_allow_or_deny +end + +script "js_regions", type:"javascript" do + parameters "ds_describe_regions", "param_regions_list", "param_regions_allow_or_deny" + result "result" + code <<-EOS + allow_deny_test = { "Allow": true, "Deny": false } + + if (param_regions_list.length > 0) { + result = _.filter(ds_describe_regions, function(item) { + return _.contains(param_regions_list, item['region']) == allow_deny_test[param_regions_allow_or_deny] + }) + } else { + result = ds_describe_regions + } +EOS +end + +datasource "ds_elasticache_clusters" do + iterate $ds_regions + request do + auth $auth_aws + host join(["elasticache.", val(iter_item, "region"), ".amazonaws.com"]) + path "/" + query "Action", "DescribeCacheClusters" + query "Version", "2015-02-02" + header "User-Agent", "RS Policies" + header "Content-Type", "text/xml" + end + result do + encoding "xml" + collect xpath(response, "//DescribeCacheClustersResponse/DescribeCacheClustersResult/CacheClusters/CacheCluster", "array") do + field "id", xpath(col_item, "CacheClusterId") + field "status", xpath(col_item, "CacheClusterStatus") + field "nodeType", xpath(col_item, "CacheNodeType") + field "numberOfNodes", xpath(col_item, "NumCacheNodes") + field "autoMinorVersionUpgrade", xpath(col_item, "AutoMinorVersionUpgrade") + field "availabilityZone", xpath(col_item, "PreferredAvailabilityZone") + field "createTime", xpath(col_item, "CacheClusterCreateTime") + field "engine", xpath(col_item, "Engine") + field "engineVersion", xpath(col_item, "EngineVersion") + field "region", val(iter_item, "region") + end + end +end + +datasource "ds_elasticache_size_list" do + request do + verb "GET" + host "raw.githubusercontent.com" + path "/flexera-public/policy_templates/master/data/aws/elasticache_types.json" + header "User-Agent", "RS Policies" + end +end + +datasource "ds_aws_instance_size_map" do + request do + verb "GET" + host "raw.githubusercontent.com" + path "/flexera-public/policy_templates/master/data/aws/instance_types.json" + header "User-Agent", "RS Policies" + end +end + +# Filter out any clusters where a smaller size isn't available since we can't downsize them anyway +datasource "ds_elasticache_clusters_resize_filtered" do + run_script $js_elasticache_clusters_resize_filtered, $ds_elasticache_clusters, $ds_elasticache_size_list, $ds_aws_instance_size_map, $ds_aws_account +end + +script "js_elasticache_clusters_resize_filtered", type: "javascript" do + parameters "ds_elasticache_clusters", "ds_elasticache_size_list", "ds_aws_instance_size_map", "ds_aws_account" + result "result" + code <<-EOS + result = [] + + _.each(ds_elasticache_clusters, function(cluster) { + normalizedType = cluster["nodeType"].split("cache.")[1] + downType = null + + if (ds_aws_instance_size_map[normalizedType] && ds_aws_instance_size_map[normalizedType]["down"]) { + if (_.contains(ds_elasticache_size_list, ds_aws_instance_size_map[normalizedType]["down"])) { + downType = "cache." + ds_aws_instance_size_map[normalizedType]["down"] + } + } + + if (downType) { + resourceARN = [ + "arn:aws:elasticache:", cluster["region"], ":", + ds_aws_account["id"], ":cluster:", cluster["id"] + ].join("") + + result.push({ + id: cluster["id"], + statId: cluster["id"].replace(/-/g, '_'), + resourceID: cluster["id"], + resourceName: cluster["id"], + resourceARN: resourceARN, + status: cluster["status"], + numberOfNodes: cluster["numberOfNodes"], + autoMinorVersionUpgrade: cluster["autoMinorVersionUpgrade"], + availabilityZone: cluster["availabilityZone"], + createTime: cluster["createTime"], + engine: cluster["engine"], + engineVersion: cluster["engineVersion"], + region: cluster["region"], + resourceType: cluster["nodeType"], + newResourceType: downType, + vcpu: ds_aws_instance_size_map[normalizedType]["vcpu"], + memory: ds_aws_instance_size_map[normalizedType]["memory"] + }) + } + }) +EOS +end + +datasource "ds_elasticache_clusters_with_tags" do + iterate $ds_elasticache_clusters_resize_filtered + request do + auth $auth_aws + host join(["elasticache.", val(iter_item, "region"), ".amazonaws.com"]) + path "/" + query "ResourceName", val(iter_item, "resourceARN") + query "Action", "ListTagsForResource" + query "Version", "2015-02-02" + header "User-Agent", "RS Policies" + header "Content-Type", "application/json" + header "Accept", "application/json" + end + result do + encoding "json" + field "tags", jmes_path(response, "ListTagsForResourceResponse.ListTagsForResourceResult.TagList") + field "id", val(iter_item, "id") + field "statId", val(iter_item, "statId") + field "resourceID", val(iter_item, "resourceID") + field "resourceName", val(iter_item, "resourceName") + field "resourceARN", val(iter_item, "resourceARN") + field "status", val(iter_item, "status") + field "numberOfNodes", val(iter_item, "numberOfNodes") + field "autoMinorVersionUpgrade", val(iter_item, "autoMinorVersionUpgrade") + field "availabilityZone", val(iter_item, "availabilityZone") + field "createTime", val(iter_item, "createTime") + field "engine", val(iter_item, "engine") + field "engineVersion", val(iter_item, "engineVersion") + field "region", val(iter_item, "region") + field "resourceType", val(iter_item, "resourceType") + field "newResourceType", val(iter_item, "newResourceType") + field "vcpu", val(iter_item, "vcpu") + field "memory", val(iter_item, "memory") + end +end + +datasource "ds_elasticache_clusters_tag_filtered" do + run_script $js_elasticache_clusters_tag_filtered, $ds_elasticache_clusters_with_tags, $param_exclusion_tags, $param_exclusion_tags_boolean +end + +script "js_elasticache_clusters_tag_filtered", type: "javascript" do + parameters "ds_elasticache_clusters_with_tags", "param_exclusion_tags", "param_exclusion_tags_boolean" + result "result" + code <<-EOS + comparators = _.map(param_exclusion_tags, function(item) { + if (item.indexOf('==') != -1) { + return { comparison: '==', key: item.split('==')[0], value: item.split('==')[1], string: item } + } + + if (item.indexOf('!=') != -1) { + return { comparison: '!=', key: item.split('!=')[0], value: item.split('!=')[1], string: item } + } + + if (item.indexOf('=~') != -1) { + value = item.split('=~')[1] + regex = new RegExp(value.slice(1, value.length - 1)) + return { comparison: '=~', key: item.split('=~')[0], value: regex, string: item } + } + + if (item.indexOf('!~') != -1) { + value = item.split('!~')[1] + regex = new RegExp(value.slice(1, value.length - 1)) + return { comparison: '!~', key: item.split('!~')[0], value: regex, string: item } + } + + // If = is present but none of the above are, assume user error and that the user intended == + if (item.indexOf('=') != -1) { + return { comparison: '==', key: item.split('=')[0], value: item.split('=')[1], string: item } + } + + // Assume we're just testing for a key if none of the comparators are found + return { comparison: 'key', key: item, value: null, string: item } + }) + + if (param_exclusion_tags.length > 0) { + result = _.reject(ds_elasticache_clusters_with_tags, function(resource) { + resource_tags = {} + + if (typeof(resource['tags']) == 'object') { + _.each(resource['tags'], function(tag) { + resource_tags[tag['Key']] = tag['Value'] + }) + } + + // Store a list of found tags + found_tags = [] + + _.each(comparators, function(comparator) { + comparison = comparator['comparison'] + value = comparator['value'] + string = comparator['string'] + resource_tag = resource_tags[comparator['key']] + + if (comparison == 'key' && resource_tag != undefined) { found_tags.push(string) } + if (comparison == '==' && resource_tag == value) { found_tags.push(string) } + if (comparison == '!=' && resource_tag != value) { found_tags.push(string) } + + if (comparison == '=~') { + if (resource_tag != undefined && value.test(resource_tag)) { found_tags.push(string) } + } + + if (comparison == '!~') { + if (resource_tag == undefined) { found_tags.push(string) } + if (resource_tag != undefined && value.test(resource_tag)) { found_tags.push(string) } + } + }) + + all_tags_found = found_tags.length == comparators.length + any_tags_found = found_tags.length > 0 && param_exclusion_tags_boolean == 'Any' + + return all_tags_found || any_tags_found + }) + } else { + result = ds_elasticache_clusters_with_tags + } +EOS +end + +datasource "ds_cloudwatch_queries" do + run_script $js_cloudwatch_queries, $ds_elasticache_clusters_tag_filtered, $param_stats_lookback +end + +script "js_cloudwatch_queries", type: "javascript" do + parameters "ds_elasticache_clusters_tag_filtered", "param_stats_lookback" + result "result" + code <<-EOS + // Create the various queries we're going to send to CloudWatch for each instance + result = {} + + _.each(ds_elasticache_clusters_tag_filtered, function(cluster) { + // Make sure the queries object has an array for the region to push items to + if (result[cluster['region']] == undefined || result[cluster['region']] == null) { + result[cluster['region']] = [] + } + + metrics = ["CPUUtilization", "FreeableMemory"] + stats = ["Average", "Minimum", "Maximum", "p99", "p95", "p90"] + lookback = param_stats_lookback * 86400 + + _.each(metrics, function(metric) { + _.each(stats, function(stat) { + id_metric = "cpu" + if (metric == "FreeableMemory") { id_metric = "mem" } + + query = { + "Id": cluster['statId'] + "___" + id_metric + "___" + stat, + "MetricStat": { + "Metric": { + "Namespace": "AWS/ElastiCache", + "MetricName": metric, + "Dimensions": [ + { "Name": "CacheClusterId", "Value": cluster['id'] } + ] + }, + "Period": lookback, + "Stat": stat + }, + "ReturnData": true + } + + result[cluster['region']].push(query) + }) + }) + }) +EOS +end + +# Combine queries into 500 item blocks so we can make bulk requests to Cloudwatch +datasource "ds_cloudwatch_requests" do + run_script $js_cloudwatch_requests, $ds_cloudwatch_queries, $param_stats_lookback +end + +script "js_cloudwatch_requests", type: "javascript" do + parameters "ds_cloudwatch_queries", "param_stats_lookback" + result "result" + code <<-EOS + // Organize the queries into discrete requests to send in. + // Queries are first sorted by region and then split into 500 item blocks. + result = [] + query_block_size = 500 + + // Round down to beginning of the hour to avoid getting multiple values + // from CloudWatch due to how the data is sliced + end_date = new Date() + end_date.setMinutes(0, 0, 0) + end_date = parseInt(end_date.getTime() / 1000) + + start_date = new Date() + start_date.setDate(start_date.getDate() - param_stats_lookback) + start_date.setMinutes(0, 0, 0) + start_date = parseInt(start_date.getTime() / 1000) + + _.each(_.keys(ds_cloudwatch_queries), function(region) { + for (i = 0; i < ds_cloudwatch_queries[region].length; i += query_block_size) { + chunk = ds_cloudwatch_queries[region].slice(i, i + query_block_size) + + result.push({ + body: { + "StartTime": start_date, + "EndTime": end_date, + "MetricDataQueries": chunk + }, + region: region + }) + } + }) +EOS +end + +datasource "ds_cloudwatch_data" do + iterate $ds_cloudwatch_requests + request do + run_script $js_cloudwatch_data, val(iter_item, "region"), val(iter_item, "body") + end + result do + encoding "json" + collect jmes_path(response, "MetricDataResults[*]") do + field "region", val(iter_item, "region") + field "id", jmes_path(col_item, "Id") + field "label", jmes_path(col_item, "Label") + field "statusCode", jmes_path(col_item, "StatusCode") + field "values", jmes_path(col_item, "Values") + end + end +end + +script "js_cloudwatch_data", type: "javascript" do + parameters "region", "body" + result "request" + code <<-EOS + // Slow down rate of requests to prevent + api_wait = 5 + var now = new Date().getTime() + while(new Date().getTime() < now + (api_wait * 1000)) { /* Do nothing */ } + + var request = { + auth: "auth_aws", + host: 'monitoring.' + region + '.amazonaws.com', + pagination: "pagination_aws_getmetricdata", + verb: "POST", + path: "/", + headers: { + "User-Agent": "RS Policies", + "Content-Type": "application/json", + "x-amz-target": "GraniteServiceVersion20100801.GetMetricData", + "Accept": "application/json", + "Content-Encoding": "amz-1.0" + } + query_params: { + 'Action': 'GetMetricData', + 'Version': '2010-08-01' + }, + body: JSON.stringify(body) + } +EOS +end + +datasource "ds_cloudwatch_data_sorted" do + run_script $js_cloudwatch_data_sorted, $ds_cloudwatch_data +end + +script "js_cloudwatch_data_sorted", type: "javascript" do + parameters "ds_cloudwatch_data" + result "result" + code <<-EOS + // Sort the CloudWatch data into an object with keys for regions and instance names. + // This eliminates the need to "double loop" later on to match it with our instances list. + result = {} + + _.each(ds_cloudwatch_data, function(item) { + region = item['region'] + id = item['id'].split('___')[0] + metric = item['id'].split('___')[1] + stat = item['id'].split('___')[2] + + // Grabbing index 0 SHOULD be safe because we should only get one result. + // Just in case AWS slices the data weirdly and returns 2 results, we make + // sure we grab the last item every time, which contains the actual data we need. + value = item['values'][item['values'].length - 1] + + if (result[region] == undefined) { result[region] = {} } + if (result[region][id] == undefined) { result[region][id] = {} } + if (result[region][id][metric] == undefined) { result[region][id][metric] = {} } + + result[region][id][metric][stat] = value + }) +EOS +end + +datasource "ds_elasticache_clusters_with_metrics" do + run_script $js_elasticache_clusters_with_metrics, $ds_elasticache_clusters_tag_filtered, $ds_cloudwatch_data_sorted +end + +script "js_elasticache_clusters_with_metrics", type: "javascript" do + parameters "ds_elasticache_clusters_tag_filtered", "ds_cloudwatch_data_sorted" + result "result" + code <<-EOS + result = _.map(ds_elasticache_clusters_tag_filtered, function(cluster) { + memoryBytes = cluster["memory"] * 1024 * 1024 * 1024 + + metrics = null + + cpuAverage = null + cpuMinimum = null + cpuMaximum = null + cpuP90 = null + cpuP95 = null + cpuP99 = null + + memAverage = null + memMinimum = null + memMaximum = null + memP90 = null + memP95 = null + memP99 = null + + if (ds_cloudwatch_data_sorted[cluster['region']]) { + if (ds_cloudwatch_data_sorted[cluster['region']][cluster['statId']]) { + metrics = ds_cloudwatch_data_sorted[cluster['region']][cluster['statId']] + } + } + + if (metrics) { + if (metrics["cpu"]) { + if (metrics["cpu"]["Average"]) { cpuAverage = metrics["cpu"]["Average"] } + if (metrics["cpu"]["Minimum"]) { cpuMinimum = metrics["cpu"]["Minimum"] } + if (metrics["cpu"]["Maximum"]) { cpuMaximum = metrics["cpu"]["Maximum"] } + if (metrics["cpu"]["p90"]) { cpuP90 = metrics["cpu"]["p90"] } + if (metrics["cpu"]["p95"]) { cpuP95 = metrics["cpu"]["p95"] } + if (metrics["cpu"]["p99"]) { cpuP99 = metrics["cpu"]["p99"] } + } + + if (metrics["mem"]) { + if (metrics["mem"]["Average"]) { memAverage = metrics["mem"]["Average"] / memoryBytes * 100 } + if (metrics["mem"]["Minimum"]) { memMinimum = metrics["mem"]["Minimum"] / memoryBytes * 100 } + if (metrics["mem"]["Maximum"]) { memMaximum = metrics["mem"]["Maximum"] / memoryBytes * 100 } + if (metrics["mem"]["p90"]) { memP90 = metrics["mem"]["p90"] / memoryBytes * 100 } + if (metrics["mem"]["p95"]) { memP95 = metrics["mem"]["p95"] / memoryBytes * 100 } + if (metrics["mem"]["p99"]) { memP99 = metrics["mem"]["p99"] / memoryBytes * 100 } + } + } + + return { + id: cluster["id"], + statId: cluster["statId"], + resourceID: cluster["resourceID"], + resourceName: cluster["resourceName"], + resourceARN: cluster["resourceARN"], + tags: cluster["tags"], + status: cluster["status"], + numberOfNodes: cluster["numberOfNodes"], + autoMinorVersionUpgrade: cluster["autoMinorVersionUpgrade"], + availabilityZone: cluster["availabilityZone"], + createTime: cluster["createTime"], + engine: cluster["engine"], + engineVersion: cluster["engineVersion"], + region: cluster["region"], + resourceType: cluster["resourceType"], + newResourceType: cluster["newResourceType"], + vcpu: cluster["vcpu"], + memory: cluster["memory"], + cpuAverage: cpuAverage, + cpuMinimum: cpuMinimum, + cpuMaximum: cpuMaximum, + cpuP90: cpuP90, + cpuP95: cpuP95, + cpuP99: cpuP99, + memAverage: memAverage, + memMinimum: memMinimum, + memMaximum: memMaximum, + memP90: memP90, + memP95: memP95, + memP99: memP99 + } + }) +EOS +end + +datasource "ds_cluster_costs" do + request do + run_script $js_cluster_costs, $ds_aws_account, $ds_top_level_bcs, rs_org_id, rs_optima_host + end + result do + encoding "json" + collect jmes_path(response, "rows[*]") do + field "resourceId", jmes_path(col_item, "dimensions.resource_id") + field "resourceType", jmes_path(col_item, "dimensions.resource_type") + field "vendorAccountName", jmes_path(col_item, "dimensions.vendor_account_name") + field "adjustmentName", jmes_path(col_item, "dimensions.adjustment_name") + field "cost", jmes_path(col_item, "metrics.cost_amortized_unblended_adj") + end + end +end + +script "js_cluster_costs", type: "javascript" do + parameters "ds_aws_account", "ds_top_level_bcs", "rs_org_id", "rs_optima_host" + result "request" + code <<-EOS + end_date = new Date() + end_date.setDate(end_date.getDate() - 2) + end_date = end_date.toISOString().split('T')[0] + + start_date = new Date() + start_date.setDate(start_date.getDate() - 3) + start_date = start_date.toISOString().split('T')[0] + + var request = { + auth: "auth_flexera", + host: rs_optima_host, + verb: "POST", + path: "/bill-analysis/orgs/" + rs_org_id + "/costs/select", + body_fields: { + dimensions: ["resource_id", "vendor_account_name", "resource_type", "adjustment_name"], + granularity: "day", + start_at: start_date, + end_at: end_date, + metrics: ["cost_amortized_unblended_adj"], + billing_center_ids: ds_top_level_bcs, + limit: 100000, + filter: { + type: "and", + expressions: [ + { + dimension: "service", + type: "equal", + value: "AmazonElastiCache" + }, + { + dimension: "vendor_account", + type: "equal", + value: ds_aws_account['id'] + }, + { + type: "not", + expression: { + dimension: "adjustment_name", + type: "substring", + substring: "Shared" + } + } + ] + } + }, + headers: { + 'User-Agent': "RS Policies", + 'Api-Version': "1.0" + }, + ignore_status: [400] + } +EOS +end + +datasource "ds_cluster_costs_grouped" do + run_script $js_cluster_costs_grouped, $ds_cluster_costs +end + +script "js_cluster_costs_grouped", type: "javascript" do + parameters "ds_cluster_costs" + result "result" + code <<-EOS + // Multiple a single day's cost by the average number of days in a month. + // The 0.25 is to account for leap years for extra precision. + cost_multiplier = 365.25 / 12 + + // Group cost data by resourceId for later use + result = {} + + _.each(ds_cluster_costs, function(item) { + if (typeof(item['resourceId']) == 'string' && item['resourceId'].indexOf("cluster:") == 0) { + id = item['resourceId'].split("cluster:")[1].toLowerCase() + + if (result[id] == undefined) { result[id] = 0.0 } + result[id] += item['cost'] * cost_multiplier + } + }) +EOS +end + +datasource "ds_underutilized_elasticache_clusters" do + run_script $js_underutilized_elasticache_clusters, $ds_elasticache_clusters_with_metrics, $ds_cluster_costs_grouped, $ds_aws_account, $ds_currency, $ds_applied_policy, $param_stats_lookback, $param_stats_threshold, $param_stats_threshold_cpu_value, $param_stats_threshold_mem_value, $param_min_savings +end + +script "js_underutilized_elasticache_clusters", type: "javascript" do + parameters "ds_elasticache_clusters_with_metrics", "ds_cluster_costs_grouped", "ds_aws_account", "ds_currency", "ds_applied_policy", "param_stats_lookback", "param_stats_threshold", "param_stats_threshold_cpu_value", "param_stats_threshold_mem_value", "param_min_savings" + result "result" + code <<-'EOS' + // Used for formatting numbers to look pretty + function formatNumber(number, separator){ + numString = number.toString() + values = numString.split(".") + formatted_number = '' + + while (values[0].length > 3) { + var chunk = values[0].substr(-3) + values[0] = values[0].substr(0, values[0].length - 3) + formatted_number = separator + chunk + formatted_number + } + + if (values[0].length > 0) { formatted_number = values[0] + formatted_number } + if (values[1] == undefined) { return formatted_number } + + return formatted_number + "." + values[1] + } + + result = [] + total_savings = 0.0 + + // Find specs for current and potential downsize in the size map + _.each(ds_elasticache_clusters_with_metrics, function(cluster) { + // Determine if the resource is underutilized based on user inputs + cpu_stat = null + mem_stat = null + + if (param_stats_threshold == "Average") { + cpu_stat = cluster["cpuAverage"] + mem_stat = cluster["memAverage"] + } + + if (param_stats_threshold == "Maximum") { + cpu_stat = cluster["cpuMaximum"] + mem_stat = cluster["memMaximum"] + } + + if (param_stats_threshold == "p90") { + cpu_stat = cluster["cpuP90"] + mem_stat = cluster["memP90"] + } + + if (param_stats_threshold == "p95") { + cpu_stat = cluster["cpuP95"] + mem_stat = cluster["memP95"] + } + + if (param_stats_threshold == "p99") { + cpu_stat = cluster["cpuP99"] + mem_stat = cluster["memP99"] + } + + cpu_underutilized = cpu_stat < param_stats_threshold_cpu_value || param_stats_threshold_cpu_value == -1 + mem_underutilized = mem_stat > param_stats_threshold_mem_value || param_stats_threshold_mem_value == -1 + + // Calculate estimated savings + savings = 0.0 + if (ds_cluster_costs_grouped[cluster['id']]) { savings = ds_cluster_costs_grouped[cluster['id']] / 2 } + + // Check if cluster is underutilized and if savings is above threshold + if (cpu_underutilized && mem_underutilized && savings >= param_min_savings) { + total_savings += savings + + recommendationDetails = [ + "Change node type of ElastiCache cluster ", cluster['id'], " ", + "in AWS Account ", ds_aws_account['name'], " ", + "(", ds_aws_account['id'], ") ", + "from ", cluster['resourceType'], " ", + "to ", cluster['newResourceType'] + ].join('') + + tags = [] + + if (typeof(cluster['tags']) == 'object') { + tags = _.map(cluster['tags'], function(tag) { return [tag['Key'], tag['Value']].join('=') }) + } + + result.push({ + accountID: ds_aws_account['id'], + accountName: ds_aws_account['name'], + id: cluster["id"], + resourceID: cluster["resourceID"], + resourceName: cluster["resourceName"], + resourceARN: cluster["resourceARN"], + status: cluster["status"], + numberOfNodes: cluster["numberOfNodes"], + autoMinorVersionUpgrade: cluster["autoMinorVersionUpgrade"], + availabilityZone: cluster["availabilityZone"], + createTime: cluster["createTime"], + engine: cluster["engine"], + engineVersion: cluster["engineVersion"], + region: cluster["region"], + resourceType: cluster["resourceType"], + newResourceType: cluster["newResourceType"], + tags: tags.join(', '), + cpuAverage: Math.round(cluster["cpuAverage"] * 100) / 100, + cpuMinimum: Math.round(cluster["cpuMinimum"] * 100) / 100, + cpuMaximum: Math.round(cluster["cpuMaximum"] * 100) / 100, + cpuP90: Math.round(cluster["cpuP90"] * 100) / 100, + cpuP95: Math.round(cluster["cpuP95"] * 100) / 100, + cpuP99: Math.round(cluster["cpuP99"] * 100) / 100, + memAverage: Math.round(cluster["memAverage"] * 100) / 100, + memMinimum: Math.round(cluster["memMinimum"] * 100) / 100, + memMaximum: Math.round(cluster["memMaximum"] * 100) / 100, + memP90: Math.round(cluster["memP90"] * 100) / 100, + memP95: Math.round(cluster["memP95"] * 100) / 100, + memP99: Math.round(cluster["memP99"] * 100) / 100, + service: "AmazonElastiCache", + recommendationDetails: recommendationDetails, + savings: Math.round(savings * 1000) / 1000, + savingsCurrency: ds_currency['symbol'], + policy_name: ds_applied_policy['name'], + // These are to avoid errors when we hash_exclude these fields + message: "", + total_savings: "" + }) + } + }) + + // Message for incident output + total_clusters = ds_elasticache_clusters_with_metrics.length.toString() + total_oversized = result.length.toString() + oversized_percentage = (total_oversized / total_clusters * 100).toFixed(2).toString() + '%' + + cluster_noun = "cluster" + if (total_clusters > 1) { cluster_noun += "s" } + + cluster_verb = "is" + if (total_oversized > 1) { cluster_verb = "are" } + + findings = [ + "Out of ", total_clusters, " AWS ElastiCache ", cluster_noun, " analyzed, ", + total_oversized, " (", oversized_percentage, + ") ", cluster_verb, " underutilized and recommended for downsizing. " + ].join('') + + if (param_stats_threshold_cpu_value != -1 && param_stats_threshold_mem_value == -1) { + settings = [ + "A cluster is considered underutilized if its ", param_stats_threshold.toLowerCase(), + " CPU usage is below ", param_stats_threshold_cpu_value, "%, ", + "regardless of its memory usage.\n\n" + ].join('') + } else if (param_stats_threshold_cpu_value == -1 && param_stats_threshold_mem_value != -1) { + settings = [ + "A cluster is considered underutilized if its ", param_stats_threshold.toLowerCase(), + " freeable memory is above ", param_stats_threshold_mem_value, "%, ", + "regardless of its CPU usage.\n\n" + ].join('') + } else if (param_stats_threshold_cpu_value != -1 && param_stats_threshold_mem_value != -1) { + settings = [ + "A cluster is considered underutilized if its ", param_stats_threshold.toLowerCase(), + " CPU usage is below ", param_stats_threshold_cpu_value, "% and its ", + param_stats_threshold.toLowerCase(), " freeable memory is above ", + param_stats_threshold_mem_value, "%.\n\n" + ].join('') + } else { + settings = [ + "All clusters are considered underutilized due to both the ", + "CPU Threshold (%) and Freeable Memory Threshold (%) parameters being set to -1.\n\n" + ].join('') + } + + disclaimer = "The above settings can be modified by editing the applied policy and changing the appropriate parameters." + + total_savings = ds_currency['symbol'] + ' ' + formatNumber(Math.round(total_savings * 100) / 100, ds_currency['separator']) + + // Dummy item to ensure the policy's check statement always executes at least once + result.push({ + accountID: "", + accountName: "", + id: "", + resourceID: "", + resourceName: "", + resourceARN: "", + tags: "", + status: "", + numberOfNodes: "", + autoMinorVersionUpgrade: "", + availabilityZone: "", + createTime: "", + engine: "", + engineVersion: "", + region: "", + resourceType: "", + newResourceType: "", + cpuAverage: "", + cpuMinimum: "", + cpuMaximum: "", + cpuP90: "", + cpuP95: "", + cpuP99: "", + memAverage: "", + memMinimum: "", + memMaximum: "", + memP90: "", + memP95: "", + memP99: "", + service: "", + recommendationDetails: "", + savings: "", + savingsCurrency: "", + policy_name: "", + message: "", + total_savings: "" + }) + + result[0]['message'] = findings + settings + disclaimer + result[0]['total_savings'] = total_savings +EOS +end + +############################################################################### +# Policy +############################################################################### + +policy "pol_underutilized_elasticache_clusters" do + validate_each $ds_underutilized_elasticache_clusters do + summary_template "{{ with index data 0 }}{{ .policy_name }}{{ end }}: {{ len data }} AWS Underutilized ElastiCache Clusters Found" + detail_template <<-'EOS' + **Potential Monthly Savings:** {{ with index data 0 }}{{ .total_savings }}{{ end }} + + {{ with index data 0 }}{{ .message }}{{ end }} + EOS + check logic_or($ds_parent_policy_terminated, eq(val(item, "resourceID"), "")) + escalate $esc_email + escalate $esc_resize_clusters + hash_exclude "message", "total_savings", "tags", "savings", "savingsCurrency" + export do + resource_level true + field "accountID" do + label "Account ID" + end + field "accountName" do + label "Account Name" + end + field "resourceID" do + label "Cluster ID" + end + field "region" do + label "Cluster Region" + end + field "tags" do + label "Cluster Tags" + end + field "numberOfNodes" do + label "Nodes (#)" + end + field "resourceType" do + label "Node Type" + end + field "newResourceType" do + label "Recommended Node Type" + end + field "recommendationDetails" do + label "Recommendation" + end + field "savings" do + label "Estimated Monthly Savings" + end + field "savingsCurrency" do + label "Savings Currency" + end + field "cpuMaximum" do + label "CPU Maximum %" + end + field "cpuMinimum" do + label "CPU Minimum %" + end + field "cpuAverage" do + label "CPU Average %" + end + field "cpuP99" do + label "CPU p99" + end + field "cpuP95" do + label "CPU p95" + end + field "cpuP90" do + label "CPU p90" + end + field "memMaximum" do + label "Freeable Memory Maximum %" + end + field "memMinimum" do + label "Freeable Memory Minimum %" + end + field "memAverage" do + label "Freeable Memory Average %" + end + field "memP99" do + label "Freeable Memory p99" + end + field "memP95" do + label "Freeable Memory p95" + end + field "memP90" do + label "Freeable Memory p90" + end + field "status" do + label "Cluster Status" + end + field "engine" do + label "Engine" + end + field "engineVersion" do + label "Engine Version" + end + field "service" do + label "Service" + end + field "resourceARN" do + label "Resource ARN" + end + field "resourceName" do + label "Resource Name" + end + field "id" do + label "ID" + end + end + end +end + +############################################################################### +# Escalations +############################################################################### + +escalation "esc_email" do + automatic true + label "Send Email" + description "Send incident email" + email $param_email +end + +escalation "esc_resize_clusters" do + automatic contains($param_automatic_action, "Downsize ElastiCache Nodes") + label "Downsize ElastiCache Nodes" + description "Approval to downsize nodes in all selected ElastiCache clusters" + run "resize_clusters", data +end + +############################################################################### +# Cloud Workflow +############################################################################### + +define resize_clusters($data) return $all_responses do + $$all_responses = [] + + foreach $clb in $data do + sub on_error: handle_error() do + call resize_cluster($cluster) retrieve $response + end + end + + if inspect($$errors) != "null" + raise join($$errors, "\n") + end +end + +define resize_cluster($cluster) return $response do + $host = "elasticache." + $cluster["region"] + ".amazonaws.com" + $href = "/" + $params = "?Action=ModifyCacheCluster&Version=2015-02-02&ApplyImmediately=true&CacheClusterId=" + $cluster["resourceID"] + "&CacheNodeType=" + $cluster["newResourceType"] + $url = $host + $href + $params + task_label("POST " + $url) + + $response = http_request( + auth: $$auth_aws, + https: true, + verb: "post", + host: $host, + href: $href, + query_strings: { + "Action": "ModifyCacheCluster", + "Version": "2015-02-02", + "CacheClusterId": $cluster["resourceID"], + "CacheNodeType": $cluster["newResourceType"], + "ApplyImmediately": "true" + } + ) + + task_label("POST AWS ElastiCache cluster response: " + $cluster["resourceID"] + " " + to_json($response)) + $$all_responses << to_json({"req": "POST " + $url, "resp": $response}) + + if $response["code"] != 204 && $response["code"] != 202 && $response["code"] != 200 + raise "Unexpected response from POST AWS ElastiCache cluster: "+ $cluster["resourceID"] + " " + to_json($response) + else + task_label("POST AWS ElastiCache cluster successful: " + $cluster["resourceID"]) + end +end + +define handle_error() do + if !$$errors + $$errors = [] + end + $$errors << $_error["type"] + ": " + $_error["message"] + # We check for errors at the end, and raise them all together + # Skip errors handled by this definition + $_error_behavior = "skip" +end + +############################################################################### +# Meta Policy [alpha] +# Not intended to be modified or used by policy developers +############################################################################### + +# If the meta_parent_policy_id is not set it will evaluate to an empty string and we will look for the policy itself, +# if it is set we will look for the parent policy. +datasource "ds_get_policy" do + request do + auth $auth_flexera + host rs_governance_host + ignore_status [404] + path join(["/api/governance/projects/", rs_project_id, "/applied_policies/", switch(ne(meta_parent_policy_id, ""), meta_parent_policy_id, policy_id)]) + header "Api-Version", "1.0" + end + result do + encoding "json" + field "id", jmes_path(response, "id") + end +end + +datasource "ds_parent_policy_terminated" do + run_script $js_decide_if_self_terminate, $ds_get_policy, policy_id, meta_parent_policy_id +end + +# If the policy was applied by a meta_parent_policy we confirm it exists if it doesn't we confirm we are deleting +# This information is used in two places: +# - determining whether or not we make a delete call +# - determining if we should create an incident (we don't want to create an incident on the run where we terminate) +script "js_decide_if_self_terminate", type: "javascript" do + parameters "found", "self_policy_id", "meta_parent_policy_id" + result "result" + code <<-EOS + var result + if (meta_parent_policy_id != "" && found.id == undefined) { + result = true + } else { + result = false + } +EOS +end + +# Two potentials ways to set this up: +# - this way and make a unneeded 'get' request when not deleting +# - make the delete request an interate and have it iterate over an empty array when not deleting and an array with one item when deleting +script "js_make_terminate_request", type: "javascript" do + parameters "should_delete", "policy_id", "rs_project_id", "rs_governance_host" + result "request" + code <<-EOS + + var request = { + auth: 'auth_flexera', + host: rs_governance_host, + path: "/api/governance/projects/" + rs_project_id + "/applied_policies/" + policy_id, + headers: { + "API-Version": "1.0", + "Content-Type":"application/json" + }, + } + + if (should_delete) { + request.verb = 'DELETE' + } + EOS +end + +datasource "ds_terminate_self" do + request do + run_script $js_make_terminate_request, $ds_parent_policy_terminated, policy_id, rs_project_id, rs_governance_host + end +end + +datasource "ds_is_deleted" do + run_script $js_check_deleted, $ds_terminate_self +end + +# This is just a way to have the check delete request connect to the farthest leaf from policy. +# We want the delete check to the first thing the policy does to avoid the policy erroring before it can decide whether or not it needs to self terminate +# Example a customer deletes a credential and then terminates the parent policy. We still want the children to self terminate +# The only way I could see this not happening is if the user who applied the parent_meta_policy was offboarded or lost policy access, the policies who are impersonating the user +# would not have access to self-terminate +# It may be useful for the backend to enable a mass terminate at some point for all meta_child_policies associated with an id. +script "js_check_deleted", type: "javascript" do + parameters "response" + result "result" + code <<-EOS + result = {"path":"/"} + EOS +end diff --git a/cost/aws/rightsize_elasticache/aws_rightsize_elasticache_meta_parent.pt b/cost/aws/rightsize_elasticache/aws_rightsize_elasticache_meta_parent.pt new file mode 100644 index 0000000000..b960550d23 --- /dev/null +++ b/cost/aws/rightsize_elasticache/aws_rightsize_elasticache_meta_parent.pt @@ -0,0 +1,1399 @@ +name "Meta Parent: AWS Rightsize ElastiCache" +rs_pt_ver 20180301 +type "policy" +short_description "**NOTE: Meta policies are an alpha feature. Please consult the [README](https://github.com/flexera-public/policy_templates/blob/master/README_META_POLICIES.md) before use.** Applies and manages \"child\" [AWS Rightsize ElastiCache](https://github.com/flexera-public/policy_templates/tree/master/cost/aws/rightsize_elasticache) Policies." +severity "low" +category "Meta" +default_frequency "15 minutes" +info( + provider: "AWS", + version: "0.1.0", # This version of the Meta Parent Policy Template should match the version of the Child Policy Template as it appears in the Catalog for best reliability + publish: "true", + deprecated: "false" +) + +############################################################################## +# Parameters +############################################################################## + +## Meta Parent Parameters +## These are params specific to the meta parent policy. +parameter "param_combined_incident_email" do + type "list" + label "Email addresses for combined incident" + description "A list of email addresses to notify with the consolidated child policy incident." + default [] +end + +parameter "param_dimension_filter_includes" do + type "list" + label "Dimension Include Filters" + description <<-EOS + Filters [`dimension_name=dimension_value` and `dimension_name=~dimension_value` pairs] to determine which AWS Accounts returned by the Flexera Bill Analysis API to **INCLUDE** and be applied to. + Use = to match the entire value and =~ to match a substring contained in the value. + During each run this policy will select AWS Accounts who match **all** the filters defined and apply a child policy for each. + If no include filters are provided, then all AWS Accounts are included by default. + Most of the dimensions in Flexera can be used [default dimensions, custom tag dimensions, rule-based dimensions]. Full list of available dimensions documented in the [Bill Analysis API Docs](https://reference.rightscale.com/bill_analysis/). + EOS + default [] +end + +parameter "param_dimension_filter_excludes" do + type "list" + label "Dimension Exclude Filters" + description <<-EOS + Filters [`dimension_name=dimension_value` and `dimension_name=~dimension_value` pairs] to determine which AWS Accounts returned by the Flexera Bill Analysis API to **EXCLUDE** and *not* have policy applied to. + Use = to match the entire value and =~ to match a substring contained in the value. + During each run this policy will select AWS Accounts who match **all** the filters defined here and excludes them from results. + Can be used to exclude specific AWS Accounts [`vendor_account=123456789012`] + Most of the dimensions in Flexera can be used [default dimensions, custom tag dimensions, rule-based dimensions]. Full list of available dimensions documented in the [Bill Analysis API Docs](https://reference.rightscale.com/bill_analysis/). + EOS + default [] +end + +parameter "param_policy_schedule" do + type "string" + label "Child Policy Schedule" + description "The interval at which the child policy checks for conditions and generates incidents." + default "weekly" + allowed_values "daily", "weekly", "monthly" +end + +parameter "param_template_source" do + type "string" + label "Child Policy Template Source" + description "By default, will use the \"AWS Rightsize ElastiCache\" Policy Template from Catalog. Optionally, you can use the \"AWS Rightsize ElastiCache\" Policy Template uploaded in the current Flexera Project." + default "Published Catalog Template" + allowed_values "Published Catalog Template", "Uploaded Template" +end + +## Child Policy Parameters +parameter "param_min_savings" do + type "number" + category "Policy Settings" + label "Minimum Savings Threshold" + description "Minimum potential savings required to generate a recommendation" + min_value 0 + default 0 +end + +parameter "param_exclusion_tags" do + type "list" + category "Filters" + label "Exclusion Tags" + description "Cloud native tags to ignore clusters that you don't want to produce recommendations for. Enter the Key name to filter clusters with a specific Key, regardless of Value, and enter Key==Value to filter clusters with a specific Key:Value pair. Other operators and regex are supported; please see the README for more details." + default [] +end + +parameter "param_exclusion_tags_boolean" do + type "string" + category "Filters" + label "Exclusion Tags: Any / All" + description "Whether to filter clusters containing any of the specified tags or only those that contain all of them. Only applicable if more than one value is entered in the 'Exclusion Tags' field." + allowed_values "Any", "All" + default "Any" +end + +parameter "param_regions_allow_or_deny" do + type "string" + category "Filters" + label "Allow/Deny Regions" + description "Allow or Deny entered regions. See the README for more details" + allowed_values "Allow", "Deny" + default "Allow" +end + +parameter "param_regions_list" do + type "list" + category "Filters" + label "Allow/Deny Regions List" + description "A list of allowed or denied regions. See the README for more details" + allowed_pattern /^([a-zA-Z-_]+-[a-zA-Z0-9-_]+-[0-9-_]+,*|)+$/ + default [] +end + +parameter "param_stats_lookback" do + type "number" + category "Statistics" + label "Statistic Lookback Period" + description "How many days back to look at CPU and memory usage data for clusters. This value cannot be set higher than 90 because AWS does not retain metrics for longer than 90 days." + min_value 1 + max_value 90 + default 30 +end + +parameter "param_stats_threshold" do + type "string" + category "Statistics" + label "Threshold Statistic" + description "Statistic to use when determining if a cluster is underutilized" + allowed_values "Average", "Maximum", "p99", "p95", "p90" + default "Average" +end + +parameter "param_stats_threshold_cpu_value" do + type "number" + category "Statistics" + label "CPU Threshold (%)" + description "The CPU threshold at which to consider a cluster to be 'underutilized' and therefore be flagged for downsizing. Set to -1 to ignore CPU utilization" + min_value -1 + max_value 100 + default 40 +end + +parameter "param_stats_threshold_mem_value" do + type "number" + category "Statistics" + label "Freeable Memory Threshold (%)" + description "The freeable memory threshold at which to consider a cluster to be 'underutilized' and therefore be flagged for downsizing. Set to -1 to ignore memory utilization" + min_value -1 + max_value 100 + default 60 +end + +parameter "param_automatic_action" do + type "list" + category "Actions" + label "Automatic Actions" + description "When this value is set, this policy will automatically take the selected action(s)" + allowed_values ["Downsize ElastiCache Nodes"] + default [] +end + +############################################################################### +# Authentication +############################################################################### +credentials "auth_aws" do + schemes "aws", "aws_sts" + label "AWS" + description "Select the AWS Credential from the list" + tags "provider=aws" + aws_account_number $param_aws_account_number +end + +credentials "auth_flexera" do + schemes "oauth2" + label "Flexera" + description "Select Flexera One OAuth2 credentials" + tags "provider=flexera" +end + +############################################################################### +# Datasources +############################################################################### + +# Get Applied Parent Policy Details +datasource "ds_self_policy_information" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/applied_policies/", policy_id]) + header "Api-Version", "1.0" + end + result do + encoding "json" + field "name", jmes_path(response, "name") + field "creator_id", jmes_path(response, "created_by.id") + field "credentials", jmes_path(response, "credentials") + field "options", jmes_path(response, "options") + end +end + +datasource "ds_child_policy_options" do + run_script $js_child_policy_options, $ds_self_policy_information +end + +script "js_child_policy_options", type: "javascript" do + parameters "ds_self_policy_information" + result "options" + code <<-EOS + // Filter Options that are not appropriate for Child Policy + var options = _.map(ds_self_policy_information.options, function(option){ + // param_combined_incident_email, param_dimension_filter_includes, param_dimension_filter_excludes, param_policy_schedule are exclusion to Meta Parent Policy Parameters + if (!_.contains(["param_combined_incident_email", "param_dimension_filter_includes", "param_dimension_filter_excludes", "param_policy_schedule", "param_template_source"], option.name)) { + return { "name": option.name, "value": option.value }; + } + }); + // Explicitly add param_email which is disabled/does not exist in meta parent policy + options.push({ + "name": "param_email", + "value": [] + }); + EOS +end + +datasource "ds_child_policy_options_map" do + run_script $js_child_policy_options_map, $ds_child_policy_options +end + +script "js_child_policy_options_map", type: "javascript" do + parameters "ds_child_policy_options" + result "options" + code <<-EOS + function format_options_keyvalue(options) { + var options_keyvalue_map = {}; + _.each(options, function(option) { + options_keyvalue_map[option.name] = option.value; + }); + return options_keyvalue_map; + } + var options = format_options_keyvalue(ds_child_policy_options) + EOS +end + +datasource "ds_format_self" do + run_script $js_format_self, $ds_self_policy_information, $ds_child_policy_options_map +end + +script "js_format_self", type: "javascript" do + parameters "ds_self_policy_information", "ds_child_policy_options_map" + result "formatted" + code <<-EOS + var formatted = { + "name": ds_self_policy_information["name"], + "creator_id": ds_self_policy_information["creator_id"], + "credentials": ds_self_policy_information["credentials"], + "options": ds_child_policy_options_map + }; + EOS +end + +# Get Pulished Policy Details +datasource "ds_get_published_child_policy_information" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/orgs/", rs_org_id, "/published_templates"]) + header "Api-Version", "1.0" + end + result do + encoding "json" + collect jmes_path(response, "items[*]") do + field "name", jmes_path(col_item, "name") + field "created_by", jmes_path(col_item, "created_by.email") + field "href", jmes_path(col_item, "href") + field "short_description", jmes_path(col_item, "short_description") + end + end +end + +# Select the published policy that is published by "support@flexera.com" and matches the name of the child policy template +datasource "ds_published_child_policy_information" do + run_script $js_published_child_policy_information, $ds_get_published_child_policy_information +end + +script "js_published_child_policy_information", type: "javascript" do + parameters "ds_get_published_child_policy_information" + result "result" + code <<-EOS + result = _.filter(ds_get_published_child_policy_information, function(item) { + return item['name'] == "AWS Rightsize ElastiCache" && item['created_by'] == "support@flexera.com" + }) +EOS +end + +# Get Uploaded Policy Details +datasource "ds_get_project_child_policy_information" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/policy_templates"]) + header "Api-Version", "1.0" + end + result do + encoding "json" + collect jmes_path(response, "items[*]") do + field "name", jmes_path(col_item, "name") + field "href", jmes_path(col_item, "href") + field "short_description", jmes_path(col_item, "short_description") + end + end +end + +# Select the uploaded policy that matches the name of the child policy template +datasource "ds_project_child_policy_information" do + run_script $js_project_child_policy_information, $ds_get_project_child_policy_information +end + +script "js_project_child_policy_information", type: "javascript" do + parameters "ds_get_project_child_policy_information" + result "result" + code <<-EOS + result = _.filter(ds_get_project_child_policy_information, function(item) { + return item['name'] == "AWS Rightsize ElastiCache" + }) +EOS +end + +datasource "ds_get_billing_centers" do + request do + auth $auth_flexera + host rs_optima_host + path join(["/analytics/orgs/",rs_org_id,"/billing_centers"]) + header "Api-Version", "1.0" + header "User-Agent", "RS Policies" + query "view", "allocation_table" + ignore_status [403] + end + result do + encoding "json" + # Select the Billing Centers that have "parent_id" undefined or "" (i.e. top-level Billing Centers) + collect jq(response, '.[] | select(.parent_id == null)' ) do + field "href", jq(col_item,".href") + field "id", jq(col_item,".id") + field "name", jq(col_item,".name") + field "parent_id", jq(col_item,".parent_id") + end + end +end + +script "js_make_billing_center_request", type: "javascript" do + parameters "rs_org_id", "rs_optima_host", "billing_centers_unformatted", "param_dimension_filter_includes", "param_dimension_filter_excludes" + result "request" + code <<-EOS + + billing_centers_formatted = [] + + for (x=0; x< billing_centers_unformatted.length; x++) { + billing_centers_formatted.push(billing_centers_unformatted[x]["id"]) + } + + finish = new Date() + finishFormatted = finish.toJSON().split("T")[0] + start = new Date() + start.setDate(start.getDate() - 30) + startFormatted = start.toJSON().split("T")[0] + + // Default dimensions and filter expressions required for meta parent policy + var dimensions = ["vendor_account", "vendor_account_name"]; + var filter_expressions = [ + { dimension: "vendor", type: "equal", value: "AWS" } + ] + + // Append to default dimensions and filter expressions using parent policy params + _.each(param_dimension_filter_includes, function (v) { + // split key=value string + if (v.indexOf('=~') == -1) { + var split = v.split("="); + var type = "equal" + } else { + var split = v.split("=~"); + var type = "substring" + } + + var k = split[0]; + var v = split[1]; + + // append to lists + dimensions.push(k); + + if (type == "equal") { + filter_expressions.push({ dimension: k, type: "equal", value: v }); + } else { + filter_expressions.push({ dimension: k, type: "substring", substring: v }); + } + }); + + // Append to filter expressions using exclude policy params + _.each(param_dimension_filter_excludes, function (v) { + // split key=value string + if (v.indexOf('=~') == -1) { + var split = v.split("="); + var type = "equal" + } else { + var split = v.split("=~"); + var type = "substring" + } + + var k = split[0]; + var v = split[1]; + + // append to lists + dimensions.push(k); + + if (type == "equal") { + filter_expressions.push({ "type": "not", "expression": { "dimension": k, "type": "equal", "value": v } }); + } else { + filter_expressions.push({ "type": "not", "expression": { "dimension": k, "type": "substring", "substring": v } }); + } + }); + + // Produces a duplicate-free version of the array + dimensions = _.uniq(dimensions); + + var body = { + "dimensions": dimensions, + "granularity":"day", + "start_at": startFormatted, + "end_at": finishFormatted, + "metrics":["cost_amortized_unblended_adj"], + "billing_center_ids": billing_centers_formatted, + "filter": + { + "type": "and", + "expressions": filter_expressions + }, + "summarized": true + } + var request = { + auth: 'auth_flexera', + host: rs_optima_host, + scheme: 'https', + verb: 'POST', + path: "/bill-analysis/orgs/"+ rs_org_id + "/costs/aggregated", + headers: { + "API-Version": "1.0", + "Content-Type":"application/json" + }, + body: JSON.stringify(body) + } + EOS +end + +# Get the AWS acounts +datasource "ds_get_aws_accounts" do + request do + run_script $js_make_billing_center_request, rs_org_id, rs_optima_host, $ds_get_billing_centers, $param_dimension_filter_includes, $param_dimension_filter_excludes + end + result do + encoding "json" + collect jmes_path(response,"rows[*]") do + field "aws_account_id", jmes_path(col_item,"dimensions.vendor_account") + field "aws_account_name", jmes_path(col_item,"dimensions.vendor_account_name") + end + end +end + +# Get Child policies +datasource "ds_get_existing_policies" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/applied_policies"]) + header "Api-Version", "1.0" + query "meta_parent_policy_id", policy_id + end + result do + encoding "json" + collect jmes_path(response, "items[*]") do + field "name", jmes_path(col_item, "name") + field "applied_policy_id", jmes_path(col_item, "id") + field "options", jmes_path(col_item, "options") + field "updated_at", jmes_path(col_item, "updated_at") + field "status", jmes_path(col_item, "status") + end + end +end + +# Get Child policies incidents +datasource "ds_get_existing_policies_incidents" do + request do + auth $auth_flexera + host rs_governance_host + path join(["/api/governance/projects/", rs_project_id, "/incidents"]) + header "Api-Version", "1.0" + query "meta_parent_policy_id", policy_id + query "state", "triggered" + end + result do + encoding "json" + collect jmes_path(response, "items[*]") do + field "incident_id", jmes_path(col_item, "id") + field "applied_policy_id", jmes_path(col_item, "applied_policy.id") + field "summary", jmes_path(col_item, "summary") + field "state", jmes_path(col_item, "state") + field "violation_data_count", jmes_path(col_item, "violation_data_count") + field "updated_at", jmes_path(col_item, "updated_at") + field "meta_parent_policy_id", jmes_path(col_item, "meta_parent_policy_id") + end + end +end + +datasource "ds_format_incidents" do + run_script $js_format_existing_policies_incidents, $ds_get_existing_policies_incidents +end + +script "js_format_existing_policies_incidents", type: "javascript" do + parameters "unformatted" + result "formatted" + code <<-EOS + formatted={} + + _.each(unformatted, function(incident) { + if (formatted[incident['applied_policy_id']] == undefined) { + formatted[incident['applied_policy_id']] = [] + } + + formatted[incident['applied_policy_id']].push(incident) + }) +EOS +end + +datasource "ds_format_existing_policies" do + run_script $js_format_existing_policies, $ds_get_existing_policies, $ds_format_incidents +end + +# format +# duplicates logic should compare updated at +# we can validate update here when destructring the existing policy options, don't need updated at +# format options +script "js_format_existing_policies", type: "javascript" do + parameters "ds_get_existing_policies", "ds_format_incidents" + result "result" + code <<-EOS + function format_options_keyvalue(options) { + var options_keyvalue_map = {}; + _.each(options, function(option) { + options_keyvalue_map[option.name] = option.value; + }); + return options_keyvalue_map; + } + + result = {} + formatted = {} + duplicates = [] + // tracking holds all existing policies and later can be used to determine if existing policies should be deleted [i.e. if cloud account was removed] + tracking = {} + + for (x=0; x newDate) { + duplicates.push({ + "applied_policy_id":ds_get_existing_policies[x]["applied_policy_id"], + "applied_policy_name":ds_get_existing_policies[x]["name"], + "status":ds_get_existing_policies[x]["status"], + "updated_at":ds_get_existing_policies[x]["updated_at"], + "incident": incident, + "incident2": incident2 + }) + } else { + duplicates.push({ + "applied_policy_id":current["applied_policy_id"], + "applied_policy_name":current["applied_policy_name"], + "status":current["status"], + "updated_at":current["updated_at"], + "incident": current["incident"], + "incident2": current["incident2"] + }) + formatted[aws_account_id] = { + "applied_policy_id":ds_get_existing_policies[x]["applied_policy_id"], + "applied_policy_name":ds_get_existing_policies[x]["name"], + "status":ds_get_existing_policies[x]["status"], + "updated_at":ds_get_existing_policies[x]["updated_at"], + "incident": incident, + "incident2": incident2, + "options": options + } + + } + } + } + + result.formatted=formatted + result.duplicates=duplicates + result.tracking=tracking + EOS +end + +datasource "ds_take_in_parameters" do + run_script $js_take_in_parameters, $ds_get_aws_accounts, $ds_format_self, first($ds_published_child_policy_information), first($ds_project_child_policy_information), $ds_format_existing_policies, $ds_child_policy_options, $ds_child_policy_options_map, $param_template_source, $param_policy_schedule, policy_id, f1_app_host, rs_org_id, rs_project_id +end + +# hardcode template href with id from catalog +# catalog policies show in customer's published templates with their org id +# "template_href": "/api/governance/orgs/" + rs_org_id + "/published_templates/62618616e3dff80001572bf0" +# update logic: the only reason we're going to update the child policies for is changes to options +# and only some options, email is always blank and aws_account_id is tied to the idenity of each policy, so: new account creation, removal of account: termination +# param_automatic_action is a list with only one action, unless the person is applying using an API and putting the same value multiple times this should either be a length of 0 or 1 +# param_log_to_cm_audit_entries is a String of Yes or No +# param_exclude_tags and param_allowed_regions are arrays. I'm doing an update on the order changing but the values remaining the same. +# If we only want to do an update on the values changing we could sort before doing the equality check. +script "js_take_in_parameters", type: "javascript" do + parameters "ds_get_aws_accounts", "ds_format_self", "ds_published_child_policy_information", "ds_project_child_policy_information", "ds_format_existing_policies", "ds_child_policy_options", "ds_child_policy_options_map", "param_template_source", "param_policy_schedule", "meta_parent_policy_id", "f1_app_host", "rs_org_id", "rs_project_id" + result "grid_and_cwf" + code <<-EOS + + // Set Child Policy Information based on param_template_source value + if (param_template_source == "Published Catalog Template") { + child_policy_information = ds_published_child_policy_information + } else { + child_policy_information = ds_project_child_policy_information + } + + max_actions = 50; + + grid_and_cwf={grid:[], to_create:[], to_update:[], to_delete:[], parent_policy:ds_format_self}; + + should_keep = ds_format_existing_policies.tracking; + + // Construct UI URL prefixes for policy template summary + ui_url_prefix = "https://" + f1_app_host + "/orgs/" + rs_org_id; + applied_policy_url_prefix = ui_url_prefix + "/automation/applied-policies/projects/" + rs_project_id + "?noIndex=1&policyId="; + incident_url_prefix = ui_url_prefix + "/automation/incidents/projects/" + rs_project_id + "?noIndex=1&incidentId="; + + function add_to_grid(ep, action) { + policy_status={ + "id": ep["applied_policy_id"], + "policy_name": ep["applied_policy_name"] + '||' + applied_policy_url_prefix + ep["applied_policy_id"], + "meta_policy_status": action, + "policy_status": ep["status"], + "policy_last_update": ep["updated_at"], + }; + + if (ep.incident != null && ep.incident != undefined) { + // Remove policy name from summary when applicable + summary_parts = ep.incident.summary.split(':') + summary = summary_parts[summary_parts.length - 1].trim() + + policy_status["incident_summary"] = summary + '||' + incident_url_prefix + ep.incident.incident_id; + policy_status["incident_state"] = ep.incident.state; + policy_status["incident_violation_data_count"] = ep.incident.violation_data_count; + policy_status["incident_last_update"] = ep.incident.updated_at; + } + + if (ep.incident2 != null && ep.incident2 != undefined) { + // Remove policy name from summary when applicable + summary_parts = ep.incident2.summary.split(':') + summary = summary_parts[summary_parts.length - 1].trim() + + policy_status["incident_summary"] = summary + '||' + incident_url_prefix + ep.incident2.incident_id; + policy_status["incident_state"] = ep.incident2.state; + policy_status["incident2_violation_data_count"] = ep.incident2.violation_data_count; + policy_status["incident2_last_update"] = ep.incident2.updated_at; + } + + grid_and_cwf.grid.push(policy_status); + } + + for (x=0; x -1) { + _.each(incident["violation_data"], function(violation) { + violation["incident_id"] = incident["id"]; + result.push(violation); + }); + } + }); +EOS +end + + +# Escalation for Downsize ElastiCache Nodes +escalation "esc_resize_clusters" do + automatic false # Do not automatically action from meta parent. the child will handle automatic escalations if param is set + label "Downsize ElastiCache Nodes" + description "Approval to downsize nodes in all selected ElastiCache clusters" + + # Run declaration should go at end, after any parameters that may exist + run "esc_resize_clusters", data, rs_governance_host, rs_project_id +end +define esc_resize_clusters($data, $governance_host, $rs_project_id) do + $actions_options = [] + call child_run_action($data, $governance_host, $rs_project_id, "Downsize ElastiCache Nodes", $action_options) +end + + +# Summary and a conditional incident which will show up if any policy is being applied, updated or deleted. +# Minimum of 1 incident, max of four +# Could swap the summary to only showing running +# Could also just have one incident and use meta_status to determine which escalation happens +policy "policy_scheduled_report" do + # Consolidated Incident Check(s) + # Consolidated incident for AWS Underutilized ElastiCache Clusters Found + validate $ds_underutilized_elasticache_clusters_combined_incidents do + summary_template "Consolidated Incident: {{ len data }} AWS Underutilized ElastiCache Clusters Found" + escalate $esc_email + escalate $esc_resize_clusters + check eq(size(data), 0) + export do + resource_level true + field "accountID" do + label "Account ID" + end + field "accountName" do + label "Account Name" + end + field "resourceID" do + label "Cluster ID" + end + field "region" do + label "Cluster Region" + end + field "tags" do + label "Cluster Tags" + end + field "numberOfNodes" do + label "Nodes (#)" + end + field "resourceType" do + label "Node Type" + end + field "newResourceType" do + label "Recommended Node Type" + end + field "recommendationDetails" do + label "Recommendation" + end + field "savings" do + label "Estimated Monthly Savings" + end + field "savingsCurrency" do + label "Savings Currency" + end + field "cpuMaximum" do + label "CPU Maximum %" + end + field "cpuMinimum" do + label "CPU Minimum %" + end + field "cpuAverage" do + label "CPU Average %" + end + field "cpuP99" do + label "CPU p99" + end + field "cpuP95" do + label "CPU p95" + end + field "cpuP90" do + label "CPU p90" + end + field "memMaximum" do + label "Freeable Memory Maximum %" + end + field "memMinimum" do + label "Freeable Memory Minimum %" + end + field "memAverage" do + label "Freeable Memory Average %" + end + field "memP99" do + label "Freeable Memory p99" + end + field "memP95" do + label "Freeable Memory p95" + end + field "memP90" do + label "Freeable Memory p90" + end + field "status" do + label "Cluster Status" + end + field "engine" do + label "Engine" + end + field "engineVersion" do + label "Engine Version" + end + field "service" do + label "Service" + end + field "resourceARN" do + label "Resource ARN" + end + field "resourceName" do + label "Resource Name" + end + field "id" do + label "ID" + end + field "incident_id" do + label "Child Incident ID" + end + end + end + + # Status Incident Check + validate $ds_take_in_parameters do + summary_template "{{ data.parent_policy.name }}: Status of Child Policies" + detail_template <<-EOS +The current status of Child Policies for **{{ data.parent_policy.name }}**: + +Total Child Applied Policies: {{ len data.grid }} +EOS + check false # always trigger this status incident + export "grid" do + resource_level true + field "id" do + label "Applied Policy ID" + end + field "policy_name" do + label "Applied Policy Name" + format "link-external" + end + field "meta_policy_status" do + label "Meta Child Policy Status" + end + field "policy_status" do + label "Policy Status" + end + field "policy_last_update" do + label "Policy Last Update" + end + field "incident_summary" do + label "Incident Summary" + format "link-external" + end + field "incident_state" do + label "Incident State" + end + field "incident_violation_data_count" do + label "Incident Violation Count" + end + field "incident_last_update" do + label "Incident Last Update" + end + field "incident2_summary" do + label "Incident 2 Summary" + format "link-external" + end + field "incident2_state" do + label "Incident 2 State" + end + field "incident2_violation_data_count" do + label "Incident 2 Violation Count" + end + field "incident2_last_update" do + label "Incident 2 Last Update" + end + end + end + + # Create Child Policies Incident Check + validate $ds_to_create do + summary_template "Policies being created" + detail_template <<-EOS + Policies Being Created: + + | Applied Policy | + | --------------- | + {{ range data -}} + | {{ .name }} | + {{ end -}} + EOS + escalate $create_policies + check eq(size(data),0) + end + + # Update Child Policies Incident Check + validate $ds_to_update do + summary_template "Policies being updated" + detail_template <<-EOS + Policies Being Updated: + + | Applied Policy | + | --------------- | + {{ range data -}} + | {{ .name }} | + {{ end -}} + EOS + escalate $update_policies + check eq(size(data),0) + end + + # Delete Child Policies Incident Check + validate $ds_to_delete do + summary_template "Policies being deleted" + detail_template <<-EOS + Policies being Deleted: + + | Applied Policy | + | --------------- | + {{ range data -}} + | {{ .name }} | + {{ end -}} + EOS + escalate $delete_policies + check eq(size(data),0) + end +end + +# Begin Shared Functions for Child Actions from Consolidated Incident +define groupByIncidentID($data) return $incidents do + # Empty hash to store incidents is incident_id + $incidents = {} + + task_label("Grouping items by Incident ID") + $index = 1 + foreach $item in $data do + task_label("Grouping items by Incident ID. "+to_s($index)+"/"+to_s(size($data))) + if !$incidents[$item["incident_id"]] + #task_label("Grouping items by Incident ID. "+to_s($index)+"/"+to_s(size($data))". New Incident: "+$item["incident_id"]) + $incidents[$item["incident_id"]] = {"id": $item["incident_id"], "resource_ids": []} + end + #task_label("Grouping items by Incident ID. "+to_s($index)+"/"+to_s(size($data))". Appending Resource: "+$item["id"]) + # Append resource id to the list for the incident + $incidents[$item["incident_id"]]["resource_ids"] = $incidents[$item["incident_id"]]["resource_ids"] + [$item["id"]] + end +end + +define child_run_action($data, $governance_host, $rs_project_id, $action_label, $action_options) do + # Empty global array for log strings, helpful for debugging + $$debug = [] + + # Group Resources by Incident ID + # This reduces the number of requests made to the Flexera API + call groupByIncidentID($data) retrieve $incidents + $$debug_incidents = to_json($incidents) + + call runActions($incidents, $action_label, $governance_host, $rs_project_id, $action_options) + + # If we encountered any errors, use `raise` to mark the CWF process as errored + if inspect($$errors) != "null" + raise join($$errors,"\n") + end + + # If we made it here, all actions completed successfully + # Celebrate Success! + task_label("All \""+$action_label+"\" actions completed successfully!") +end + +define runActions($incidents, $action_label, $governance_host, $rs_project_id, $action_options) do + foreach $id in keys($incidents) do + sub on_error: handle_error() do + $incident = $incidents[$id] + task_label("Triggering action \""+$action_label+"\" on "+size($incident["resource_ids"])+" count resources via incident "+$incident["id"]) + $request = { + auth: $$auth_flexera, + verb: "get", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/incidents/", $incident["id"]]), + headers: { "Api-Version": "1.0" }, + query_strings: { "view": "extended" } + } + $response = http_request($request) + $$debug << to_json({ + "request": $request, + "response": $response + }) + $action_id = "" + foreach $action in $response["body"]["available_actions"] do + # If we have not already found the action id, and the label matches, set the action id + # The first check is to prevent looking through the entire list if we already have the id + if $action["label"] == $action_label + $action_id = $action["id"] + end + end + if $action_id == "" + raise "Could not find action id for \""+$action_label+"\" response="+to_json($response) + end + # Now we are reach to trigger the action + $request = { + auth: $$auth_flexera, + verb: "post", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/incidents/", $incident["id"],"/actions/", $action_id,"/run_action"]), + headers: { "Api-Version": "1.0" }, + body: { "options":[{ "name": "ids", "value": $incident["resource_ids"] }] } + } + # If the action has parameters, add them to the request body + if type($action_options) == "array" && size($action_options) > 0 + $request["body"]["options"] = $request["body"]["options"] + $action_options + end + $response = http_request($request) + $$debug << to_json({ + "request": $request, + "response": $response + }) + # Get the action status from response header + $action_location = $response["headers"]["Location"] + + # Setup some variables for the wait loop + $action_status = "" + $loop_count = 0 + $loop_endtime = now() + (3600*2) # 2 hours from now + # [ queued, aborted, pending, running, completed, failed, denied ] + while ($action_status !~ /^(aborted|completed|failed|denied)/) && (now() <= $loop_endtime) do + # Using Loop Count to slowly increment the sleep time + # This is to prevent the loop from hammering our APIs + $loop_count = $loop_count + 1 + task_label("action_status=\""+$action_status+"\" Sleeping for "+to_s($loop_count)+" seconds") + sleep($loop_count) + task_label("action_status=\""+$action_status+"\" Getting action status") + $request = { + auth: $$auth_flexera, + verb: "get", + https: true, + host: $governance_host, + href: $action_location, + headers: { "Api-Version": "1.0" }, + query_strings: { "view": "extended" } + } + $response = http_request($request) + $$debug << to_json({ + "request": $request, + "response": $response + }) + $action_status = $response["body"]["status"] + end + if ($action_status != "completed") + # Check if we are out of time first + if (now() > $loop_endtime) + raise "action_status=\""+$action_status+"\" Action did not complete in time. Aborting to prevent endless loop. action_status_json="+to_json($response) + else + # If not, then it was aborted, failed or denied + raise "action_status=\""+$action_status+"\" Action did not complete as expected. action_status_json="+to_json($response) + end + end + # If we made it here, the action completed successfully + task_label("action_status=\""+$action_status+"\" Action completed successfully") + end + end +end +# End Shared Functions for Child Actions from Consolidated Incident + +# CWF function to handle errors +define handle_error() do + if !$$errors + $$errors = [] + end + $$errors << $_error["type"] + ": " + $_error["message"] + # We check for errors at the end, and raise them all together + # Skip errors handled by this definition + $_error_behavior = "skip" +end + +# Used only for emailing the combined child incident if so desired +escalation "esc_email" do + automatic true + label "Send Email" + description "Send incident email" + email $param_combined_incident_email +end + +escalation "create_policies" do + run "create_applied_policies", data, rs_governance_host, rs_project_id +end + +# if name !=null +define create_applied_policies($data, $governance_host, $rs_project_id) return $responses do + $responses = [] + $$debug = [] + $item_index = 0 + $item_total = size($data) + foreach $item in $data do + $item_index = $item_index + 1 + $status = to_s("("+$item_index+"/"+$item_total+")") + task_label($status+" Creating Applied Policy with Options: " + to_json($item["options"])) + $response = http_request( + auth: $$auth_flexera, + verb: "post", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/applied_policies"]), + headers: { "Api-Version": "1.0" }, + body: { + "name": $item["name"], + "description": $item["description"], + "template_href": $item["template_href"], + "frequency": $item["frequency"], + "options": $item["options"], + "credentials": $item["credentials"], + "meta_parent_policy_id": $item["meta_parent_policy_id"] + } + ) + $responses << $response + $$debug << to_json({ + "response": $response, + "item": $item, + "governance_host": $governance_host + }) + end +end + +escalation "update_policies" do + run "update_applied_policies", data, rs_governance_host, rs_project_id +end + +define update_applied_policies($data, $governance_host, $rs_project_id) return $responses do + $responses = [] + $$debug = [] + $item_index = 0 + $item_total = size($data) + foreach $item in $data do + $item_index = $item_index + 1 + $status = to_s("("+$item_index+"/"+$item_total+")") + task_label($status+" Updating Applied Policy with Options: " + to_json($item["options"])) + $response = http_request( + auth: $$auth_flexera, + verb: "patch", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/applied_policies/", $item["applied_policy_id"]]), + headers: { "Api-Version": "1.0" }, + body: { + "options": $item["options"] + } + ) + $responses << $response + $$debug << to_json({ + "response": $response, + "item": $item, + "governance_host": $governance_host + }) + end +end + +escalation "delete_policies" do + run "delete_applied_policies", data, rs_governance_host, rs_project_id +end + +define delete_applied_policies($data, $governance_host, $rs_project_id) return $responses do + $responses = [] + $$debug = [] + $item_index = 0 + $item_total = size($data) + foreach $item in $data do + $item_index = $item_index + 1 + $status = to_s("("+$item_index+"/"+$item_total+")") + task_label($status+" Deleting Applied Policy: " + $item["id"]) + $response = http_request( + auth: $$auth_flexera, + verb: "delete", + https: true, + host: $governance_host, + href: join(["/api/governance/projects/", $rs_project_id, "/applied_policies/", $item["id"]]), + headers: { "Api-Version": "1.0" } + ) + $responses << $response + $$debug << to_json({ + "response": $response, + "item": $item, + "governance_host": $governance_host + }) + end +end diff --git a/data/policy_permissions_list/master_policy_permissions_list.json b/data/policy_permissions_list/master_policy_permissions_list.json index 2d4209aa7a..5d1d500d8a 100644 --- a/data/policy_permissions_list/master_policy_permissions_list.json +++ b/data/policy_permissions_list/master_policy_permissions_list.json @@ -2230,6 +2230,59 @@ } ] }, + { + "id": "./cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt", + "name": "AWS Rightsize ElastiCache", + "version": "0.1.0", + "providers": [ + { + "name": "aws", + "permissions": [ + { + "name": "sts:GetCallerIdentity", + "read_only": true, + "required": true + }, + { + "name": "cloudwatch:GetMetricData", + "read_only": true, + "required": true + }, + { + "name": "ec2:DescribeRegions", + "read_only": true, + "required": true + }, + { + "name": "elasticache:DescribeCacheClusters", + "read_only": true, + "required": true + }, + { + "name": "elasticache:ListTagsForResource", + "read_only": true, + "required": true + }, + { + "name": "elasticache:ModifyCacheCluster", + "read_only": false, + "required": false, + "description": "Only required for taking action (terminating or downsizing); the policy will still function in a read-only capacity without these permissions." + } + ] + }, + { + "name": "flexera", + "permissions": [ + { + "name": "billing_center_viewer", + "read_only": true, + "required": true + } + ] + } + ] + }, { "id": "./cost/aws/rightsize_rds_instances/aws_rightsize_rds_instances.pt", "name": "AWS Rightsize RDS Instances", diff --git a/data/policy_permissions_list/master_policy_permissions_list.yaml b/data/policy_permissions_list/master_policy_permissions_list.yaml index a646e71ec1..de2fea3f0e 100644 --- a/data/policy_permissions_list/master_policy_permissions_list.yaml +++ b/data/policy_permissions_list/master_policy_permissions_list.yaml @@ -1280,6 +1280,37 @@ - name: billing_center_viewer read_only: true required: true +- id: "./cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt" + name: AWS Rightsize ElastiCache + version: 0.1.0 + :providers: + - :name: aws + :permissions: + - name: sts:GetCallerIdentity + read_only: true + required: true + - name: cloudwatch:GetMetricData + read_only: true + required: true + - name: ec2:DescribeRegions + read_only: true + required: true + - name: elasticache:DescribeCacheClusters + read_only: true + required: true + - name: elasticache:ListTagsForResource + read_only: true + required: true + - name: elasticache:ModifyCacheCluster + read_only: false + required: false + description: Only required for taking action (terminating or downsizing); the + policy will still function in a read-only capacity without these permissions. + - :name: flexera + :permissions: + - name: billing_center_viewer + read_only: true + required: true - id: "./cost/aws/rightsize_rds_instances/aws_rightsize_rds_instances.pt" name: AWS Rightsize RDS Instances version: 5.4.1 diff --git a/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb b/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb index 8b184983ec..b04f901d58 100644 --- a/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb +++ b/tools/meta_parent_policy_compiler/meta_parent_policy_compiler.rb @@ -26,6 +26,7 @@ "../../cost/aws/rightsize_rds_instances/aws_rightsize_rds_instances.pt", "../../cost/aws/rds_instance_license_info/rds_instance_license_info.pt", "../../cost/aws/rightsize_ec2_instances/aws_rightsize_ec2_instances.pt", + "../../cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt", "../../cost/aws/rightsize_redshift/aws_rightsize_redshift.pt", "../../cost/aws/s3_bucket_size/aws_bucket_size.pt", "../../cost/aws/s3_storage_policy/aws_s3_bucket_policy_check.pt", diff --git a/tools/policy_master_permission_generation/validated_policy_templates.yaml b/tools/policy_master_permission_generation/validated_policy_templates.yaml index fab421ed46..cd9b6e34d4 100644 --- a/tools/policy_master_permission_generation/validated_policy_templates.yaml +++ b/tools/policy_master_permission_generation/validated_policy_templates.yaml @@ -29,6 +29,7 @@ validated_policy_templates: - "./cost/aws/rds_instance_license_info/rds_instance_license_info.pt" - "./cost/aws/superseded_instances/aws_superseded_instances.pt" - "./cost/aws/rightsize_ec2_instances/aws_rightsize_ec2_instances.pt" +- "./cost/aws/rightsize_elasticache/aws_rightsize_elasticache.pt" - "./cost/aws/rightsize_rds_instances/aws_rightsize_rds_instances.pt" - "./cost/aws/rightsize_redshift/aws_rightsize_redshift.pt" - "./cost/aws/reserved_instances/coverage/reserved_instance_coverage.pt"