Version 5.0.3 (#56)

* Version bump and update AMIs to be worldwide Signed-off-by: Irving Popovetsky <[email protected]> * Reduce unhealthy threshold from 10 minutes to 2 and double the check frequency Signed-off-by: Irving Popovetsky <[email protected]> * upload files to S3 bucket on tags automatically. Update python version to latest. test marketplace.yaml Signed-off-by: Irving Popovetsky <[email protected]> * speedup and smartify travis Signed-off-by: Irving Popovetsky <[email protected]> * Some JMESPath magic in order to ensure we get the latest AMI Signed-off-by: Irving Popovetsky <[email protected]> * Major reliability improvement: Make the Chef Server frontend ASG automatically rebuild instances that go unhealthy, but not bootstrap instances. Add cloudwatch alerts for all unhealthy hosts. Add clarity about the email address to the PREREQS.md doc Signed-off-by: Irving Popovetsky <[email protected]> * bump AMIs again to pick up Automate 20190722 release Signed-off-by: Irving Popovetsky <[email protected]>
chef-boneyard · Aug 1, 2019 · a0c503a · a0c503a
1 parent f638394
commit a0c503a
Show file tree

Hide file tree

Showing 12 changed files with 200 additions and 50 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -1,17 +1,16 @@
 language: python
+cache: pip
 python:
-- '3.6'
+- '3.7'
 install:
 - pip install awscli cfn-lint
 script:
-- cfn-lint -f json *.yaml
-- aws cloudformation validate-template --template-body file://main.yaml
-- aws cloudformation validate-template --template-body file://chef_elasticsearch.yaml
-- aws cloudformation validate-template --template-body file://chef_rds.yaml
-- aws cloudformation validate-template --template-body file://automate.yaml
-- aws cloudformation validate-template --template-body file://chef_server_ha.yaml
-- aws cloudformation validate-template --template-body file://chef_server_ops_dashboard.yaml
-- aws cloudformation validate-template --template-body file://supermarket.yaml
+- cfn-lint -I -f parseable *.yaml
+- |
+  for template in $(ls *.yaml); do
+    echo "Validating template: ${template}"
+    aws cloudformation validate-template --template-body file://${template} > /dev/null
+  done
 env:
   global:
   - AWS_DEFAULT_REGION=us-east-1
@@ -20,8 +19,13 @@ env:
   # AWS_SECRET_ACCESS_KEY
   - secure: UgSuLr08ONqGJ5W5BLfVGC47P8AHh8/4u89GQEaR36Z1UdEo2e1buFOdMjZuBccsidB721LRCgn4yn+QafswTD7PvF7K/KoybUmseu3yR8LTf1HXInvYhqDnzI463i79B9A99GHFfjl/gZe4rNWYjCaJwIrAPkpsc11YOZU59BErL8RVwP8mCLXVlH4bEMUQFXHIddmDBBVV4M7+Dc1UBWc+ntHJPPoMpuzYJl7Lq32c5JPKs9Q3eEJuSnS3mAKCISobAG84Ck2VvcCxMfYJ/OARMl5eNmBZkLTAV3alSlrhhbwRzpAKc4YnmaWz6WJ49tNhZSwABPl+Om6LYiMVJg2a35Ocg7ru/5nz5VhaNUPCQeSOMmV2cM49MIdL4ZH/1hoB4StST4/vjcWQ5+HSIQEuEpEJx43J+oklIu8mgac9TpblgeoI50kryadFwgTqiAqMepsEvG9JKTIgcTOhb1ijAl+IVoKWLns04t2+oHuTFbqUrC6IlEi+5nATXUng65e1yiySZVgWuLwQyLa1Sajmgi9TiAKiUlTxt2eIzT7ok5SRSJxFZX3yGKXSw1SYi0knazQ2BUvcWsDbwLrp/SkQkzDh9ALjdT2027vtdIJviR+nyfzpKgx2PAmY5/JhzGK5pEZtE25XmOs1QLLC/XVKeDOp9t7Atgpl0QVW5x8=
 deploy:
-  provider: script
+- provider: script
   script: aws s3 sync . s3://aws-native-chef-server/ --exclude "*" --include "*.yaml" --include "files/*" --acl public-read
   skip_cleanup: true
   on:
     branch: master
+- provider: script
+  script: aws s3 sync . s3://aws-native-chef-server/${TRAVIS_TAG#v}/ --exclude "*" --include "*.yaml" --include "files/*" --acl public-read
+  skip_cleanup: true
+  on:
+    tags: true
diff --git a/PREREQS.md b/PREREQS.md
@@ -14,20 +14,27 @@ You must create or upload at least one SSL certficate to AWS Certificate Manager
 In the parameters, fill in the `ChefSSLCertificateARN`, `AutomateSSLCertificateARN` and `SupermarketSSLCertificateARN` values with the ARNs for the certificates. The ARNs are always viewable in the ACM console, an example ACM ARN looks like: `arn:aws:acm:us-west-2:446539779517:certificate/82d30a13-b420-4f43-80de-9e7872f70b96`
 
 
-## Network
+### Network
 
 You must already have a VPC setup properly before continuing setting up the stack, it should;
 
 * Have enough IP's available to assign to nodes
 * Be split up into 3 subnets, each in different Availability Zones (AZ's)
 * Provide the VPC ID and associated subnets to the `VPC` and `ServerSubnets` parameters
 
-## Security
+### Contact Email Adress
+
+Supply a valid email address or team/DL alias here.  Cloudwatch alerts will be sent to this address!
+
+* At stack launch time you will receive an email titled `AWS Notification - Subscription Confirmation`
+  * It's important you click the `Confirm subscription` button in this email or else you will not receive notifications
+
+### Security
 
 * You should already have created/uploaded an SSH key to AWS. Provide the keypair name to the `KeyName` parameter
 * If you're using the `main.yaml` stack, you must also create a security group in the referenced VPC to define your administrative access.  Provide sg ID to the `InboundAdminSecurityGroupId` parameter
 
-## Amazon Elasticsearch and Service Linked Role (SLR)
+### Amazon Elasticsearch and Service Linked Role (SLR)
 
 Amazon Elasticsearch requires a specific SLR to be created prior to running this CloudFormation template, specifically one called `AWSServiceRoleForAmazonElasticsearchService`. This role cannot be created programmatically as it is created automatically when setting up a VPC access domain in the AWS console. For more information on this [please see this doc from AWS](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/slr-es.html), at this time, even though the doc points to a way to create this manually via the CLI, it still only works via console setup, however AWS adds new features all the time, and by the time you do this, it may work programmatically, please follow their documentation. Once you've created the VPC access domain for AWS ElasticSearch, you can then delete this domain, the role will still be there and you should be able to continue.
 

diff --git a/README.md b/README.md
@@ -24,8 +24,6 @@ A complete Chef Stack including:
 ## Requirements
 * A working knowledge and comfort level with CloudFormation so that you can read and understand this template for your self
 * Permissions to create all of the types of resources specified in this template (IAM roles, Database subnets, etc)
-* A valid SSL certificate ARN (from the AWS Certificate Manager service)
-* A Route53 hosted zone (optional but strongly recommended)
 
 ## Prerequisites
 
@@ -39,8 +37,8 @@ You can launch this stack with the push of a button:
 
 | Edition | Version | Est Monthly cost | Launch button |
 | ------- | ------- | ---------------- | ------------- |
-| High Performance | 5.0.2 | [$1500*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=calc-E9251374-D80C-45B7-BED7-C9C9778B1D6C) | <p><a href="https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https:%2F%2Fs3.amazonaws.com%2Faws-native-chef-server%2F5.0.2%2Fmain.yaml&amp;stackName=my-chef-stack" target="_blank"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack" /></a></p> |
-| Lowest Cost | 5.0.2 | [$320*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=files/calc-165638db8e331664846c0d6654d743377bd3eac6&v=ver20190604sQ) | <p><a href="https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https:%2F%2Fs3.amazonaws.com%2Faws-native-chef-server%2F5.0.2%2Fmarketplace.yaml&amp;stackName=my-chef-stack" target="_blank"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack" /></a></p> |
+| High Performance | 5.0.3 | [$1500*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=calc-E9251374-D80C-45B7-BED7-C9C9778B1D6C) | <p><a href="https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https:%2F%2Fs3.amazonaws.com%2Faws-native-chef-server%2F5.0.3%2Fmain.yaml&amp;stackName=my-chef-stack" target="_blank"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack" /></a></p> |
+| Lowest Cost | 5.0.3 | [$320*](https://calculator.s3.amazonaws.com/index.html#r=IAD&s=EC2&key=files/calc-165638db8e331664846c0d6654d743377bd3eac6&v=ver20190604sQ) | <p><a href="https://console.aws.amazon.com/cloudformation/home#/stacks/new?templateURL=https:%2F%2Fs3.amazonaws.com%2Faws-native-chef-server%2F5.0.3%2Fmarketplace.yaml&amp;stackName=my-chef-stack" target="_blank"><img src="https://s3.amazonaws.com/cloudformation-examples/cloudformation-launch-stack.png" alt="Launch Stack" /></a></p> |
 
 _*NOTE: Estimated costs are just that, your mileage will vary. Use reserved instances to greatly reduce costs for long-running clusters._
 
@@ -50,7 +48,7 @@ However, the most repeatable and least error-prone way to launch this stack is t
 ```bash
 MYBUCKET=aws-native-chef-server
 MYID=mycompany
-VERSION=5.0.2
+VERSION=5.0.3
 EDITION=main  # main or marketplace
 # Configure the automate_stack_parameters.json and then launch the cloudformation stack:
 # If you're using your own bucket, uncomment the next line:
@@ -71,7 +69,7 @@ If you've made changes to the template content or parameters and you wish to upd
 ```bash
 MYBUCKET=aws-native-chef-server
 MYID=mycompany
-VERSION=5.0.2
+VERSION=5.0.3
 EDITION=main  # main or marketplace
 # If you're using your own bucket, uncomment the next line:
 # aws s3 sync . s3://$MYBUCKET/$VERSION/ --exclude "*" --include "*.yaml" --include "files/*" && \

diff --git a/automate.yaml b/automate.yaml
@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: '2010-09-09'
-Description: AWS Native Automate 2 (auto-recovering standalone) v5.0.2
+Description: AWS Native Automate 2 (auto-recovering standalone) v5.0.3
 
 Parameters:
   # Required Parameters
@@ -139,8 +139,9 @@ Resources:
   ChefTargetGroup:
       Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
       Properties:
-        HealthCheckIntervalSeconds: 60
-        UnhealthyThresholdCount: 10
+        HealthCheckIntervalSeconds: 30
+        UnhealthyThresholdCount: 2
+        HealthyThresholdCount: 5
         HealthCheckPath: /
         VpcId: !Ref VPC
         Port: 443
@@ -537,14 +538,33 @@ Resources:
       MetricName: HTTPCode_ELB_5XX_Count
       Namespace: AWS/ApplicationELB
       ComparisonOperator: GreaterThanOrEqualToThreshold
-      EvaluationPeriods: 5
+      EvaluationPeriods: 2
       Period: 60
       Statistic: Sum
       Threshold: 10
       Dimensions:
         - Name: LoadBalancer
           Value: !GetAtt ChefALB.LoadBalancerFullName
 
+  TargetGroupUnHealthyHostsAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: !Sub Automate-TG-Unhealthy-Hosts-${AWS::StackName}
+      AlarmDescription: Alarms when UnHealthyHostCount is greater than threshold.
+      AlarmActions: [!Ref AlertNotificationTopic]
+      MetricName: UnHealthyHostCount
+      Namespace: AWS/ApplicationELB
+      ComparisonOperator: GreaterThanOrEqualToThreshold
+      EvaluationPeriods: 2
+      Period: 60
+      Statistic: Sum
+      Threshold: 1
+      Dimensions:
+        - Name: LoadBalancer
+          Value: !GetAtt ChefALB.LoadBalancerFullName
+        - Name: TargetGroup
+          Value: !GetAtt ChefTargetGroup.TargetGroupFullName
+
 Outputs:
   DNSName:
     Description: The DNS name of the load balancer

diff --git a/chef_elasticsearch.yaml b/chef_elasticsearch.yaml
@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: '2010-09-09'
-Description: Chef Server Elasticsearch 5.0.2
+Description: Chef Server Elasticsearch 5.0.3
 
 Parameters:
   # Required Parameters

diff --git a/chef_rds.yaml b/chef_rds.yaml
@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: '2010-09-09'
-Description: Chef Server PostgreSQL v5.0.2
+Description: Chef Server PostgreSQL v5.0.3
 
 Parameters:
   # Required Parameters

diff --git a/chef_server_ha.yaml b/chef_server_ha.yaml
@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: '2010-09-09'
-Description: AWS Native Chef Server (auto-scaling HA) v5.0.2
+Description: AWS Native Chef Server (auto-scaling HA) v5.0.3
 
 Parameters:
   # Required Parameters
@@ -185,7 +185,7 @@ Parameters:
     Type: String
   TemplateVersion:
     Type: String
-    Default: "5.0.2"
+    Default: "5.0.3"
 
 Metadata:
   cfn-lint:
@@ -246,6 +246,8 @@ Resources:
       - !Ref ChefTargetGroup
       MaxSize: !Sub '${MaxFrontendInstances}'
       MinSize: !Sub '${MinFrontendInstances}'
+      HealthCheckGracePeriod: 900
+      HealthCheckType: ELB
       NotificationConfigurations:
       - TopicARN: !Ref AlertNotificationTopic
         NotificationTypes:
@@ -365,8 +367,9 @@ Resources:
   ChefTargetGroup:
       Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
       Properties:
-        HealthCheckIntervalSeconds: 60
-        UnhealthyThresholdCount: 10
+        HealthCheckIntervalSeconds: 30
+        UnhealthyThresholdCount: 2
+        HealthyThresholdCount: 5
         HealthCheckPath: /_status
         VpcId: !Ref VPC
         Port: 443
@@ -385,8 +388,9 @@ Resources:
   ChefBootstrapTargetGroup:
       Type: "AWS::ElasticLoadBalancingV2::TargetGroup"
       Properties:
-        HealthCheckIntervalSeconds: 60
-        UnhealthyThresholdCount: 10
+        HealthCheckIntervalSeconds: 30
+        UnhealthyThresholdCount: 2
+        HealthyThresholdCount: 5
         HealthCheckPath: /_status
         VpcId: !Ref VPC
         Port: 443
@@ -840,14 +844,52 @@ Resources:
       MetricName: HTTPCode_ELB_5XX_Count
       Namespace: AWS/ApplicationELB
       ComparisonOperator: GreaterThanOrEqualToThreshold
-      EvaluationPeriods: 5
+      EvaluationPeriods: 2
       Period: 60
       Statistic: Sum
       Threshold: 10
       Dimensions:
         - Name: LoadBalancer
           Value: !GetAtt ChefALB.LoadBalancerFullName
 
+  BootstrapTGUnhealthyHosts:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: !Sub Chef-Bootstrap-TG-Unhealthy-Hosts-${AWS::StackName}
+      AlarmDescription: Alarms when UnHealthyHostCount is greater than threshold.
+      AlarmActions: [!Ref AlertNotificationTopic]
+      MetricName: UnHealthyHostCount
+      Namespace: AWS/ApplicationELB
+      ComparisonOperator: GreaterThanOrEqualToThreshold
+      EvaluationPeriods: 2
+      Period: 60
+      Statistic: Sum
+      Threshold: 1
+      Dimensions:
+        - Name: LoadBalancer
+          Value: !GetAtt ChefALB.LoadBalancerFullName
+        - Name: TargetGroup
+          Value: !GetAtt ChefBootstrapTargetGroup.TargetGroupFullName
+
+  FETGUnhealthyHosts:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmName: !Sub Chef-FE-TG-Unhealthy-Hosts-${AWS::StackName}
+      AlarmDescription: Alarms when UnHealthyHostCount is greater than threshold.
+      AlarmActions: [!Ref AlertNotificationTopic]
+      MetricName: UnHealthyHostCount
+      Namespace: AWS/ApplicationELB
+      ComparisonOperator: GreaterThanOrEqualToThreshold
+      EvaluationPeriods: 2
+      Period: 60
+      Statistic: Sum
+      Threshold: 1
+      Dimensions:
+        - Name: LoadBalancer
+          Value: !GetAtt ChefALB.LoadBalancerFullName
+        - Name: TargetGroup
+          Value: !GetAtt ChefTargetGroup.TargetGroupFullName
+
   CPUAlarmHigh:
     Type: AWS::CloudWatch::Alarm
     Properties:

diff --git a/chef_server_ops_dashboard.yaml b/chef_server_ops_dashboard.yaml
@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: '2010-09-09'
-Description: AWS Native OpsDashboard for Chef Server v5.0.2
+Description: AWS Native OpsDashboard for Chef Server v5.0.3
 
 Parameters:
   # Required Parameters

diff --git a/generate_os_mappings.sh b/generate_os_mappings.sh
@@ -1,7 +1,7 @@
 #!/bin/bash
 
 CENTOS_RELEASE='aws-native-chef-server-5.0.*'
-IMAGE_OWNERID="406084061336"
+IMAGE_OWNERID="446539779517"
 
 printf "Mappings:\n  AMI:\n"
 
@@ -10,7 +10,7 @@ for region in $regions; do
   centos_ami=$(aws --region $region ec2 describe-images \
   --owners $IMAGE_OWNERID \
   --filters "Name=name,Values=${CENTOS_RELEASE}" \
-  --query "Images[0].ImageId" \
+  --query "sort_by(Images, &CreationDate)[*].ImageId | [-1]" \
   --output "text")
 
   printf "    $region:\n      centos: $centos_ami\n"

diff --git a/main.yaml b/main.yaml
@@ -1,5 +1,5 @@
 AWSTemplateFormatVersion: '2010-09-09'
-Description: AWS Native Chef Stack v5.0.2
+Description: AWS Native Chef Stack v5.0.3
 
 Parameters:
   # Required Parameters
@@ -43,8 +43,9 @@ Parameters:
     MaxLength: 64
     ConstraintDescription: Must be 64 alphanumeric characters
   AutomateSSLCertificateARN:
-    Description: SSL Certficate ARN for SSL Certficate (must begin with arn:aws:iam::)
+    Description: ARN for SSL Certficate, pre-create this in AWS Certificate Manager
     Type: String
+    AllowedPattern: "arn:aws:acm:.*"
   AutomateDnsRecordName:
     Description: The DNS A-record name to automatically create in the Route53 zone (if enabled)
     Type: String
@@ -65,8 +66,9 @@ Parameters:
   ###############################################################################
   # Chef Server Stack Settings
   ChefSSLCertificateARN:
-    Description: SSL Certficate ARN for SSL Certficate (must begin with arn:aws:iam::)
+    Description: ARN for SSL Certficate, pre-create this in AWS Certificate Manager
     Type: String
+    AllowedPattern: "arn:aws:acm:.*"
   ChefServerDnsRecordName:
     Description: The DNS A-record name to automatically create in the Route53 zone (if enabled)
     Type: String
@@ -137,8 +139,9 @@ Parameters:
   ###############################################################################
   # Automate Stack Settings
   SupermarketSSLCertificateARN:
-    Description: SSL Certficate ARN for SSL Certficate (must begin with arn:aws:iam::)
+    Description: ARN for SSL Certficate, pre-create this in AWS Certificate Manager
     Type: String
+    AllowedPattern: "arn:aws:acm:.*"
   SupermarketDnsRecordName:
     Description: The DNS A-record name to automatically create in the Route53 zone (if enabled)
     Type: String
@@ -191,7 +194,7 @@ Parameters:
   TemplateVersion:
     Description: The version of this template to use (do not change this unless directed by support)
     Type: String
-    Default: "5.0.2"
+    Default: "5.0.3"
   Route53HostedZone:
     Type: String
     Default: ''
@@ -271,10 +274,38 @@ Conditions:
 #        on AMIs that are pre-baked with all needed packages and scripts.
 Mappings:
   AMI:
+    eu-north-1:
+      centos: ami-0768c8ad843f26cbb
+    ap-south-1:
+      centos: ami-098dad5e1e61c8693
+    eu-west-3:
+      centos: ami-01e9cfbd4a3d29f04
+    eu-west-2:
+      centos: ami-0b107e0d9819e9282
+    eu-west-1:
+      centos: ami-02fccd4149ee6792a
+    ap-northeast-2:
+      centos: ami-03a176d0e576e5d36
+    ap-northeast-1:
+      centos: ami-0c6a211faced0a6b9
+    sa-east-1:
+      centos: ami-0b36d7af9bd43808a
+    ca-central-1:
+      centos: ami-0699076e5a3573ba0
+    ap-southeast-1:
+      centos: ami-03118dc2d58fe9031
+    ap-southeast-2:
+      centos: ami-0979abde4cc2545d3
+    eu-central-1:
+      centos: ami-0b6717e5a3e152845
     us-east-1:
-      centos: ami-08d306e6de29d5cc1
+      centos: ami-0f7f03419a9d3acfe
+    us-east-2:
+      centos: ami-0c31bba08ffd888ca
+    us-west-1:
+      centos: ami-0d3a8f9b7321fd2bb
     us-west-2:
-      centos: ami-06fbb1e0ba7c99cfd
+      centos: ami-0cc579c65b77193d4
 
 Resources:
   ChefRole: