API Beta Release (#106)

* cloudfront and certificate manager
* added list layers
* refactored common code
* fixed LastEvaluatedKey bug
* added api endpoints
* added aws-requests-auth
* invalidate cache
* updated diagram

README.MD

@@ -1,6 +1,6 @@
 # Keith's Layers (Klayers)
 
-🐍 A collection of AWS Lambda Layers for Python 🐍
+🐍 A collection of AWS Lambda(λ) Layers for Python 🐍
 
 [![Python 3.7](https://img.shields.io/badge/python-3.7-green.svg)](https://www.python.org/downloads/release/python-375/)  [![Python 3.8](https://img.shields.io/badge/python-3.8-green.svg)](https://www.python.org/downloads/release/python-380/)
 
@@ -59,6 +59,10 @@ Click links below for your preferred version of python, and then select your reg
 
 * [Arns](deployments/python3.8/arns)
 
+Once selected, you can add the arn directly from the console, by selecting Layers->Add a Layer->Specify an Arn:
+
+![Screenshot](documentation/add_arn.png)
+
 ### Option 2: Download copy of layer
 
 Use the `Get Layer Version by ARN` in [python](https://boto3.amazonaws.com/v1/documentatio/api/latest/reference/services/lambda.html#Lambda.Client.get_layer_version_by_arn) or [aws-cli](https://docs.aws.amazon.com/cli/latest/reference/lambda/get-layer-version-by-arn.html) command which will provide an S3 location to download the layer as a zip.
@@ -106,6 +110,44 @@ Special hand-crafted binaries for layers. These layers are not automatically bui
 
 ![Screenshot](documentation/Klayers-Architecture.png)
 
+## API (beta)
+
+We've recently added an API under beta. All API calls are http-based, and work only with https (TLS1.2 and above). The API is heavily cached, so there could be minor delays in updates.
+
+### Get latest ARN for specific package in region
+
+Returns data on the latest layer for a specific *{package}* in a specific *{region}*
+
+*https://api.klayers.cloud/api/v1/layers/latest/{region}/{package}*
+
+example:
+
+  * https://api.klayers.cloud/api/v1/layers/latest/us-east-1/requests
+  * https://api.klayers.cloud/api/v1/layers/latest/ap-southeast-1/boto3
+
+
+### Get all ARNs for specific package in region
+
+Returns data on the all layers (latest and deprecated) for a specific *{package}* in a specific *{region}*
+
+*https://api.klayers.cloud/api/v1/layers/{region}/{package}*
+
+
+example:
+
+  * https://api.klayers.cloud/api/v1/layers/us-east-1/requests
+  * https://api.klayers.cloud/api/v1/layers/ap-southeast-1/boto3
+
+### Get list of all built packages
+
+Returns list of all packages currently being built, including dependencies and build date.
+
+*https://api.klayers.cloud/api/v1/builds/latest*
+
+example:
+
+  * https://api.klayers.cloud/api/v1/builds/latest
+
 ## Special Thanks
 
 * [Chahna107](https://github.com/chahna107) for adding tesseract config files into the tesseract layer.

pipeline/Serverless/01_invoke_pipelines/invalidate_cf_cache.py

@@ -0,0 +1,35 @@
+import boto3
+import os
+import uuid
+
+from lambda_cache import ssm
+
+from aws_lambda_powertools.logging import Logger
+
+logger = Logger()
+
+
+@logger.inject_lambda_context
+@ssm.cache(parameter=os.environ["DISTRIBUTION_NAME"])
+def main(event, context):
+    distribution_id = getattr(context, "id")
+    CallerReference = str(uuid.uuid4())
+
+    client = boto3.client("cloudfront")
+    response = client.create_invalidation(
+        DistributionId=distribution_id,
+        InvalidationBatch={
+            "Paths": {"Quantity": 1, "Items": ["/api/v1/*",]},
+            "CallerReference": CallerReference,
+        },
+    )
+
+    logger.info(
+        {
+            "CallerReference": CallerReference,
+            "message": f"Invalidated Cache for {distribution_id}",
+            "invalidation_location": response["Location"],
+        }
+    )
+
+    return response["Location"]

pipeline/Serverless/01_invoke_pipelines/invoke_pipelines.py

@@ -6,7 +6,7 @@
 
 logger = Logger()
 
-import get_config
+from common.get_config import get_packages
 
 
 def log_eventbridge_errors(response, function_logger):
@@ -36,7 +36,7 @@ def main(event, context):
       response: Entries in EventBridge for processing
     """
 
-    packages = get_config.get_packages()
+    packages = get_packages()
     client = boto3.client("events")
     stage = os.environ["STAGE"]
 

pipeline/Serverless/01_invoke_pipelines/invoke_pipelines.yml

@@ -22,7 +22,7 @@ invoke_pipeline:
       Action: s3:GetObject
       Resource: ${self:custom.s3LayersArn}/*
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
 
 
 download_config:
@@ -43,5 +43,40 @@ download_config:
         - ${self:custom.s3LayersArn}/config/*
   layers:
     - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-lambda-cache:1
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-requests:5
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-requests:5
+
+invalidate_cache:
+  handler: 01_invoke_pipelines/invalidate_cf_cache.main
+  description: Invalidated CF cache after deployment
+  runtime: python3.8
+  memorySize: 256
+  timeout: 30
+  environment:
+    POWERTOOLS_SERVICE_NAME: Klayers.invalidate_cf_cache
+    # this parameter deployed as part of api Cloudformation
+    DISTRIBUTION_NAME: /${self:service.name}/${self:provider.stage}/api/cfDistribution/id
+  iamRoleStatementsName: ${self:provider.stage}-invalidate_cache
+  iamRoleStatements:
+    - Effect: Allow
+      Action:
+        - ssm:GetParameter
+      Resource:
+      - Fn::Join:
+        - ":"
+        - - arn:aws:ssm
+          - ${self:provider.region}
+          - Ref: AWS::AccountId
+          - parameter/${self:service.name}/${self:provider.stage}/api/cfDistribution/id
+    - Effect: Allow
+      Action: 
+        - cloudfront:CreateInvalidation
+      Resource:
+      - Fn::Join:
+        - ":"
+        - - "arn:aws:cloudfront:"
+          - Ref: AWS::AccountId
+          - distribution/${ssm:/${self:service.name}/${self:provider.stage}/api/cfDistribution/id}
+  layers:
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-lambda-cache:1
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32

pipeline/Serverless/02_pipeline/build.py

@@ -53,6 +53,7 @@ def put_requirements_hash(package, version, requirements_txt, requirements_hash)
         "rqrmntsHsh": {"S": requirements_hash},
         "bltVrsn": {"S": new_version},
         "crtdDt": {"S": created_date},
+        "pckg": {"S": package},
     }
 
     # Insert new record

pipeline/Serverless/02_pipeline/deploy.py

@@ -10,7 +10,7 @@
 
 logger = Logger()
 
-import get_config
+import common.get_config as get_config
 
 
 def check_regions_to_deploy(package, requirements_hash, regions):

pipeline/Serverless/02_pipeline/pipeline.yml

@@ -7,7 +7,7 @@ check:
   layers:
     - arn:aws:lambda:${self:provider.region}:113088814899:layer:Klayers-python37-packaging:1
     - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-requests:5
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
 
 build:
   handler: 02_pipeline/build.main
@@ -35,7 +35,7 @@ build:
     - dynamodb:GetItem
     Resource: ${self:custom.dbArn}
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
 
 
 deploy:
@@ -75,4 +75,4 @@ deploy:
   layers:
     - arn:aws:lambda:${self:provider.region}:113088814899:layer:Klayers-python37-boto3:2
     - arn:aws:lambda:${self:provider.region}:113088814899:layer:Klayers-python37-packaging:1
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32

pipeline/Serverless/03_publish/publish.yml

@@ -35,7 +35,7 @@ publish_reqs:
       - dynamodb:Query
       Resource: ${self:custom.dbArn}
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
 
 publish_github:
   handler: 03_publish/publish_to_github.handler

pipeline/Serverless/03_publish/publish_arns.py

@@ -1,30 +1,16 @@
 import os
-import json
 import logging
-import decimal
 import csv
 from datetime import datetime
+from common.get_config import get_aws_regions
 
 from boto3.dynamodb.conditions import Key
 import boto3
 
-import get_config
-
 logger = logging.getLogger()
 logger.setLevel(logging.INFO)
 
 
-# Helper class to convert a DynamoDB item to JSON.
-class DecimalEncoder(json.JSONEncoder):
-    def default(self, o):
-        if isinstance(o, decimal.Decimal):
-            if o % 1 > 0:
-                return float(o)
-            else:
-                return int(o)
-        return super(DecimalEncoder, self).default(o)
-
-
 def convert_to_csv(items):
     """
     Args:
@@ -102,7 +88,7 @@ def main(event, context):
     Gets layer arns for each region and publish to S3
     """
 
-    regions = get_config.get_aws_regions()
+    regions = get_aws_regions()
 
     dynamodb = boto3.resource("dynamodb")
     table = dynamodb.Table(os.environ["DB_NAME"])

pipeline/Serverless/03_publish/publish_reqs.py

@@ -1,6 +1,4 @@
 import os
-import json
-import decimal
 from boto3.dynamodb.conditions import Key
 
 import boto3
@@ -12,16 +10,6 @@
 build_v0 = "bldVrsn0#"
 package_prefix = "pckg#"
 
-# Helper class to convert a DynamoDB item to JSON.
-class DecimalEncoder(json.JSONEncoder):
-    def default(self, o):
-        if isinstance(o, decimal.Decimal):
-            if o % 1 > 0:
-                return float(o)
-            else:
-                return int(o)
-        return super(DecimalEncoder, self).default(o)
-
 
 def query_requirements():
     """

pipeline/Serverless/04_stream_processor/streams.yml

@@ -25,7 +25,7 @@ stream_handler:
         arn: ${self:custom.dbStreamArn}
         batchSize: 10
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
 
 layer_delete:
   handler: 04_stream_processor/layer_delete.main
@@ -65,4 +65,4 @@ layer_delete:
             pk_type:
               - "lyr"
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32

pipeline/Serverless/05_slack_notification/slack.yml

@@ -20,8 +20,8 @@ slack_pipeline:
             - Ref: AWS::AccountId
             - parameter${self:custom.slackTokenParameter}
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:3
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:9
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
 
 slack_publish:
   handler: 05_slack_notification/slack_notification.slack_notification_publish
@@ -45,8 +45,8 @@ slack_publish:
             - Ref: AWS::AccountId
             - parameter${self:custom.slackTokenParameter}
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:3
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:9
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
 
 post_message_to_slack:
   handler: 05_slack_notification/slack_notification.post_message_to_slack
@@ -70,8 +70,8 @@ post_message_to_slack:
             - Ref: AWS::AccountId
             - parameter${self:custom.slackTokenParameter}
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:3
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:9
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
   events:
     - eventBridge:
         pattern:
@@ -100,5 +100,5 @@ slack_invokepipeline:
             - Ref: AWS::AccountId
             - parameter${self:custom.slackTokenParameter}
   layers:
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:3
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-slackclient:9
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32

pipeline/Serverless/06_webhook/webhook.yml

@@ -37,6 +37,6 @@ webhook_github:
           events:source: "github.webhook"
   layers:
     - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-lambda-cache:1
-    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:23
+    - arn:aws:lambda:${self:provider.region}:770693421928:layer:Klayers-python38-aws-lambda-powertools:32
   events:
     - httpApi: 'POST /api/v1/github_webhook'