Merge pull request #127 from Cray-HPE/release/3.16.0

Release 3.16.0

.gitignore

@@ -7,4 +7,6 @@ myimage.yaml
 *.pyc
 .version
 .env
-.vscode
+.vscode
+.idea/*
+.idea/inspectionProfiles/*

CHANGELOG.md

@@ -5,7 +5,42 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [3.16.0] - 2024-06-26
+### Added
+- CASMCMS-8915: IMS API features for tagging built images.
+### Dependencies
+- CASMCMS-8022 - update python dependencies to most recent versions. 
+| Package | From | To |
+|-----------------------|------------|----------|
+| `aniso8601` | 3.0.2 | 9.0.1 |
+| `boto3` | 1.12.49 | 1.34.114 |
+| `botocore` | 1.15.49 | 1.34.114 |
+| `cachetools` | 3.0.0 | 5.3.3 |
+| `certifi` | 2019.11.28 | 2024.2.2 |
+| `chardet` | 3.0.4 | 5.2.0 |
+| `click` | 6.7 | 8.1.7 |
+| `docutils` | 0.14 | 0.21.2 |
+| `Flask` | 1.1.4 | 3.0.3 |
+| `flask-marshmallow` | 0.9.0 | 1.2.1 |
+| `google-auth` | 1.6.3 | 2.29.0 |
+| `gunicorn` | 19.10.0 | 22.0.0 |
+| `idna` | 2.8 | 3.7 |
+| `itsdangerous` | 0.24 | 2.2.0 |
+| `Jinja2` | 2.10.3 | 3.1.4 |
+| `jmespath` | 0.9.5 | 1.0.1 |
+| `MarkupSafe` | 1.1.1 | 2.1.5 |
+| `marshmallow` | 3.0.0b16 | 3.21.2 |
+| `oauthlib` | 2.1.0 | 3.2.2 |
+| `pyasn1` | 0.4.8 | 0.6.0 |
+| `pyasn1-modules` | 0.2.8 | 0.4.0 |
+| `pytz` | 2018.4 | 2024.1 |
+| `requests` | 2.23.0 | 2.31.0 |
+| `requests-oauthlib` | 1.0.0 | 1.3.1 |
+| `rsa` | 4.7.2 | 4.9 |
+| `s3transfer` | 0.3.7 | 0.10.1 |
+| `urllib3` | 1.25.11 | 1.26.18 |
+| `websocket-client` | 0.54.0 | 1.8.0 |
+| `Werkzeug` | 0.15.6 | 3.0.3 |
 
 ## [3.15.0] - 2024-05-20
 ### Dependencies

README.md

@@ -180,7 +180,7 @@ before trying to build:
 ```
 $ docker login artifactory.algol60.net
 ```
-See more information on authetication here:
+See more information on authentication here:
 https://rndwiki-pro.its.hpecorp.net/display/CSMTemp/Client+Authentication#ClientAuthentication-SecurityConsiderations
 
 ## Running Locally
@@ -206,16 +206,47 @@ This will start the IMS server on `http://localhost:9100`. An S3 instance is
 required for the IMS server to do anything meaningful. See the [Configuration Options](#Configuration-Options)
 section for more information and further configuration possibilities.
 
+Fetch recipes and images:
+
 ```
 $ curl http://127.0.0.1:9100/images
 []
 $ curl http://127.0.0.1:9100/recipes
 []
 ```
 
+Add a public key:
+
+```
+$ curl http://127.0.0.1:9100/public-keys -X POST -H "Content-Type: application/json" \
+ --data '{"name":"test","public_key":"TEST_KEY_DATA"}'
+```
+
+Get a recipe description:
+
+```
+$ curl http://127.0.0.1:9100/recipes/RECIPE_ID
+```
+
+Patch an image record:
+
+```
+curl http://127.0.0.1:9100/images/IMAGE_ID -X PATCH -H "Content-Type: application/json" \
+ --data '{"platform":"aarch64"}'
+```
+
+Create a job:
+
+```
+curl http://127.0.0.1:9100/jobs -X POST -H "Content-Type: application/json" \
+ --data '{ "job_type":"create","require_dkms":"False","image_root_archive_name":"Test","artifact_id":"RECIPE_ID", \
+ "public_key_id":"PUBLIC_KEY_ID"}'
+```
+
 **NOTE:** To successfully post jobs to the IMS Jobs endpoint, you must be running under 
 kubernetes as the IMS Service tries to launch a new K8S job. This is not expected to 
-work when running the IMS Service locally under Docker.
+work when running the IMS Service locally under Docker. However running locally you
+can watch the logs and verify the request posts correctly.
 
 ## Testing
 
@@ -228,6 +259,47 @@ $ ./runCodeStyleCheck.sh
 $ ./runLint.sh
 ```
 
+### Setting up PyTest
+
+PyTest runs the unit tests directly, not inside a container. This means that it needs
+some additional changes to get the correct configuration to run locally.
+
+The following are required beyond the development system setup:
+* The Python requirements in requirements-test.txt
+
+1. Activate the virtual environment created for development
+
+ ```
+ $ . .env/bin/activate
+ ```
+
+1. Install required python modules
+
+ ```
+ $ pip3 install -r requirements-test.txt
+ ```
+
+1. Create pytest.ini file with env vars
+
+ In the parent directory that IMS is cloned into, create the file 'pytest.ini'
+ and put the following contents in it:
+
+ ```
+ [pytest]
+ env =
+ FLASK_ENV=development 
+ ```
+
+1. Create a directory for the IMS data files
+
+ This is the directory that the unit tests will use to create the
+ data file for IMS object records.
+
+ ```
+ cd ~
+ mkdir -p ims/data
+ ```
+
 ### CT Tests
 
 See cms-tools repo for details on running CT tests for this service.

api/openapi.yaml

@@ -1,7 +1,7 @@
 #
 # MIT License
 #
-# (C) Copyright 2021-2023 Hewlett Packard Enterprise Development LP
+# (C) Copyright 2021-2024 Hewlett Packard Enterprise Development LP
 #
 # Permission is hereby granted, free of charge, to any person obtaining a
 # copy of this software and associated documentation files (the "Software"),
@@ -2245,6 +2245,55 @@ components:
  type: array
  items:
  $ref: '#/components/schemas/RecipeKeyValuePair'
+ ImageMetadataAnnotationKeyValuePair:
+ description: Key/value pair used to further define an Image
+ type: object
+ required:
+ - key
+ - value
+ properties:
+ key:
+ description: Template variable to associate with the IMS image
+ example: includes_additional_packages
+ type: string
+ value:
+ description: Value variable to associate with the IMS image
+ example: "foo,bar,baz"
+ type: string
+ ImagePatchRecord:
+ description: Values used to update an existing IMS Image Record
+ type: object
+ properties:
+ link:
+ $ref: '#/components/schemas/ArtifactLinkRecord'
+ arch:
+ description: Target architecture for the recipe.
+ example: aarch64
+ enum:
+ - aarch64
+ - x86_64
+ type: string
+ metadata:
+ description: An object which indicates a number of annotation patch operations relating to image tags.
+ type: object
+ required:
+ - operation
+ - key
+ properties:
+ operation:
+ description: How to update a given key within the context of a patch operation
+ type: string
+ enum:
+ - set
+ - remove
+ key:
+ description: The key to update for a given image
+ type: string
+ example: includes_additional_packages
+ value:
+ description: The value to associate with a key during a patch operation
+ type: string
+ example: "vim,emacs,man"
  ImageRecord:
  description: An Image Record
  type: object
@@ -2277,6 +2326,11 @@ components:
  - aarch64
  - x86_64
  type: string
+ metadata:
+ description: An object of key/value associated with an Image
+ type: object
+ properties:
+ $ref: '#/components/schemas/ImageMetadataAnnotationKeyValuePair'
  DeletedImageRecord:
  description: A Deleted Image Record
  type: object
@@ -2316,19 +2370,11 @@ components:
  - aarch64
  - x86_64
  type: string
- ImagePatchRecord:
- description: Values to update an ImageRecord with
- type: object
- properties:
- link:
- $ref: '#/components/schemas/ArtifactLinkRecord'
- arch:
- description: Target architecture for the recipe.
- example: aarch64
- enum:
- - aarch64
- - x86_64
- type: string
+ metadata:
+ description: List of key/value pairs to associate with an Image
+ type: object
+ properties:
+ $ref: '#/components/schemas/ImageMetadataAnnotationKeyValuePair'
  JobRecord:
  description: A Job Record
  type: object

constraints.txt

@@ -1,37 +1,38 @@
-aniso8601==3.0.2
-boto3==1.12.49
-botocore==1.15.49
-cachetools==3.0.0
-certifi==2019.11.28
-chardet==3.0.4
-click==6.7
-docutils==0.14
+# Constraints updated 5/29/2024
+aniso8601==9.0.1
+boto3==1.34.114
+botocore==1.34.114
+cachetools==5.3.3
+certifi==2024.2.2
+chardet==5.2.0
+click==8.1.7
+docutils==0.21.2
 fabric==3.2.2
-Flask==1.1.4
-flask-marshmallow==0.9.0
+Flask==3.0.3
+flask-marshmallow==1.2.1
 Flask-RESTful==0.3.10
-google-auth==1.6.3
-gunicorn==19.10.0
+google-auth==2.29.0
+gunicorn==22.0.0
 httpproblem==0.2.0
-idna==2.8
-itsdangerous==0.24
-Jinja2==2.10.3
-jmespath==0.9.5
+idna==3.7
+itsdangerous==2.2.0
+Jinja2==3.1.4
+jmespath==1.0.1
 # CSM 1.6 uses Kubernetes 1.22, so use client v22.x to ensure compatability
 kubernetes==22.6.0
-MarkupSafe==1.1.1
-marshmallow==3.0.0b16
-oauthlib==2.1.0
-pyasn1==0.4.8
-pyasn1-modules==0.2.8
-pytest==8.1.1
+MarkupSafe==2.1.5
+marshmallow==3.21.2
+oauthlib==3.2.2
+pyasn1==0.6.0 # most recent: 1.6.1, pyasn1-modules 0.4.0 requires <0.7.0
+pyasn1-modules==0.4.0
 python-dateutil==2.8.2
-pytz==2018.4
+pytest==8.1.1
+pytz==2024.1
 PyYAML==6.0.1
-requests==2.23.0
-requests-oauthlib==1.0.0
-rsa==4.7.2
-s3transfer==0.3.7
-urllib3==1.25.11
-websocket-client==0.54.0
-Werkzeug==0.15.6
+requests==2.31.0
+requests-oauthlib==1.3.1
+rsa==4.9
+s3transfer==0.10.1
+urllib3==1.26.18 # most recent 2.2.1, botocore 1.34.114 requires <1.27.0
+websocket-client==1.8.0
+Werkzeug==3.0.3

requirements-test.txt

@@ -5,6 +5,7 @@ testtools
 fixtures
 pytest
 pytest-cov
+pytest-env
 pycodestyle
 pylint
 mock

src/server/__init__.py

@@ -29,6 +29,7 @@
 # TODO CASMCMS-1154 Get a real data store
 import os
 import os.path
+from marshmallow import EXCLUDE
 
 class DataStoreHACK(collections.abc.MutableMapping):
  """ A dictionary that reads/writes to a file """
@@ -51,7 +52,7 @@ def _read(self):
  # Setting 'unknown="Exclude" allows downgrades by just dropping any data
  # fields that are no longer part of the current schema.
  with open(self.store_file, 'r') as data_file:
- obj_data = self.schema.loads(data_file.read(), many=True, unknown="EXCLUDE")
+ obj_data = self.schema.loads(data_file.read(), many=True, unknown=EXCLUDE)
  self.store = {str(getattr(obj, self.key_field)): obj for obj in obj_data}
 
  def _write(self):

src/server/models/__init__.py

@@ -29,9 +29,10 @@
 
 class ArtifactLink(Schema):
  """ A schema specifically for validating artifact links """
- path = fields.Str(required=True, description="URL or path to the artifact",
- validate=Length(min=1, error="name field must not be blank"))
- etag = fields.Str(required=False, default="", description="Artifact entity tag")
+ path = fields.Str(required=True, validate=Length(min=1, error="name field must not be blank"),
+ metadata={"metadata": {"description": "URL or path to the artifact"}})
+ etag = fields.Str(required=False, dump_default="", load_default="",
+ metadata={"metadata": {"description": "Artifact entity tag"}})
  type = fields.Str(required=True, allow_none=False,
- description="The type of artifact link",
+ metadata={"metadata": {"description": "The type of artifact link"}},
  validate=OneOf(ARTIFACT_LINK_TYPES, error="Type must be one of: {choices}."))