From 028d422455321d2c1d8596d62da5547d090bd535 Mon Sep 17 00:00:00 2001 From: Vinicius Canaa Medeiros Ruela Date: Thu, 12 Sep 2019 11:27:25 -0700 Subject: [PATCH] Documentation for verification system. (#401) --- docs/verification-integration.md | 108 +++++++++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 docs/verification-integration.md diff --git a/docs/verification-integration.md b/docs/verification-integration.md new file mode 100644 index 00000000..4fa07ae0 --- /dev/null +++ b/docs/verification-integration.md @@ -0,0 +1,108 @@ +# Verification system + +Applications are executed in our Verification system to ensure that: + +1. Installation succeeds: All resources are applied and waited for to become healthy. +1. Functionality tests pass: The deployer starts the Tester Pod and watches its exit status. Zero means +success, non-zero means failure. +1. Uninstallation succeeds: Application and all its resources are successfully removed from the cluster. + +Successful results are required before application can be published to the GCP Marketplace. + +## Functionality tests + +Packaging functionality tests involves the following steps. + +### 1. Create a `/data-test` folder + +The structure is similar to the one in `/data` folder, but with the configuration and manifests that +are only meant to be applied when the application is installed in verification mode. Some examples of +`/data-test` for different deployers: + +* Helm based deployer: + + ``` + data-test/ + chart/ + nginx.tar.gz + schema.yaml + ``` + +* Kubectl based deployer: + + ``` + data-test/ + manifest/ + tester.yaml.template + schema.yaml + ``` + +### 3. Writing the test schema.yaml + +The `/data-test/schema.yaml` file should be used to: + +* Declare properties needed for test, which are not already declared in `/data/schema.yaml`. One +common example is the image for the Tester Pod. +* Declare default values for the properties in `/data/schema.yaml` that do not include default values. +The Verification system will use the default values to run the application. Refer to [mpdev](https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/master/docs/mpdev-references.md#smoke-test-an-application) +for more details. + +### 4. Include a Pod manifest to execute the tests + +The deployer starts the Tester Pod after all the Kubernetes resources are healthy. It should run all +tests that assert the functionality of the application. If all tests pass, the Tester Pod must complete +with a zero exit status. Otherwise, it must complete with a non-zero exit status. + +The Tester Pod needs to be annotated with `marketplace.cloud.google.com/verification:test`, so the Verification +can detect which Pod to watch for test results. + +Helm based deployers can alternatively use `helm.sh/hook:test-success`, to keep compatibility with +existing helm tests. Our tools will replace the helm annotation with the verification annotation during +deployment time. + + +### 5. (Optional) Add or modify resources as needed + +The `/data-test` folder is a way of including additional Kubernetes resources for verification purposes, +or even modifying existing ones. The configuration defined in `/data-test` overrides the default configuration +in `/data` according to the following rules: + ++ Helm `values.yaml` files are merged. Values defined in `/data-test` + override the values in `/data`. ++ Schema properties are merged. Schema entries defined in `properties` from + `/data-test/schema.yaml` will override entries defined in `properties` + from `/data/schema.yaml` ++ All other files in `/data-test` will override files in `/data`. In case of helm deployer, which + contains a .tar.gz file, the files are extracted first and then replaced. + +You can also refer to [examples](https://github.com/GoogleCloudPlatform/click-to-deploy/tree/master/k8s) of open source apps packaged by GCP Marketplace. + +## Running Verification locally + +The verification triggered locally is the same run in our Verification system. It should be run prior +to submitting the application, so mistakes can be caught sooner. Use the [mpdev](https://github.com/GoogleCloudPlatform/marketplace-k8s-app-tools/blob/master/docs/mpdev-references.md#smoke-test-an-application) command to run verification: + +``` +mpdev verify \ + --deployer= +``` + +Make sure to include both the image name and the sha256/tag of the image, ex: + +``` +--deployer=gcr.io/my-project/my-app/deployer@sha256:123456789abcdef123456789abcdef123456789abcdef123456789abcdef1234 +``` +or +``` +--deployer=gcr.io/my-project/my-app/deployer:1.0 +``` + +In summary, this is what happens: + +1. Folder `/data-test` is overlayed on to `/data`, to prepare the resources such as the Tester Pod. +1. A test namespace is created. +1. Manifests for the app is applied in the test namespace. +1. Status on Kubernetes resources are checked to ensure that the app is ready for testing. +1. Tester Pods are started. If a Tester Pod fails, the deployer Job fails. +1. Application and its resources are removed. +1. Test namespace is deleted.