diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..01111e4 --- /dev/null +++ b/.gitignore @@ -0,0 +1,7 @@ +.jekyll-metadata +Gemfile +Gemfile.lock +_config.yml +_site/ +*.pdf +*~ diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..261eeb9 --- /dev/null +++ b/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/README.md b/README.md new file mode 100644 index 0000000..d63be1b --- /dev/null +++ b/README.md @@ -0,0 +1,6 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2020 Intel Corporation +``` + +# Smart Edge Open Documentation diff --git a/application-onboarding/application-onboarding-cmdline.md b/application-onboarding/application-onboarding-cmdline.md new file mode 100644 index 0000000..b0b184a --- /dev/null +++ b/application-onboarding/application-onboarding-cmdline.md @@ -0,0 +1,230 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Onboarding Applications to an Intel® Smart Edge Open Cluster + +## Overview + +Intel® Smart Edge Open uses Helm Charts to onboard applications to edge node clusters. + +In this guide, you'll use the Helm command-line interface (CLI) to install an example application to an edge cluster. Once you've installed the application, you'll verify it's running on the cluster and then uninstall it. + +Our example application is an NGINX web server. You can use these instructions to onboard any application from a Helm Chart. + +These instructions apply to clusters created by deploying the [Developer Experience Kit](). + +### Requirements +#### Hardware +- An Intel® Smart Edge Open cluster that has been deployed by running the [Developer Experience Kit](). + +#### Software +The Helm v3 CLI must be installed on the system hosting the cluster. For instructions, see the [Helm documentation](https://helm.sh/docs/intro/install/) + +#### Knowledge +Basic knowlege of Kubernetes and Helm. +- For a list of commonly used `kubectl` commands and flags, see the [`kubectl` Cheat Sheet](https://kubernetes.io/docs/reference/kubectl/cheatsheet/) +- For more information on Helm commands, see the [Helm documentation](https://helm.sh/docs/) + +### Install the Application + +#### Prepare the System +##### Verify Pods are Running +Before onboarding an application, check the server deployment status and verify that all pods are running: + +```shell +$ kubectl get pods -A +NAMESPACE NAME READY STATUS RESTARTS AGE +harbor harbor-app-harbor-chartmuseum-7df97c958d-k4zvr 1/1 Running 0 48m +harbor harbor-app-harbor-clair-59c867fcb9-cmv42 2/2 Running 1 48m +harbor harbor-app-harbor-core-74457d7998-87x7c 1/1 Running 1 48m +harbor harbor-app-harbor-database-0 1/1 Running 0 48m +harbor harbor-app-harbor-jobservice-6c9ffddc7c-f275d 1/1 Running 3 48m +harbor harbor-app-harbor-nginx-57dff4794c-8rkqp 1/1 Running 0 48m +harbor harbor-app-harbor-notary-server-6f5fc7645b-bjj57 1/1 Running 3 48m +harbor harbor-app-harbor-notary-signer-55886d599d-lc5hp 1/1 Running 6 48m +harbor harbor-app-harbor-portal-6f6b56bd68-nh65f 1/1 Running 0 48m +harbor harbor-app-harbor-redis-0 1/1 Running 0 48m +harbor harbor-app-harbor-registry-7c956f55b5-5s2rf 2/2 Running 0 48m +harbor harbor-app-harbor-trivy-0 1/1 Running 0 48m +kube-system calico-kube-controllers-cf4844b67-ltt6q 1/1 Running 0 48m +kube-system calico-node-wm7d2 1/1 Running 0 48m +kube-system coredns-558bd4d5db-mt47m 1/1 Running 0 48m +kube-system coredns-558bd4d5db-vzwlg 1/1 Running 0 48m +kube-system etcd-puzz2 1/1 Running 0 48m +kube-system kube-apiserver-puzz2 1/1 Running 0 48m +kube-system kube-controller-manager-puzz2 1/1 Running 0 48m +kube-system kube-multus-ds-amd64-qt6p6 1/1 Running 0 32m +kube-system kube-proxy-kplxg 1/1 Running 0 48m +kube-system kube-scheduler-puzz2 1/1 Running 0 48m +kubernetes-dashboard dashboard-metrics-scraper-665b55f67d-bgpdt 1/1 Running 0 5m49s +kubernetes-dashboard kubernetes-dashboard-5b5bd8dd4c-6l8zl 1/1 Running 0 5m49s +smartedge-system nfd-release-node-feature-discovery-master-894c57d89-2n9bz 1/1 Running 0 14m +smartedge-system nfd-release-node-feature-discovery-worker-5vmm7 1/1 Running 0 14m +sriov-network-operator network-resources-injector-vtkpd 1/1 Running 0 6m5s +sriov-network-operator operator-webhook-9w9t4 1/1 Running 0 6m5s +sriov-network-operator sriov-network-config-daemon-v99rw 0/3 ContainerCreating 0 5m57s +sriov-network-operator sriov-network-operator-6898664bff-jzld7 1/1 Running 0 7m48s +telemetry cadvisor-92xpr 2/2 Running 0 30m +telemetry collectd-xgcfq 3/3 Running 0 30m +telemetry grafana-86b89d6bcc-k9bj6 3/3 Running 0 28m +telemetry prometheus-node-exporter-jtsrr 1/1 Running 3 30m +telemetry prometheus-server-b5c7b657d-zx2pn 3/3 Running 0 30m +telemetry statsd-exporter-86586dbf79-sqf56 2/2 Running 0 30m +telemetry telemetry-node-certs-jjt6f 1/1 Running 0 30m + +``` + + +##### Add the Repository +Log into the server. + +```shell +ssh smartedge-open@ +``` +Add the NGINX Helm repository. This will download the Helm Chart from the registry to deploy the example application. +> NOTE: To install an application using a Helm Chart that is not available from the registry, you can untar the chart's tarball locally to use for application deployment. + +```shell +$ helm repo add nginx https://helm.nginx.com/stable + +# Typical stdout +NAME CHART VERSION APP VERSION DESCRIPTION +nginx/nginx-ingress 0.10.0 1.12.0 NGINX Ingress Controller +stable/nginx-ingress 1.41.3 v0.34.1 DEPRECATED! An nginx Ingress controller that us... +stable/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego +``` + +##### Install the Helm Chart + +Use `helm search` to find charts in the repository you added. In the example below, we're searching for the chart called `nginx-ingress`. + +```shell +$ helm search repo nginx-ingress + +# Typical stdout +NAME CHART VERSION APP VERSION DESCRIPTION +nginx/nginx-ingress 0.10.0 1.12.0 NGINX Ingress Controller +stable/nginx-ingress 1.41.3 v0.34.1 DEPRECATED! An nginx Ingress controller that us... +stable/nginx-lego 0.3.1 Chart for nginx-ingress-controller and kube-lego +``` + + +Use `helm install` to install the chart. In the example below, we set the chart's release name to `my-ingress-controller`. Once we set it, we'll be able refer to the chart by its release name at the CLI. + +```shell +$ helm install my-ingress-controller nginx/nginx-ingress + +#Typical stdout +NAME: my-ingress-controller +LAST DEPLOYED: Tue Sep 7 10:39:24 2021 +NAMESPACE: default +STATUS: deployed +REVISION: 1 +TEST SUITE: None +NOTES: +The NGINX Ingress Controller has been installed. +``` +### Verify the Installation Status +**Option 1:** Use the `helm status` command to check the status of the application installation. From this point on, we refer to the chart by its release name, `my-ingress-controller`. + +```shell +$ helm status my-ingress-controller + +#Typical stdout +NAME: my-ingress-controller +LAST DEPLOYED: Tue Sep 7 10:39:24 2021 +NAMESPACE: default +STATUS: deployed +REVISION: 1 +TEST SUITE: None +NOTES: +The NGINX Ingress Controller has been installed. +``` +**Option 2:** Alternatively you can use the `kubectl` command to check the deployment of the ingress controller in the default namespace. + +```shell +$ kubectl get deployments + +#Typical stdout +NAME READY UP-TO-DATE AVAILABLE AGE +my-ingress-controller-nginx-ingress 0/1 1 0 7s +``` + +### Uninstall the Application + +To remove the application, use the `helm uninstall` command. + +```shell +$ helm uninstall my-ingress-controller + +# Typical stdout +release "my-ingress-controller" uninstalled +``` + +### Cleanup +The uninstall steps above will perform the cleanup of the application. If you do not require the Docker container images, you can remove them now. + +Below is the command to check if a container image still exists: + +```shell +$ docker images | grep nginx-ingress + +# Typical stdout +nginx/nginx-ingress 1.12.0 411b997b75dd 2 months ago 175MB +docker images | grep nginx-ingress + +docker rmi -f + +# Typical stdout +Untagged: nginx/nginx-ingress:1.12.0 +Untagged: nginx/nginx-ingress@sha256:a8a89dea468022c4d6e38c2ddaec649fb2bcd4bdda1ecbcd9d08cc846d7df483 +Deleted: sha256:411b997b75ddb834af0c5b8b3a71298df624a9136f7b51e77adfed68c20b3884 +Deleted: sha256:de053c75b648cc064ef402f739ce77c5fa6c70643c409290c5a09f4ebb6c6da6 +Deleted: sha256:2b87bb2095fa239af067093cf8a7911ff42762a670a6f593af136626586598d8 +Deleted: sha256:277bbd45d69b05bc14190d4c426bf7ecea9c61cca3fff92d56e7f1f63dc2f4d6 +Deleted: sha256:be1bd68b26a410f136652f2347ed04cf1f38631881b7bbe1a04d5ddf7e589262 +Deleted: sha256:2855bbcefcf95050e64049447e99e77efa2bff32374e586982d69be4612467ce +Deleted: sha256:bad169ad8b30eab551acbb8cd8fbdcd824528189e3dd0cc52dd88a37bbf121cd +Deleted: sha256:36d83ebf5fec7ae1be4c431f0945f2dbe6828ecdc936c604daa48f17c0b50ed7 +Deleted: sha256:b4c9a251dc81d52dd1cca9b4c69ca9e4db602a9a7974019f212846577f739699 +Deleted: sha256:038ca5b801cea48e9f40f6ffb4cda61a2fe0b6b0f378a7434a0d39d2575a4082 +Deleted: sha256:764055ebc9a7a290b64d17cf9ea550f1099c202d83795aa967428ebdf335c9f7 + +# Execute the command again to check if the containers are deleted +$ docker images | grep nginx-ingress + +#Typical output is - None +``` + +### Troubleshooting + + + +- Make sure you have an active internet connection during the full installation. If you lose Internet connectivity at any time, the installation might fail. + +- Make sure you are using a fresh installation of the Developer Experience Kit. Earlier software, especially Docker* and Docker Compose*, can cause issues. + +- If the server hosting the cluster accesses the Internet from behind a proxy, ensure the following proxy environment variables are set properly set: + +```shell +# ensure the correct proxy settings in enviroment file +vi /etc/environment + +http_proxy=http://: +https_proxy=http://: +ftp_proxy=http://: +``` +- If your pod reports a status of `CrashLoopBackOff`, check your Docker account plan. Docker Hub limits the number of pulls available to users with certain account types. If you are pull limited, consider upgrading your Docker account to increase the number of pulls you are allowed per day. + +```shell +docker login + +``` + +If you're unable to resolve your issues, you can post to the [Support Forum](). + +### Summary + +In this guide, you learned how to install and uninstall an application on a single-node cluster created by deploying the [Developer Experience Kit](). diff --git a/application-onboarding/images/README.md b/application-onboarding/images/README.md new file mode 100644 index 0000000..d63be1b --- /dev/null +++ b/application-onboarding/images/README.md @@ -0,0 +1,6 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2020 Intel Corporation +``` + +# Smart Edge Open Documentation diff --git a/components/networking/images/multus-pod-image.svg b/components/networking/images/multus-pod-image.svg new file mode 100644 index 0000000..263634c --- /dev/null +++ b/components/networking/images/multus-pod-image.svg @@ -0,0 +1,64 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/components/networking/multus.md b/components/networking/multus.md new file mode 100644 index 0000000..0c36eaf --- /dev/null +++ b/components/networking/multus.md @@ -0,0 +1,94 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Multus + +## Overview + +Edge deployments consist of both network functions and applications. Cloud-native solutions such as Kubernetes\* typically expose only one interface to the application or network function pods. These interfaces are typically bridged interfaces. This means that network functions like Base station or Core network User plane functions and applications such as CDN are limited by the default interface. +To address this, two key networking features must be enabled: +1) Enable a Kubernetes like orchestration environment to provision more than one interface to the application and network function pods. +2) Enable the allocation of dedicated hardware interfaces to application and network function pods. + +To enable multiple interface support in pods, Intel® Smart Edge Open uses the Multus\* container network interface. [Multus CNI](https://github.com/k8snetworkplumbingwg/multus-cni) is a container network interface (CNI) plugin for Kubernetes that enables the attachment of multiple network interfaces to pods. + +## How It Works + +Typically, in Kubernetes, each pod only has one network interface (apart from a loopback). With Multus, user can create a multi-homed pod that has multiple interfaces. To accomplish this, Multus acts as a “meta-plugin”, a CNI plugin that can call multiple other CNI plugins. The Multus CNI follows the Kubernetes Network Custom Resource Definition De-facto Standard to provide a standardized method by which to specify the configurations for additional network interfaces. This standard is put forward by the Kubernetes Network Plumbing Working Group. + +The figure below illustrates the network interfaces attached to a pod, as provisioned by the Multus CNI. The diagram shows the pod with three interfaces: eth0, net0, and net1. eth0 connects to the Kubernetes cluster network to connect with the Kubernetes server/services (kubernetes api-server, kubelet, etc.). net0 and net1 are additional network attachments and they connect to other networks by using other CNI plugins (e.g., vlan/vxlan/ptp). + +![Multus overview](images/multus-pod-image.svg) + +_Figure - Multus Overview_ + +## How To + +### Create and list `NetworkAttachmentDefinition` + +The following example creates a `NetworkAttachmentDefinition` that can be used to provide an additional macvlan interface to a pod: + +```bash +cat <**NOTE**: More networks can be added after a comma in the same annotation. + +### Verify that the additional interface is configured + +Run `ip a` in the deployed pod. The output should look similar to the following: + +```bash + 1: lo: mtu 65536 qdisc noqueue state UNKNOWN qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever + 2: net1@if178: mtu 1500 qdisc noqueue state LOWERLAYERDOWN + link/ether 06:3d:10:e3:34:a4 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 192.168.1.200/24 scope global net1 + valid_lft forever preferred_lft forever + 308: eth0@if309: mtu 1400 qdisc noqueue state UP + link/ether 0a:00:00:10:00:12 brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.245.0.17/16 brd 10.245.255.255 scope global eth0 + valid_lft forever preferred_lft forever +``` diff --git a/components/networking/sriov-network-operator.md b/components/networking/sriov-network-operator.md new file mode 100644 index 0000000..e5631af --- /dev/null +++ b/components/networking/sriov-network-operator.md @@ -0,0 +1,156 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2020 Intel Corporation +``` +# SR-IOV Network Operator + +- [SR-IOV Network Operator](#sr-iov-network-operator) + - [Overview](#overview) + - [Overview of SR-IOV CNI](#overview-of-sr-iov-cni) + - [Overview of SR-IOV Device Plugin](#overview-of-sr-iov-device-plugin) + - [Overview of SR-IOV Network Operator](#overview-of-sr-iov-network-operator) + - [SR-IOV Network Operator configuration and usage](#sr-iov-network-operator-configuration-and-usage) + - [Configuration](#configuration) + - [SR-IOV Network Node Policy](#sr-iov-network-node-policy) + - [SR-IOV Network](#sr-iov-network) + - [Usage](#usage) + - [Limitations](#limitations) + - [Reference](#reference) + +## Overview +Edge deployments consist of both network functions and applications. Cloud-native solutions such as Kubernetes* typically expose only one interface to the application or network function pods. These interfaces are typically bridged interfaces. This means that network functions like Base station or Core network User plane functions and applications such as CDN are limited by the default interface. To address this, two key networking features must be enabled: + +1. Enable a Kubernetes like orchestration environment to provision more than one interface to the application and network function pods. +2. Enable the allocation of dedicated hardware interfaces to application and network function pods. + +### Overview of SR-IOV CNI +The Single Root I/O Virtualization (SR-IOV) feature provides the ability to partition a single physical PCI resource into virtual PCI functions that can be allocated to application and network function pods. The SR-IOV CNI plugin enables the Kubernetes pod to be attached directly to an SR-IOV virtual function (VF) using the standard SR-IOV VF driver in the container host’s kernel. + +### Overview of SR-IOV Device Plugin +The Intel SR-IOV Network device plugin discovers and exposes SR-IOV network resources as consumable extended resources in Kubernetes. This works with SR-IOV VFs in both Kernel drivers and DPDK drivers. When a VF is attached with a kernel driver, the SR-IOV CNI plugin can be used to configure this VF in the pod. When using the DPDK driver, a VNF application configures this VF as required. + +### Overview of SR-IOV Network Operator +To enable SR-IOV device resource allocation and CNI, Intel® Smart Edge Open uses the SR-IOV Network Operator which wraps SR-IOV CNI and SR-IOV Device Plugin to one simple component. Operator uses custom resources like `SriovNetworkNodePolicy` and `SriovNetwork` to configure SR-IOV plugins and [Multus](./multus.md) CNI. + +## SR-IOV Network Operator configuration and usage + +To deploy the SR-IOV Network Operator to the Intel® Smart Edge Open cluster, `sriov_network_operator_enable: True` must be set in `inventory/default/group_vars/all/10-default.yml`. This component is enabled by default in [Developer Experience Kit](../../experience-kits/developer-experience-kit-open.md). This will perform Operator install and label every Edge Node as `sriov-operator-node=yes` which is required by config Daemon. It is deploying SR-IOV CNI plugin and SR-IOV Device plugin on labeled node when first `SriovNetworkNodePolicy` is applied. It will also automatically install Multus CNI. +SR-IOV Network Operator is deployed using Makefiles and it requires additional packages to be installed, which Intel® Smart Edge Opens provides with Operator deployment. +Images for Operator are downloaded from Openshift repository and stored in local registry. + +### Configuration + +SR-IOV Network Operator provides two custom resources to configure SR-IOV network devices and network attachments: + +#### SR-IOV Network Node Policy +Specifies SR-IOV network device configuration by e.g. creating VFs from given NIC interface/PCI address or its priority and defines Config Map for SR-IOV Device plugin. To apply resource to the Operator just simply use command: `kubectl apply -f sample-policy.yml`. +Sample Policy can look like: +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetworkNodePolicy +metadata: + name: sample-netdevice-policy + namespace: sriov-network-operator +spec: + nodeSelector: + feature.node.kubernetes.io/network-sriov.capable: "true" + resourceName: intel_sriov_netdevice + numVfs: 4 + nicSelector: + vendor: "8086" + deviceID: "159b" + rootDevices: ["0000:31:00.0"] +``` +> **NOTE:** After applying it, Operator will create 4 VFs from given Interface and create Config Map for SR-IOV Device plugin with defined `intel_sriov_netdevice` as allocable resource and mapped from given interface. + +> **NOTE:** If `SriovNetworkNodePolicy` should be applied only on specific node, `kubernetes.io/hostname: ""` should be added to `nodeSelector` field. + +#### SR-IOV Network +It defines and configures Network attachment for Multus/SR-IOV CNI. To apply resource to the Operator just simply use command: `kubectl apply -f sample-network.yml`. +Sample network can look like: +```yaml +apiVersion: sriovnetwork.openshift.io/v1 +kind: SriovNetwork +metadata: + name: sriov-seo + namespace: sriov-network-operator +spec: + resourceName: intel_sriov_netdevice + networkNamespace: default + ipam: |- + { + "type": "host-local", + "subnet": "192.168.2.0/24", + "routes": [{ + "dst": "0.0.0.0/0" + }], + "gateway": "192.168.2.1" + } +``` +> **NOTE:** After applying it, Operator will create Network Attachment Definition for Multus/SR-IOV CNI plugin with defined `sriov-seo` network, which applies to `intel_sriov_netdevice` allocable resource in `default` namespace. + +### Usage +To create a pod with an attached SR-IOV device, add the network annotation (`sriov-seo`) to the pod definition and request access to the SR-IOV capable device (`intel.com/intel_sriov_netdevice`): + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: samplepod + annotations: + k8s.v1.cni.cncf.io/networks: sriov-seo +spec: + containers: + - name: samplecent + image: centos/tools + resources: + requests: + intel.com/intel_sriov_netdevice: "1" + limits: + intel.com/intel_sriov_netdevice: "1" + command: ["sleep", "infinity"] +``` + +To verify that the additional interface was configured, run `ip a` in the deployed pod. The output should look similar to the following: + +```shell +1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 + link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 + inet 127.0.0.1/8 scope host lo + valid_lft forever preferred_lft forever +2: tunl0@NONE: mtu 1480 qdisc noop state DOWN group default qlen 1000 + link/ipip 0.0.0.0 brd 0.0.0.0 +4: eth0@if3991: mtu 1480 qdisc noqueue state UP group default + link/ether 36:5a:98:29:a3:8d brd ff:ff:ff:ff:ff:ff link-netnsid 0 + inet 10.245.136.180/32 brd 10.245.136.180 scope global eth0 + valid_lft forever preferred_lft forever +1872: net1: mtu 1500 qdisc mq state DOWN group default qlen 1000 + link/ether 1a:6d:18:3b:e9:9d brd ff:ff:ff:ff:ff:ff + inet 192.168.2.2/24 brd 192.168.2.255 scope global net1 + valid_lft forever preferred_lft forever +``` +> **NOTE**: Interface `net1` is SR-IOV net device. + +## Limitations +It was observed that on Ubuntu 20.04, configuring SR-IOV devices via `SriovNetworkNodePolicy` using `pfNames` on CLV NIC can cause some problems. To avoid this issue it is recommended to use `rootDevices`(see [example above](#sr-iov-network-node-policy)) instead of `pfNames`. To get PCI address of devices `lshw -c network -businfo` command can be used, i.g: +```bash +$ lshw -c network -businfo +Bus info Device Class Description +============================================================= +pci@0000:04:00.0 eno8303 network NetXtreme BCM5720 2-port Gigabit Ethernet PCIe +pci@0000:04:00.1 eno8403 network NetXtreme BCM5720 2-port Gigabit Ethernet PCIe +pci@0000:31:00.0 eno12399 network Intel Corporation +pci@0000:31:00.1 eno12409 network Intel Corporation +pci@0000:b1:00.0 ens5f0 network Intel Corporation +pci@0000:b1:00.1 ens5f1 network Intel Corporation +``` +There is also observed that on Ubuntu 20.04 with only one type of SR-IOV NIC which is CLV, there can be lack of label on the node: `feature.node.kubernetes.io/network-sriov.capable: "true"`. To properly apply CLV SR-IOV NIC configuration via `SriovNetworkNodePolicy` using `nodeSelector` use entries like `kubernetes.io/hostname: ""` (`` should be replaced by node hostname) or `sriov-network-operator-node: "yes"`. Using [example above](#sr-iov-network-node-policy) `feature.node.kubernetes.io/network-sriov.capable: "true"` can be replaced with given examples. + +## Reference +For further details: +- Multus: https://github.com/k8snetworkplumbingwg/multus-cni +- SR-IOV CNI: https://github.com/k8snetworkplumbingwg/sriov-cni +- SR-IOV Network Device Plugin: https://github.com/k8snetworkplumbingwg/sriov-network-device-plugin +- SR-IOV Network Operator: https://github.com/k8snetworkplumbingwg/sriov-network-operator +- SR-IOV network node policy [(`SriovNetworkNodePolicy`)](#sr-iov-network-node-policy): https://docs.openshift.com/container-platform/4.7/networking/hardware_networks/configuring-sriov-device.html +- Ethernet network device [(`SriovNetwork`)](#sr-iov-network): https://docs.openshift.com/container-platform/4.7/networking/hardware_networks/configuring-sriov-net-attach.html diff --git a/components/provisioning/provisioning.md b/components/provisioning/provisioning.md new file mode 100644 index 0000000..9e5f866 --- /dev/null +++ b/components/provisioning/provisioning.md @@ -0,0 +1,424 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Intel® Smart Edge Open Provisioning Process + +* [Overview](#overview) +* [Provisioning Process Scenarios](#provisioning-process-scenarios) + * [Default Provisioning Scenario](#default-provisioning-scenario) + * [Repository Cloning](#default-repository-cloning) + * [Configuration](#default-configuration) + * [Artifacts Building](#default-artifacts-building) + * [Services Start Up](#default-services-start-up) + * [Installation Media Flashing](#default-installation-media-flashing) + * [System Installation](#default-system-installation) + * [Services Shut Down](#default-services-shut-down) + * [Custom Provisioning Scenario](#custom-provisioning-scenario) + * [Configuration](#custom-configuration) + * [Artifacts Building](#custom-artifacts-building) +* [Provisioning Configuration](#provisioning-configuration) + * [Configuration Methods](#configuration-methods) + * [Command Line Arguments](#command-line-arguments) + * [Configuration File Generation](#configuration-file-generation) + * [Configuration File Summary](#configuration-file-summary) + * [Experience Kit Configuration](#experience-kit-configuration) +* [GitHub Credentials](#github-credentials) +* [Docker Pull Rate Limit](#docker-pull-rate-limit) + * [Registry Mirror](#registry-mirror) + * [Docker Hub Credentials](#docker-hub-credentials) +* [Troubleshooting](#troubleshooting) + +## Overview + +The Intel® Smart Edge Open automated provisioning process relies on the [Intel® Edge Software +Provisioner](https://github.com/intel/Edge-Software-Provisioner) (ESP). It provides a method of +automatic operating system installation and the Intel® Smart Edge Open cluster deployment. + +The Developer Experience Kit provides the `dek_provision.py` command-line utility, using the +Intel® Edge Software Provisioner toolchain to deliver a smooth installation experience. + +The provisioning process requires a temporary provisioning system operating on a separate machine +and routable from the subnet the provisioned machines are supposed to work on. + +## Provisioning Process Scenarios + +### Default Provisioning Scenario + +The default provisioning process consists of the following stages: + +* [Repository Cloning](#default-repository-cloning) +* [Configuration](#default-configuration) +* [Artifacts Building](#default-artifacts-building) +* [Services Start-Up](#default-services-start-up) +* [Installation Media Flashing](#default-installation-media-flashing) +* [System Installation](#default-system-installation) +* [Services Shut Down](#default-services-shut-down) + + +#### Repository Cloning + +Each of the Intel® Smart Edge Open experience kits comes with its provisioning utility tailored to the kit's +requirements. This script resides in the root directory of an experience kit repository, and its name matches the +following pattern: `_provision.py`, e.g., `dek_provision.py`. + +To be able to run the provisioning utility, clone the chosen experience kit repository. You can checkout the `main` +branch to access the latest experience kit version or select a specific release. In the second case, it is advised to +use the provisioning instruction published with the release to avoid incompatibilities caused by the process evolution. +For convenience, you can change the current directory to the directory the kit is cloned to, e.g.: + +```Shell.bash +[Provisioning System] # git clone https://github.com/smart-edge-open/open-developer-experience-kits.git --branch=smart-edge-open-21.09 ~/dek +[Provisioning System] # cd ~/dek +``` + + + +#### Configuration + +The Intel® Smart Edge Open default provisioning process is designed not to require any special configuration steps. The +provisioning scripts should work without any configuration options specified. In some environments, it may, however, be +necessary to customize some of them. For this purpose, the operator can set some of the most common parameters using +the command line interface. For details, see the [Command Line Arguments](#command-line-arguments) section. + +If there is a need to adjust +[configuration parameters exposed by the configuration file](#configuration-file-options-summary) +then the [Custom Provisioning Scenario](#custom-provisioning-scenario) should be followed. + + + +#### Artifacts Building + +To build the provisioning services, run the following command from the root directory of the Developer Experience Kit +repository. You can also use command-line arguments like `--registry-mirror` to specify some typical options: + +```Shell.bash +[Provisioning System] # ./dek_provision.py +``` + + +#### Services Start-Up + +```Shell.bash +[Provisioning System] # ./dek_provision.py --run-esp-for-usb-boot +``` + + +#### Installation Media Flashing + +To flash the installation image onto the flash drive, insert the drive into a USB port on the provisioning system and +run the following command: + +```Shell.bash +[Provisioning System] # ./esp/flashusb.sh --image ./out/SEO_DEK-efi.img --bios efi +``` + +The command should present an interactive menu allowing the selection of the destination device. You can also use the +`--dev` option to explicitly specify the device. + + + +#### System Installation + +Begin the installation by inserting the flash drive into the target system. Reboot the system, and enter the BIOS to +boot from the installation media. + +##### Log Into the System After Reboot + +The system will reboot as part of the installation process. + +The login screen will display the system's IP address and the status of the experience kit deployment. +To log into the system, use `smartedge-open` as both the user name and password. + +##### Check the Status of the Installation + +When logging in using remote console or SSH, a message will be displayed that informs about the status of the deployment, for example, +```Smart Edge Open Deployment Status: in progress``` + +Three statuses are possible: +- `in progress` - deployment is in progress +- `deployed` - deployment was successful - Developer Experience Kit cluster is ready +- `failed` - error occurred during the deployment + +Check the installation logs by running the following command: + +```Shell.bash +[Provisioned System] $ sudo journalctl -xefu seo +``` +Alternatively, you can inspect the deployment log found in `/opt/seo/logs`. + + +#### Services Shut Down + +``` +[Provisioning System] # ./dek_provision.py --stop-esp +``` + +### Custom Provisioning Scenario + +The custom provisioning scenario is very similar to the [default scenario](#default-provisioning-scenario). The only +difference is that it uses the [configuration file](#configuration-file-summary) to adjust some of the +provisioning parameters. + +See the [Default Provisioning Scenario](#default-provisioning-scenario) for the description of the common stages. + + +#### Configuration + +Generate a new configuration file as described in the [Configuration File Generation](#configuration-file-generation) section: + +```bash +[Provisioning System] # ./dek_provision.py --init-config.yml > custom.yml +``` + + +#### Artifacts Building + +The provisioning artifacts are built in the same way as in the case of the +[default scenario](#default-artifacts-building). The only difference is that the custom config file has to be specified using +the `--config` command-line option: + +```bash +[Provisioning System] # ./dek_provision.py --config=custom.yml +``` + +## Provisioning Configuration + +### Configuration Methods + +The provisioning utility and consequently the provisioning process allows two configuration methods: + +* Configuration via command line-arguments of the provisioning utility +* Configuration via provisioning configuration YAML file + +These methods can be used exclusively or mixed. The configuration options provided by the +command line arguments always override specific options provided by the configuration file. + +Not all the options possible to be customized in the configuration file can also be customized using the command line +arguments. However, the provisioning script is designed to allow the deployment of a standard experience kit cluster +using the command-line options only. + +### Command Line Arguments + +For the description of the options available for the command line use, see the provisioning utility help: + +``` +[Provisioning System] # ./dek_provision.py -h +``` + +### Configuration File Generation + +To generate a custom configuration file, use the `--init-config` option of the provisioning utility. When handling +this command, the utility prints an experience kit default configuration in the YAML format to the standard output. It +has to be redirected to a file of choice to keep it for further use: + +```bash +[Provisioning System] # ./dek_provision.py --init-config.yml > custom.yml +``` + +The operator can then modify the file to adjust needed options. To instruct the provisioning utility to use the custom +configuration file, use the `--config` option, e.g.: + +``` +[Provisioning System] # ./dek_provision.py --config=custom.yml +``` + +### Configuration File Summary + +For the description of the options available for the configuration file, see comments within the generated +configuration file. + +### Experience Kit Configuration + +For each provisioned experience kit, it is possible to adjust its configuration and specify user-provided files if +needed for a specific deployment variant. Both of these configuration adjustments are currently possible +through the [configuration file](#configuration-file-summary) only. For each of the provisioned experience kits (item +of the `profiles` list), it is possible to set its deployment variables through the `group_vars`, and `hosts_vars` +objects and the list of operator-provided files through the `sideload` list: + +``` +profiles: + - name: Smart_Edge_Open_Developer_Experience_Kits +[…] + group_vars: + groups: + all: + controller_group: + edgenode_group: + + host_vars: + hosts: + controller: + node01: + + sideload: [] +``` + +The experience kit configuration variables specified in the provisioning configuration override the default values +provided by the experience kit, so there is no need to adjust them in the +[default provisioning scenario](#default-provisioning-scenario). + +The operator-provided files specified in the `sideload` list are read from a local location, copied to the +provisioning artifacts, and finally to the provisioned system. + +## GitHub Credentials + +The access to some of the experience kits may be limited and controlled using git credentials. +In such a case, the operator has to provide these credentials to the provisioning script. + +The first method of providing them is through the `github` object of a custom +[configuration file](#configuration-file-summary): + +```yml +github: + user: '' + token: '' +``` + +The second method is to use the GitHub credentials options of the provisioning script: + +```bash +[provisioning system] # ./dek_provision.py -h +[…] + --github-user NAME NAME of the GitHub user to be used to clone required Smart Edge Open repositories + --github-token VALUE GitHub token to be used to clone required Smart Edge Open repositories +[…] +``` + +The credentials are used during the provisioning script (e.g., `dek_provision.py`) execution and other contexts like +provisioning services containers and installer system, so the operator has to provide them explicitly. + +The script will try to verify if it can access all the repositories specified through the configuration file and fail +if they cannot be accessed anonymously or with the operator-provided credentials. This functionality doesn't always +work, and eventually, it is the operator's responsibility to provide the credentials if needed. + +The scenario in which different repositories can be accessed using different credentials is currently not +supported. All the repositories must be either public or available for the specific user. + +## Docker Pull Rate Limit + +It is possible to use local Docker registry mirrors or Docker Hub +credentials to mitigate the [Docker pull rate limit](https://docs.docker.com/docker-hub/download-rate-limit/) consequences. + +### Registry Mirror + +It is the operator's responsibility to provide a working Docker registry mirror. When it is up and running, its URL can +be provided to the provisioning script. + +The first method of providing it is through the `docker` object of a custom +[configuration file](#configuration-file-summary): + +```yml +docker: + registry_mirrors: ['http://example.local:5000'] +``` + +The second method is to use the `--registry-mirror` option of the provisioning script: + +```bash +[provisioning system] # ./dek_provision.py -h +[…] + --registry-mirror URL + add the URL to the list of local Docker registry mirrors +[…] +``` + +If the custom configuration file contains some `registry_mirrors` list items, then the URL specified using the +`--registry-mirror` option will be appended to the end of the list. + +It is important to note that the provisioning script won't configure the provisioning system to use the registry +mirrors. It is the operator's responsibility to set it up. The script only takes care of the configuration of the +installer and the provisioned system. + +### Docker Hub Credentials + +The provisioning script provides a possibility to specify Docker Hub credentials to be used by the installer when +pulling images from Docker Hub. + +The first method of providing it is through the `docker` object of a custom +[configuration file](#configuration-file-summary): + +```yml +docker: + dockerhub: + username: "" + password: "" +``` + +The second method is to use the Docker Hub credentials options of the provisioning script: + +```bash +[provisioning system] # ./dek_provision.py -h +[…] + --dockerhub-user NAME + NAME of the user to authenticate with DockerHub during Live System stage + --dockerhub-pass VALUE + Password used to authenticate with DockerHub during Live System stage +[…] +``` + +Only the installer will use these credentials. They won't affect the provisioning and the provisioned systems. + +## Troubleshooting + +### Docker has to be installed + +#### Problem + +One of the following error messages may appear when you attempt to run the provisioning script (e.g., +`dek_provision.py`): + +```text +Docker has to be installed on this machine for the provisioning process to succeed. […] +Docker CLI has to be installed on this machine for the provisioning process to succeed. […] +The containerd.io runtime has to be installed on this machine for the provisioning process to succeed. […] +``` + +#### Solution + +Install Docker software according to the official instruction: +[Install Docker Engine on Ubuntu](https://docs.docker.com/engine/install/ubuntu/). + + +### The docker-compose tool has to be installed + +#### Problem + +The following error message appears when you attempt to run the provisioning script (e.g., `dek_provision.py`): + +```text +The docker-compose tool has to be installed on this machine for the provisioning process to succeed. […] +``` + +#### Solution + +Install `docker-compose` tool according to the official instruction: +[Install Docker Compose](https://docs.docker.com/compose/install/). + + +### Installation image couldn't be found in the expected location + +#### Problem + +The following error is displayed when you attempt to run the provisioning script (e.g., `dek_provision.py`): + +```text +ERROR: Installation image couldn't be found in the expected location + +``` + +#### Solution + +Retry the build attempt by rerunning the same provisioning command: + +```Shell.bash +[Provisioning System] # ./dek_provision.py […] +``` + +If it doesn't help, retry the build one more time with the `--cleanup` flag added. This option will force the complete +rebuild. + +```Shell.bash +[Provisioning System] # ./dek_provision.py --cleanup […] +``` diff --git a/components/resource-management/core-pinning.md b/components/resource-management/core-pinning.md new file mode 100644 index 0000000..f1890a3 --- /dev/null +++ b/components/resource-management/core-pinning.md @@ -0,0 +1,136 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Core Pinning + +## Overview + +CPU Manager is a Kubernetes feature that enables better placement of workloads in the Kubelet, the Kubernetes node agent, by allocating exclusive CPUs to certain pod containers. + +## How It Works + +A Kubernetes cluster can be divided into namespaces. If a container is created in a namespace that has a default CPU limit, and the container does not specify its own CPU limit, then the container is assigned the default CPU limit +Kubernetes keeps many aspects of how pods execute on nodes abstracted from the user. This is by design. However, some workloads require stronger guarantees in terms of latency and/or performance in order to operate acceptably. The kubelet provides methods to enable more complex workload placement policies while keeping the abstraction free from explicit placement directives. + +Default kubelet configuration uses [CFS quota](https://en.wikipedia.org/wiki/Completely_Fair_Scheduler) to manage PODs execution times and enforce imposed CPU limits. For such a solution it is possible that individual PODs are moved between different CPU because of changing circumistances on Kubernetes node. When cetrains pods end itheir lifespan or CPU throttling comes in place, then a pod can be moved to another CPU. + +Another solution, default for Intel® Smart Edge Open, supported by Kubernetes is CPU manager. CPU manager uses [Linux CPUSET](https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt) mechanism to schedule PODS to invividual CPUs. Kubernetes defines shared pool of CPUs which initially contains all the system CPUs without CPUs reverved for system and kubelet itself. CPU selection is configurable with kubelet options. Kubernetes uses shared CPU pool to schedule PODs with three QoS classes `BestEffort`, `Burstable` and `Guaranteed`. +When a pod is qualified as `Guaranteed` QoS class then kubelet removes requested CPUs amount from shared pool and assigns the pod exclusively to the CPUs. + +## How To + +### Configure the CPU manager before cluster deployment + +Kubernetes CPU Management needs CPU manager policy to be set to `static` which is a default option in Intel® Smart Edge Open. This can be adjusted in the ESP provisioning configuration file, before deploying an experience kit. Amount of CPUs reserved for Kubernetes and operating system is defined in the same file: + +- generate a custom configuration file with `./dek_provision.py --init-config > custom.yml` +- edit generated file and set `policy` or `reserved_cpus` under `group vars: all:`, e.g. + +```yaml +profiles: + - name: SEO_DEK + [...] + group_vars: + groups: + all: + policy: "static" + reserved_cpus: "0,1" +``` +- use the custom configuration for all the following `dek_provision.py` command invocations, i.e. `./dek_provision.py --config=custom.yml [...]` + +### Create pod definitions + +The Kubernetes CPU Manager defines three quality of service classes for pods: +- `BestEffort` QoS class is assigned to pods which do not define any memory and CPU limits and requests. pods from this QoC class run in the shared pool +- `Burstable` QoS class is assigned to pods which define memory or CPU limits and requests which do not match. Pods from `Bustrable` QoS class run in the shared pool. +- `Guaranteed` QoS class is assigned to pods which define memory and CPU limits and requests and those two values are equal. The values set to CPU limits and request have to be integral, fractional CPU specified caused the pod to be run on the shared pool. + +**Pod defined without any constraints.** This will be assigned `BestEffort` QoS class and will run on shared poll. + +```yaml +spec: + containers: + - name: nginx + image: nginx +``` + +**Pod defined with some constraints.** This will be assigned `Burstable` QoS class and will run on shared poll. + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "2" + requests: + memory: "100Mi" + cpu: "1" +``` + +**Pod defined with constraints, limits are equal to requests and CPU is integral bigger than or equal to one.** This will be assigned `Guaranteed` QoS classs and will run exclusively on CPUs assigned by Kubernetes. + +```yaml +spec: + containers: + - name: nginx + image: nginx + resources: + limits: + memory: "200Mi" + cpu: "2" + requests: + memory: "200Mi" + cpu: "2" +``` + +Pod defined with constraints even when limits are equal to request but CPU is specified as a fractional number will not get exclusive CPUs but will be run on the shared pool. Still, QoS class for such a pod is `Guaranteed`. + +**Example POD using CPU Manager feature.** + +```yaml +apiVersion: v1 +kind: Pod +metadata: + labels: + app: test-pod + name: test-pod +spec: + containers: + - name: nginx + image: nginx + imagePullPolicy: "IfNotPresent" + resources: + limits: + cpu: 1 + memory: "200Mi" + requests: + cpu: 1 + memory: "200Mi" + restartPolicy: Never + ``` + +Scheduled pod is assigned `Guaranteed` quality of service class, this can be examined by issuing `kubectl describe pod/test-pod`. + +Part of sample ouput is: + +```yaml +QoS Class: Guaranteed +``` + +Invidual processes/threads processor affinity can be checked on the node where the pod was scheduled with `taskset` command. +Process started by a container with `Guaranteed` pod QoS class has set CPU affinity according to the POD definition. It runs exclusively on CPUs removed from shared pool. All processes spawned from pod assigned to `Guaranteed` QoS class are scheduled to run on the same exclusive CPU. Processes from `Burstable` and `BestEffort` QoS classes PODs are scheduled to run on shared pool CPUs. This can be examined with example nginx container. + +```bash +[root@vm ~]# for p in `top -n 1 -b|grep nginx|gawk '{print $1}'`; do taskset -c -p $p; done +pid 5194's current affinity list: 0,1,3-7 +pid 5294's current affinity list: 0,1,3-7 +pid 7187's current affinity list: 0,1,3-7 +pid 7232's current affinity list: 0,1,3-7 +pid 17715's current affinity list: 2 +pid 17757's current affinity list: 2 +``` diff --git a/components/resource-management/images/README.md b/components/resource-management/images/README.md new file mode 100644 index 0000000..d63be1b --- /dev/null +++ b/components/resource-management/images/README.md @@ -0,0 +1,6 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2020 Intel Corporation +``` + +# Smart Edge Open Documentation diff --git a/components/resource-management/images/nfd0.png b/components/resource-management/images/nfd0.png new file mode 100644 index 0000000..2a68aff Binary files /dev/null and b/components/resource-management/images/nfd0.png differ diff --git a/components/resource-management/images/nfd1.png b/components/resource-management/images/nfd1.png new file mode 100644 index 0000000..31abff4 Binary files /dev/null and b/components/resource-management/images/nfd1.png differ diff --git a/components/resource-management/images/nfd2.png b/components/resource-management/images/nfd2.png new file mode 100644 index 0000000..c583f58 Binary files /dev/null and b/components/resource-management/images/nfd2.png differ diff --git a/components/resource-management/images/tm1.png b/components/resource-management/images/tm1.png new file mode 100644 index 0000000..a0f551d Binary files /dev/null and b/components/resource-management/images/tm1.png differ diff --git a/components/resource-management/images/tm2.png b/components/resource-management/images/tm2.png new file mode 100644 index 0000000..287f5b7 Binary files /dev/null and b/components/resource-management/images/tm2.png differ diff --git a/components/resource-management/node-feature-discovery.md b/components/resource-management/node-feature-discovery.md new file mode 100644 index 0000000..5b9ad9a --- /dev/null +++ b/components/resource-management/node-feature-discovery.md @@ -0,0 +1,135 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Node Feature Discovery + +## Overview + +Node Feature Detection (NFD) is a Kubernetes\* add-on that detects and advertises the hardware and software capabilities of a platform. + +NFD is installed by the following experience kits: +- [Developer Experience Kit](experience-kits/developer-experience-kit.md) + +Commercial, off-the-shelf (COTS) platforms used for edge deployment offer features workloads can take advantage of to provide better performance. When such COTS platforms are deployed in a cluster as part of a cloud-native deployment, it becomes important to detect the hardware and software features and also special accelerator hardware (FPGA, GPU, Non-Volatile Memory Express (NVMe)*, etc.) on all nodes that are part of that cluster. NFD supports targeting of intelligent configuration and capacity consumption of platform capabilities. + +## How It Works + +Consider an edge application that is deployed in the cloud-native edge cloud. It is favorable for a container orchestrator like Kubernetes to detect the nodes that have hardware and software features (NVMe, media extensions, etc.) required by the application. + +Consider a Container Network Function (CNF). It is favorable for the container orchestrator to detect nodes that have hardware and software features—FPGA acceleration for Forward Error Correction (FEC), advanced vector instructions to implement math functions, real-time kernel, etc. + +NFD detects hardware features available on each node in a Kubernetes cluster and advertises those features using node labels. + +NFD runs as a separate container on each node of the cluster. It discovers the capabilities of the node and publishes them as node labels using the Kubernetes API. NFD only handles non-allocable features. + +NFD consists of two software components: + +1. nfd-master is responsible for labeling Kubernetes node objects +2. nfd-worker detects features and communicates them to the nfd-master. One instance of nfd-worker should be run on each node of the cluster. + +Some of the Node features that NFD can detect include: + +![Sample NFD Features](images/nfd1.png) +![Sample NFD Features](images/nfd2.png) +*Figure - Sample NFD Features* + +The figure below illustrates how sample application: CDN will be deployed on the correct platform when NFD is utilized, where the required key hardware like NVMe and the AVX instruction set support is available. + +[![CDN app deployment with NFD Features](images/nfd0.png)](images/nfd0.png) +*Figure - CDN app deployment with NFD Features* + +The connection between nfd-nodes and nfd-control-plane is secured by certificates generated before running NFD pods. + +NFD automatically collects features from nodes and labels them in Kubernetes. + +## How To + +### Enable NFD add-on + +NFD is enabled by default and does not require any configuration or user input. + +### Disable NFD add-on + +NFD can be disabled by changing the `ne_nfd_enable` variable to `false` in the ESP provisioning configuration file (before Smart Edge Open deployment): + +- generate a custom configuration file with `./dek_provision.py --init-config > custom.yml` +- edit generated file and set `ne_nfd_enable` under `group vars: all:`, e.g. + +```yaml +profiles: + - name: SEO_DEK + [...] + group_vars: + groups: + all: + ne_nfd_enable: false +``` +- use the custom configuration for all the following `dek_provision.py` command invocations, i.e. `./dek_provision.py --config=custom.yml [...]` + +### Get a list of features + +To list the features found and labeled by NFD, use the following command: + +``` +kubectl get no -o json | jq '.items[].metadata.labels' +``` + +Example output : + +``` +{ + "beta.kubernetes.io/arch": "amd64", + "beta.kubernetes.io/os": "linux", + "feature.node.kubernetes.io/cpu-cpuid.ADX": "true", + "feature.node.kubernetes.io/cpu-cpuid.AESNI": "true", + "feature.node.kubernetes.io/cpu-cpuid.AVX": "true", + "feature.node.kubernetes.io/cpu-cpuid.AVX2": "true", + "feature.node.kubernetes.io/cpu-cpuid.FMA3": "true", + "feature.node.kubernetes.io/cpu-cpuid.HLE": "true", + "feature.node.kubernetes.io/cpu-cpuid.RTM": "true", + "feature.node.kubernetes.io/cpu-rdt.RDTCMT": "true", + "feature.node.kubernetes.io/cpu-rdt.RDTL3CA": "true", + "feature.node.kubernetes.io/cpu-rdt.RDTMBM": "true", + "feature.node.kubernetes.io/cpu-rdt.RDTMON": "true", + "feature.node.kubernetes.io/iommu-enabled": "true", + "feature.node.kubernetes.io/kernel-config.NO_HZ": "true", + "feature.node.kubernetes.io/kernel-config.NO_HZ_FULL": "true", + "feature.node.kubernetes.io/kernel-config.PREEMPT": "true", + "feature.node.kubernetes.io/kernel-version.full": "3.10.0-957.21.3.rt56.935.el7.x86_64", + "feature.node.kubernetes.io/kernel-version.major": "3", + "feature.node.kubernetes.io/kernel-version.minor": "10", + "feature.node.kubernetes.io/kernel-version.revision": "0", + "feature.node.kubernetes.io/memory-numa": "true", + "feature.node.kubernetes.io/network-sriov.capable": "true", + "feature.node.kubernetes.io/pci-0300_102b.present": "true", + "feature.node.kubernetes.io/system-os_release.ID": "centos", + "feature.node.kubernetes.io/system-os_release.VERSION_ID": "7", + "feature.node.kubernetes.io/system-os_release.VERSION_ID.major": "7", + "feature.node.kubernetes.io/system-os_release.VERSION_ID.minor": "", + "kubernetes.io/arch": "amd64", + "kubernetes.io/hostname": "edgenode-kubeovn", + "kubernetes.io/os": "linux", + "node-role.kubernetes.io/worker": "worker" +} +``` + +### Specify the features available to a pod + +To specify which features should be available by the node at deploying pod time, the `nodeSelector` field should be defined in the application pod `.yaml` file. Example application `golang-test` pod definition `yaml` file with `nodeSelector`: + +``` +apiVersion: v1 +kind: Pod +metadata: + labels: + env: test + name: golang-test +spec: + containers: + - image: golang + name: go1 + nodeSelector: + feature.node.kubernetes.io/cpu-pstate.turbo: 'true' +``` diff --git a/components/resource-management/topology-manager.md b/components/resource-management/topology-manager.md new file mode 100644 index 0000000..d4ec422 --- /dev/null +++ b/components/resource-management/topology-manager.md @@ -0,0 +1,93 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Topology Manager + +## Overview + +Topology Manager is a Kubelet component that aims to co-ordinate the set of components that are responsible for these optimizations. + +## How It Works + +Multi-core and Multi-Socket commercial, off-the-shelf (COTS) systems are widely used for the deployment of application and network functions. COTS systems provide a variety of IO and memory features. In order to achieve determinism and high performance, mechanisms like CPU isolation, IO device locality, and socket memory allocation are critical. Cloud-native stacks such as Kubernetes are beginning to leverage resources such as CPU, hugepages, and I/O, but are agnostic to the Non-Uniform Memory Access (NUMA) alignment of these. Non-optimal, topology-aware NUMA resource allocation can severely impact the performance of latency-sensitive workloads. Topology Manager addresses the requirement. + +Let us look at an Analytics application that is consuming multiple, high-definition video streams and executing an analytics algorithm. This analytics application pod is compute-, memory-, and network performance-sensitive. To improve performance of the analytics application on a typical dual-socket or multi-NUMA node system, an Orchestrator like Kubernetes needs to place the analytics pod on the same NUMA node where the Network Card is located and the memory is allocated. Without the topology manager, the deployment may be like that shown in the diagram below, where the analytics application is on NUMA 1 and the Device is on NUMA 2. This leads to poor and unreliable performance. + +![Pod deployment issue without Topology Manager](tm-images/tm1.png) +_Figure - Pod deployment issue without Topology Manager_ + +With Topology manager, this issue is addressed and the analytics pod placement will be such that the resource locality is maintained. + +![Pod deployment with Topology Manager](tm-images/tm2.png) +_Figure - Pod deployment with Topology Manager_ + +## How To + +### Enable Topology Manager + +Set policy to `best-effort` in the ESP provisioning configuration file (before Smart Edge Open deployment): + +- generate a custom configuration file with `./dek_provision.py --init-config > custom.yml` +- edit generated file and set `topology_manager: policy` under `group vars: all:`, e.g. + +```yaml +profiles: + - name: SEO_DEK + [...] + group_vars: + groups: + all: + topology_manager: + policy: "best-effort" +``` + +Available values of Kubernetes Topology Manager policy: `none` (disabled), `best-effort` (default), `restricted`, `single-numa-node`. Refer to the [Kubernetes Documentation](https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/) for details of these policies. + +User can also set `reserved_cpus` to a number that suits best. This parameter specifies the logical CPUs that will be reserved for a Kubernetes system Pods and OS daemons. + +```yaml +profiles: + - name: SEO_DEK + [...] + group_vars: + groups: + all: + reserved_cpus: "0,1" +``` +- use the custom configuration for all the following `dek_provision.py` command invocations, i.e. `./dek_provision.py --config=custom.yml [...]` + +### Use Topology Manager + +Create a Pod with a `guaranteed` QoS class (requests equal to limits). For example: + +```yaml +kind: Pod +apiVersion: v1 +metadata: + name: examplePod +spec: + containers: + - name: example + image: alpine + command: ["/bin/sh", "-ec", "while :; do echo '.'; sleep 5 ; done"] + resources: + limits: + cpu: "8" + memory: "500Mi" + requests: + cpu: "8" + memory: "500Mi" +``` + +Then apply it with `kubectl apply`. Check in the kubelet's logs on the node (`journalctl -xeu kubelet`), that Topology Manager obtained all the info about preferred affinity and deployed the Pod accordingly. The logs should be similar to the one below. + +``` +Nov 05 09:22:52 tmanager kubelet[64340]: I1105 09:22:52.548692 64340 topology_manager.go:308] [topologymanager] Topology Admit Handler +Nov 05 09:22:52 tmanager kubelet[64340]: I1105 09:22:52.550016 64340 topology_manager.go:317] [topologymanager] Pod QoS Level: Guaranteed +Nov 05 09:22:52 tmanager kubelet[64340]: I1105 09:22:52.550171 64340 topology_hints.go:60] [cpumanager] TopologyHints generated for pod 'examplePod', container 'example': [{0000000000000000000000000000000000000000000000000000000000000001 true} {0000000000000000000000000000000000000000000000000000000000000010 true} {0000000000000000000000000000000000000000000000000000000000000011 false}] +Nov 05 09:22:52 tmanager kubelet[64340]: I1105 09:22:52.550204 64340 topology_manager.go:285] [topologymanager] ContainerTopologyHint: {0000000000000000000000000000000000000000000000000000000000000010 true} +Nov 05 09:22:52 tmanager kubelet[64340]: I1105 09:22:52.550216 64340 topology_manager.go:329] [topologymanager] Topology Affinity for Pod: 4ad6fb37-509d-4ea6-845c-875ce41049f9 are map[example:{0000000000000000000000000000000000000000000000000000000000000010 true}] + +``` diff --git a/components/telemetry/images/README.md b/components/telemetry/images/README.md new file mode 100644 index 0000000..d63be1b --- /dev/null +++ b/components/telemetry/images/README.md @@ -0,0 +1,6 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2020 Intel Corporation +``` + +# Smart Edge Open Documentation diff --git a/components/telemetry/images/dashboards_manage.png b/components/telemetry/images/dashboards_manage.png new file mode 100644 index 0000000..9aa5dab Binary files /dev/null and b/components/telemetry/images/dashboards_manage.png differ diff --git a/components/telemetry/images/grafana_add_panel_cpu.png b/components/telemetry/images/grafana_add_panel_cpu.png new file mode 100644 index 0000000..effba65 Binary files /dev/null and b/components/telemetry/images/grafana_add_panel_cpu.png differ diff --git a/components/telemetry/images/grafana_add_panel_memory.png b/components/telemetry/images/grafana_add_panel_memory.png new file mode 100644 index 0000000..e75b928 Binary files /dev/null and b/components/telemetry/images/grafana_add_panel_memory.png differ diff --git a/components/telemetry/images/grafana_explore.png b/components/telemetry/images/grafana_explore.png new file mode 100644 index 0000000..8a5d77e Binary files /dev/null and b/components/telemetry/images/grafana_explore.png differ diff --git a/components/telemetry/images/grafana_explore_option.png b/components/telemetry/images/grafana_explore_option.png new file mode 100644 index 0000000..36be7fd Binary files /dev/null and b/components/telemetry/images/grafana_explore_option.png differ diff --git a/components/telemetry/images/grafana_login_page.png b/components/telemetry/images/grafana_login_page.png new file mode 100644 index 0000000..d0669b2 Binary files /dev/null and b/components/telemetry/images/grafana_login_page.png differ diff --git a/components/telemetry/images/telemetry_2109.svg b/components/telemetry/images/telemetry_2109.svg new file mode 100644 index 0000000..3479777 --- /dev/null +++ b/components/telemetry/images/telemetry_2109.svg @@ -0,0 +1,3 @@ + + +
EdgeNode
EdgeNode
Prometheus
server
Prometheus...
Collectd Daemonset
Collectd Daemonset
NodeExporter Daemonset
NodeExporter Daemons...
StatsdExporter Service
StatsdExporter Servi...
Cadvisor
Daemonset
Cadvisor...
Container metrics
Container m...
System metrics
System metr...
Grafana
Grafana
KubeVirt metrics
KubeVirt metr...


StatsD instrumented application


StatsD instrumented a...Viewer does not support full SVG 1.1 \ No newline at end of file diff --git a/components/telemetry/telemetry.md b/components/telemetry/telemetry.md new file mode 100644 index 0000000..e2708a2 --- /dev/null +++ b/components/telemetry/telemetry.md @@ -0,0 +1,259 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Intel® Smart Edge Open Telemetry Documentation + +## Overview +Intel® Smart Edge Open comes with a set of telemetry components, providing user with the ability to monitor the performance and health of the cluster nodes. This support allows users to retrieve information about the platform, the underlying hardware, cluster, and applications deployed. The data gathered by telemetry can be used to visualize metrics and resource consumption, set up alerts for certain events, and aid in making scheduling decisions based on the received telemetry. + +Currently, the support for telemetry is focused on metrics; support for application tracing telemetry is planned in the future. + + +Telemetry bundle is installed by the following experience kits: +- [Developer Experience Kit]() +- [Private Wireless Experience Kit]() + +## How It Works + +The telemetry components used in Intel® Smart Edge Open are deployed from the Edge Controller as Kubernetes* (K8s) pods. Components for telemetry support include: + +- collectors +- metric aggregators +- monitoring and visualization tools + +### How it's deployed + +Depending on the role of the component, it is deployed as either a Deployment or Deamonset. Generally, global components receiving inputs from local collectors are deployed as a Deployment type with a single replica set, whereas local collectors running on each host are deployed as Daemonsets. Local collectors running on Edge Nodes that collect platform metrics are deployed as privileged containers, using host networking. Communication between telemetry components is secured with TLS either using native TLS support for a given feature or using a reverse proxy running in a pod as a container. All the components are deployed as Helm charts. +
+
+21.09 Telemetry overview + + +The deployment of telemetry components in Intel® Smart Edge Open is easily configurable from the open-developer-experience-kits (DEK). The deployment of the Grafana dashboard is optional (telemetry_grafana_enable enabled by default). There are four distinctive flavors for the deployment of the CollectD collector, enabling the respective set of plugins (telemetry_flavor): + +- common (default) +- flexran +- smartcity +- corenetwork + +Further information on what plugins each flavor enables can be found in the Collectd section. All flags can be changed in ESP provisioning configuration file (before Smart Edge Open deployment): + +- generate a custom configuration file with `./dek_provision.py --init-config > custom.yml` +- edit generated file and set telemetry flavor under `group vars: all:`, e.g. + +```yaml +profiles: + - name: SEO_DEK + [...] + group_vars: + groups: + all: + telemetry_flavor: common + telemetry_grafana_enable: true + telemetry_prometheus_enable: true + telemetry_statsd_exporter_enable: true + telemetry_statsd_exporter_udp_port: 8125 + telemetry_statsd_exporter_tcp_port: 8125 + telemetry_collectd_enable: true + telemetry_cadvisor_enable: true +``` +- use the custom configuration for all the following `dek_provision.py` command invocations, i.e. `./dek_provision.py --config=custom.yml [...]` + +### Collectd + +Collectd is a daemon/collector enabling the collection of hardware metrics from computers and network equipment. It provides support for Collectd plugins, which extends its functionality for specific metrics collection such as Intel® RDT, Intel PMU, and ovs-dpdk. The metrics collected are exposed to the Prometheus monitoring tool via the [CollectdExporter](https://github.com/prometheus/collectd_exporter) sidecar. In Intel® Smart Edge Open, Collectd is supported with the help of the OPNFV Barometer project - using its Docker image and available plugins. As part of the Intel® Smart Edge Open release, a Collectd plugin for Intel® FPGA Programmable Acceleration Card (Intel® FPGA PAC) N3000 telemetry is now available from Intel® Smart Edge Open (power and temperature telemetry). In Intel® Smart Edge Open, the Collectd pod is deployed as a K8s Daemonset on every available Edge Node, and it is deployed as a privileged container running in host network. + +In DEK metrics obtained by NodeExporter are subject to `collectd_` pattern. + +#### Plugins + +There are four distinct sets of plugins (flavors) enabled for CollectD deployment that can be used depending on the use-case/workload being deployed on Intel® Smart Edge Open. `Common` is the default flavor in Intel® Smart Edge Open. The flavors available are: `common`, `corenetwork`, `flexran`, and `smartcity`. Below is a table specifying which CollectD plugins are enabled for each flavor. +The various CEEK flavors are enabled for CollectD deployment as follows: + + +| Common | Core Network | FlexRAN | SmartCity | +|------------------|:-----------------:|------------------:|-------------------:| +| cpu | cpu | cpu | cpu | +| cpufreq | cpufreq | cpufreq | cpufreq | +| load | load | load | load | +| hugepages | hugepages | hugepages | hugepages | +| intel_pmu | intel_pmu | intel_pmu | intel_pmu | +| intel_rdt | intel_rdt | intel_rdt | intel_rdt | +| ipmi | ipmi | ipmi | ipmi | +| network | network | network | network | +| | ovs_pmd_stats | ovs_stats | ovs_pmd_stat | +| | ovs_stats | fpga_telemetry | | + +#### Specifying logging threshold + +By default only logs of "WARNING" and above thresholds will be displayed. User can change this changing the variable `telemetry_collectd_log_level` to one of "INFO","WARNING","ERROR". Collectd is not built to support "DEBUG" level logs. Please be wary that setting this variable to lower threshold (like INFO) will result in increased disk usage. + +#### Collectd collection interval + +Due to Collectd architecture relying on Prometheus exporter, Collectd collection interval is half of polling interval of Prometheus set via `telemetry_prometheus_scrape_interval_seconds` variable. For more information please refer to Prometheus' documentation. +### NodeExporter + +Node Exporter is a Prometheus exporter that exposes hardware and OS metrics of \*NIX kernels. The metrics are gathered within the kernel and exposed on a web server so they can be scraped by Prometheus. In Intel® Smart Edge Open, the Node Exporter pod is deployed as a K8s Daemonset; it is a privileged pod, using host networking that runs on every Edge Node in the cluster. It is enabled by default by DEK. + +Node exporter is configured to expose the [default set of collectors and metrics](https://github.com/prometheus/node_exporter/tree/v1.0.0-rc.0#enabled-by-default). Support for distinct collector and metrics vary depending on Operating System and hardware configuration - for details refer to Node Exporter specification. + +In Intel® Smart Edge Open metrics obtained by NodeExporter are subject to `node_` pattern. + +### Cadvisor + +The cAdvisor is a running daemon that provides information about containers running in the cluster. It collects and aggregates data about running containers such as resource usage, isolation parameters, and network statistics. Once the data is processed, it is exported. The data can be easily obtained by metrics monitoring tools such as Prometheus. In Intel® Smart Edge Open, cAdvisor is deployed as a K8s daemonset on every Edge Node and scraped by Prometheus. Cadvisor deployment can be disabled by setting the `telemetry_cadvisor_enable` flag to `false` in provisioning configuration file. + +#### Cadvisor and performance + +CAdvisor is often reported to consume large portions of resources. For deployments with heavy resource constraints, where the availability of container related metrics is not paramount, it is advised to disable it. + +### StatsD + +[StatsD](https://github.com/statsd/statsd#statsd---) is a network daemon which listens for statistics like counters and timers sent over UDP or TCP. Usage of StatsD exporter allows to convert received StatsD-style metrics to Prometheus metrics and export them. By default, any non-alphanumeric characters, including periods, is translated into underscores. There is also a possibility to define customized mapping for selected metrics and personalized labels. Mapping can be configured in `statsd:mappingConfig` section in `roles/telemetry/statsd-exporter/templates/values.yml.j2` helm chart configuration file. Example mapping configuration can be found in [StatsD exporter repository](https://github.com/prometheus/statsd_exporter#metric-mapping-and-configuration). StatsD exporter pod is deployed on a control plane as a K8s `Deployment` which receives StatsD metrics from pods and is scraped by Prometheus. It is enabled by default in DEK and can be enabled/disabled by changing the `telemetry_statsd_exporter_enable` flag (in provisioning configuration file). + +### Prometheus + +Prometheus is an open-source, community-driven toolkit for systems monitoring and alerting. The main features include: + +- PromQL query language +- multi-dimensional, time-series data model +- support for dashboards and graphs + +The main idea behind Prometheus is that it defines a unified metrics data format that can be hosted as part of any application that incorporates a simple web server. The data can be then scraped (downloaded) and processed by Prometheus using a simple HTTP/HTTPS connection. + +In Intel® Smart Edge Open, Prometheus is deployed as a K8s Deployment with a single pod/replica on the EdgeNode. It is configured out of the box to scrape all other telemetry endpoints/collectors enabled in Intel® Smart Edge Open and gather data from them. Prometheus is enabled in the DEK by default with the telemetry/prometheus role. + +#### Prometheus' collection interval + +By default Prometheus' collection interval is set to 60 second. This value can be changed via `telemetry_prometheus_scrape_interval_seconds` variable. Prometheus exporters query metrics on demand, with notable exception of Collectd, where the exporter works as a cache for metrics provided by collectd. In order to make sure that Prometheus query always yields updated metrics the interval of Collectd was decreased to match half of Prometheus' polling interval. + +#### Prometheus' retention setting + +Prometheus is not long-term storage for metrics. By default the retention of metrics stored by prometheus is set to 15d. If user wants to override this setting, this can be achieved by setting the `telemetry_prometheus_retention` value to desired duration. + +#### Prometheus and KubeVirt + +Prometheus is configured to scrape metrics related to VMs operated by KubeVirt by default. +#### 21.09 and the availability of Prometheus Dashboard + +Users coming from Openness environment might be surprised by the lack of Prometheus Dashboard http endpoint running on port 3000. Due to security constraints for 21.09, the NodePort access to Prometheus dashboard is disabled. User concerned with this change can re-enable the NodePort (details for that will be presented in "How To" section) or use [port-forwarding feature](https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/), forwarding port 80 of prometheus-server service. + +### Grafana + +Grafana is an open-source visualization and analytics software. It takes the data provided from external sources and displays relevant data to the user via dashboards. It enables the user to create customized dashboards based on the information the user wants to monitor and allows for the provision of additional data sources. In Intel® Smart Edge Open, the Grafana pod is deployed as a K8s Deployment type and is by default provisioned with data from Prometheus. It is enabled by default in DEK and can be enabled/disabled by changing the `telemetry_grafana_enable` flag (in provisioning config file). The detailed usage of Grafana will be presented in the "How To" section. + +Default account is created with username `admin` and random password. Method of obtaining this password is shown in "How to" section. + + +## How To + +### Log into Grafana + +Requirements: + +- Access to the node configured for `kubectl` usage with Intel® Smart Edge Open cluster. +- Network connection to EdgeNode +- One of [Grafana-supported browsers](https://grafana.com/docs/grafana/latest/installation/requirements/#supported-web-browsers) + +Steps: + +- Execute following command to obtain the password: + ```[bash] + kubectl get secrets/grafana -n telemetry -o json | jq -r '.data."admin-password"' | base64 -d + ``` +- Open `https://:3200` in the browser +- Enter login and obtained password + ![Grafana login](images/grafana_login_page.png) + + +### Check the collected of metrics in Prometheus using Grafana + +Requirements: + +- Network connection to EdgeNode +- Obtained login informations + +Steps: + +- Log in to Grafana using browser +- Select the "Explore" panel from the side-menu + ![Explore option](images/grafana_explore_option.png) +- Enter the metric of interests (in this example it's `collectd_memory`) and press `Run Query` + ![Result of executed query](images/grafana_explore.png) + +### Create Grafana dashboard + +Requirements: + +- Network connection to EdgeNode +- Obtained login informations + +Steps: + +- Log into Grafana +- Go to "Dashboard panel" -> Manage. Alternatively user can click `Create` panel (big plus sign) and select `Dashboard` from the panel menu. + ![Dashboard managment view](images/dashboards_manage.png) +- Click on `New Dashboard` +- In the following steps we will create simple dashboard showing memory and cpu usage on the node + - Click `Add an empty panel`. First we're gonna add a Table showing cpu usage by container (cAdvisor must be running) + - Enter `container_cpu_usage_seconds_total` in the `Metric browser` field. After selection grafana should ask to use the `rate` function. Do it and change the resolution to 1m. + - Add title to the panel. For this example the panel is named `Container CPU usage`. + - Change the panel resolution to `Last 5 minutes` + ![Filled out panel info for cpu](images/grafana_add_panel_cpu.png) + - Click `Apply` + - Let's repeat the steps above for memory usage, this time the name of namespace is `Container memory usage`, metric is `container_memory_usage_bytes`. Set the rate function to 1m. + - Set the unit to `bytes(IEC)` + ![Filled out panel info for memory](images/grafana_add_panel_memory.png) +- Click `Save dashboard` and fill out the name. We used `Simple resource consumption stats`. Click `Save` + +Comment: + +This "How to" is non-exhaustive there is a plethora of available metrics and panels to chose from. For further we recommend visiting [official documentation for Grafana](https://grafana.com/docs/grafana/latest/). + +### Enable prometheus Dashboard http NodePort + +Requirements: + +- Access to the node configured for `kubectl` usage with Intel® Smart Edge Open cluster. + +Steps: + +- Connect to the node +- Execute following command to expose the http dashboard endpoint on port 30000: + + ```[bash] + kubectl patch svc -n telemetry prometheus-server --type='json' -p '[{"op":"replace","path":"/spec/type","value":"NodePort"},{"op":"replace","path":"/spec/ports/0/nodePort","value":30000}]' + ``` + + +### Instrument application to write StatsD metrics + +For this "How to" a listing of simple Python application will be presented. For information how to use it with the language of your choosing please refer to language specific documentation: + +Listing: +```[Python] +import random +import statsd +import time +from datetime import datetime + +HOST = 'statsd-exporter.telemetry.svc' #This is the default statsd-exporter address in DEK +PORT = 8125 + +client = statsd.StatsClient(HOST, PORT) +start = time.time() + +random.seed() +while True: + random_sleep_interval = random.randint(5, 15) + print('Got random interval to sleep: {}'.format(random_sleep_interval)) + + time.sleep(random_sleep_interval) + client.incr('statsd_script.sleep_calls') + dt = int((time.time() - start) * 1000) + client.timing('example.timer', dt) + + client.incr('example.counter', random_sleep_interval) +``` diff --git a/experience-kits/developer-experience-kit.md b/experience-kits/developer-experience-kit.md new file mode 100644 index 0000000..c9dd23d --- /dev/null +++ b/experience-kits/developer-experience-kit.md @@ -0,0 +1,168 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` + +# Intel® Smart Edge Open Developer Experience Kit + +## Overview + +Intel® Smart Edge Open experience kits provide customized infrastructure deployments for common network and on-premises edge use cases. Combining Intel cloud-native technologies, wireless networking, and high-performance compute, experience kits let you deliver AI, video, and other services optimized for performance at the edge. + +The Developer Experience Kit (DEK) lets you easily install and instantiate an Intel® Smart Edge Open edge cluster. Once the cluster has been installed, you can onboard edge applications and run reference implementations -- example end-to-end solutions built on Intel® Smart Edge Open -- to get familiar with operating a stand-alone edge node or to start creating your own solution. + +## How It Works + +[![Smart Edge Open Developer Experience Kit Edge Node Component Diagram](images/dek-component-diagram.png)](images/dek-component-diagram.png) + +*Intel® Smart Edge Open Developer Experience Kit building blocks* + +The Developer Experience Kit uses [Edge Software Provisioner](https://github.com/intel/Edge-Software-Provisioner), which automates the process of provisioning bare-metal or virtual machines with an operating system and software stack. Intel® Smart Edge Open provides a fork of the [Ubuntu OS ESP +Profile](https://github.com/intel/rni-profile-base-ubuntu) tailored for its specific needs. + +### Building Blocks + +Building blocks provide specific functionality in the platform you'll deploy. Each experience kit installs a set of building blocks as part of deployment. You can use additional building blocks to customize your platform, or develop your own custom solution by combining building blocks. + +The Developer Experience Kit includes the following building blocks: + +| Building Block | Functionality | +| :------------- | :------------- | +|[Calico CNI](https://docs.projectcalico.org/about/about-calico) | Default container network interface | +[SR-IOV Network Operator](/components/networking/sriov-network-operator.md) | Additional container network interface | +[Multus CNI](/components/networking/multus.md) | Support for multiple network interfaces | +[Harbor](https://goharbor.io/) | Cloud native registry service that stores and distributes container images | +[Telemetry](/components/telemetry/telemetry.md) | Remote collection of device data for real-time monitoring| +[Node Feature Discovery (NFD)](/components/resource-management/node-feature-discovery.md) | Detects and advertises the hardware features available in each node of a Kubernetes* cluster | +[Topology Manager](/components/resource-management/topology-manager.md) | Coordinates the resources allocated to a workload | +[Core Pinning](/components/resource-management/core-pinning.md) | Dedicated CPU core for workload | +[Provisioning](/components/provisioning/provisioning.md) | Automated system provisioning | + +## Get Started +The instructions below walk you through provisioning the operating system and Developer Experience Kit on a target system. After completing these instructions, you will have created a single edge node cluster capable of hosting edge applications. You can then optionally install reference implementations from the Intel® Edge Software Hub. + +[![Smart Edge Open Developer Experience Kit Edge Node Component Diagram](images/dek-workflow-diagram.png)](images/dek-workflow-diagram.png) + +### Requirements +You will need two machines: a provisioning system where you will build a bootable image of the experience kit, and a target system where you will install the experience kit to create an edge cluster. + +#### Provisioning System +- Memory: At least 4GB RAM +- Hard drive: At least 20GB +- USB flash drive +- Operating system: Ubuntu 20.04. +- Git +- Docker and Docker Compose +- Python 3.6 or later, with the PyYAML module installed +- Internet access + +> NOTE: You must add the user account on the provisioning system to /etc/sudoers. + +#### Target System +- A server with two sockets, each populated with a 3rd Generation Intel® Xeon® Scalable Processor +- Memory: At least 32GB RAM +- Hard drives: Two SATA SSDs, one for booting and one for data caching +- Network adapters: Two NICs, one connected to each socket +- Connection to the provisioning system + +> NOTE: The provisioning process will install Ubuntu 20.04 on the target machine. Any existing operating system will be overwritten. + +#### Knowledge + +Basic knowledge of operating system, Kubernetes, and server administration. + +### Install the Developer Experience Kit + +The Developer Experience Kit provides a command line utility (`dek_provision.py`) which uses the +Intel® Edge Software Provisioner toolchain to deliver a smooth installation experience. + + + + + +You must be logged in as root on the provisioning system for the following steps. To become the root user, run the following command: + +```Shell.bash +[Provisioning System] $ sudo su - +``` +> NOTE: In order for the provisioning script to have the proper permissions, you must run the `sudo` command as shown above. Using `sudo` with the `dek_provision.py` command will not work. + +#### Clone the Repository + +Clone the kit's repository to the provisioning system: + +```Shell.bash +[Provisioning System] # git clone https://github.com/smart-edge-open/open-developer-experience-kits.git --branch=smart-edge-open-21.09 ~/dek +[Provisioning System] # cd ~/dek +``` + +#### Create the Installation Image + +##### Run the Provisioning Script +The `dek_provision.py` script builds and runs the provisioning services and prepares the installation media. + +##### Build and Run the Provisioning Services + +To build and run the provisioning services in a single step, run the following command from the root directory of the +Developer Experience Kit repository: + +```Shell.bash +[Provisioning System] # ./dek_provision.py --run-esp-for-usb-boot +``` + +Alternatively, to specify the Docker registry mirror to be used during the Developer Experience Kit deployment use the `--registry-mirror` option: +```Shell.bash +[Provisioning System] # ./dek_provision.py --registry-mirror=http://example.local:5000 --run-esp-for-usb-boot +``` + +The script will create an installation image in the `out` subdirectory of the current working directory. + + +##### Flash the Installation Image + +To flash the installation image onto the flash drive, insert the drive into a USB port on the provisioning system and run the following command: + +```Shell.bash +[Provisioning System] # ./esp/flashusb.sh --image ./out/SEO_DEK-efi.img --bios efi +``` + +The command should present an interactive menu allowing the selection of the destination device. You can also use the `--dev` option to explicitly specify the device. + +#### Install the Image on the Target System + +Begin the installation by inserting the flash drive into the target system. Reboot the system, and enter the BIOS to boot from the installation media. + +##### Log Into the System After Reboot + +The system will reboot as part of the installation process. + +The login screen will display the system's IP address and the status of the experience kit deployment. +To log into the system, use `smartedge-open` as both the user name and password. + +#### Check the Status of the Installation +When logging in using remote console or SSH, a message will be displayed that informs about status of the deployment, for example: +```Smart Edge Open Deployment Status: in progress``` + +Three statuses are possible: +- `in progress` - Deployment is in progress. +- `deployed` - Deployment was successful. The Developer Experience Kit cluster is ready. +- `failed` - An error occurred during the deployment. + +Check the installation logs by running the following command: + +```Shell.bash +[Provisioned System] $ sudo journalctl -xefu seo +``` +Alternatively, you can inspect the deployment log found in `/opt/seo/logs`. + +## Summary and Next Steps +In this guide, you created an Intel® Smart Edge Open edge node cluster capable of hosting edge applications. You can then optionally install reference implementations from the Intel® Edge Software Hub. +- Learn how to [onboard a sample application to your cluster](/application-onboarding/application-onboarding-cmdline.md) +- Download reference implementations from the [Intel® Edge Software Hub](https://software.intel.com/content/www/us/en/develop/topics/iot/edge-solutions.html) + + diff --git a/experience-kits/images/README.md b/experience-kits/images/README.md new file mode 100644 index 0000000..ab02239 --- /dev/null +++ b/experience-kits/images/README.md @@ -0,0 +1,6 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2019-2021 Intel Corporation +``` + +# Smart Edge Open Documentation diff --git a/experience-kits/images/dek-component-diagram.drawio b/experience-kits/images/dek-component-diagram.drawio new file mode 100644 index 0000000..34fcc07 --- /dev/null +++ b/experience-kits/images/dek-component-diagram.drawio @@ -0,0 +1 @@ +7LzXsqVMkiX8NH3ZY6iNuERrrblDa7HR8PQ/ZOb3dVVn1d89Y901ZTZ9LPNsCNgBhLsvX8sjOP8C0/3Jz/FUqWOWd/8CAdn5LzDzLxAEgR/w+Xhbrp8tKI79bCjnOvvZBP5bg13f+a9G4FfrVmf58lcnruPYrfX0143pOAx5uv5VWzzP4/HXpxVj99dXneIy/63BTuPu91a/ztbqVysIAP92QMjrsvp1afzz60Af/3Hyr4alirPx+IsmmP0XmJ7Hcf251Z903r2D98e4/Pwe93eO/nljcz6s/5kvLAcWBtkk+v+qCfnat9bGk//6q5c97rZfD/wvENo9/VHF+HQLoeW7/XzG/fRsDMky/dgHfm/6y68By3r9GkH0u41/HPjX5Yd9yecEEJnOfzv4x4XY7LEGBGiPGz0f9NhP4/A83vJH78/T/bv7+tvN0F/dAjSP25Dl7zgAz+GjqtfcnuL0PXo8bvu0VWvfPXvgsxl3dTk8211evJ3t+bzWjz+Qv5rXcfr1NFzc193r1uKw5p0+vLfs5OcfD0uP3Tj/uD7M/fh52pd1Htv8bx353Zp/mOa5fn7+RdMv6/L52OfrfD2n/Dr6r8gfTnn96aU/949/81wC/9VW/YXTIsivxvhXtJR/dv5vDvVs/PKp/w3/gv4P/evnOcn8xxlCPCfPmEHAnJf18uPO/v0p9NNhXA/5X58FCHnXv75UxfPrR8+hafzP+s0TstO7mV5d/TjQDP/H3pP8dDUl+bMhTtvyhwPq2/p0k/9q/xkKDPj5C5dLH+M/9/8fu1fddX/hRADwwA75f83tYPzfuR36u9uB0N9wOxDE/pvcDv7N7dxkG9btaYOA/wUgz6fi2D+RiMmLeOte0Fqrd4y7eM2Xd7fN5+HNZs8tAOSw1MnjFj92jGutxuHn9nIta/462JLPe53my48E9QP/h/gHmMXr21+81n/9leznTjam7euy9O8x8HdBDPzPg9j/oUexBAr+9Mw5/ZWTH1v9szgYQnx+xzXkbzgYDP83+Rfym3/Z1r+Kuvfmrnw9xrl9tvQpn+P1HZHXzf48IctfP3kHpdvKevivsfOvZPUfWHl5+qmHUvlxMoP90yEJhPw7QyO/GxoH/pah/7uA5POboTWO+WnQ4SdJ6eLk6ecZ1deK2zSN8/p/NXT/yUwIA/9ZE/53URD0NxPSzxCn408rZn+iP62J/2O4vzDc3wDZf6zhsN8Mpz6W2pafhuu3VwS+KRkY/oTc+h35Iv4Br/8TjL/b9PM7M/vH2hT/zabp2HWPcM/+TIi/7PuMbP7rmv9ExOifzKAQCP9mUOjz+QcyIeLvM6EXUH+Y8geyAsUP/fbnUY/7C4r9l4H7/6x1/9RNf7Af+HcEhol/ZLT+UT/42+H6iwDl5wuz+fwrbmnD/als+rwffzze/0Ty77aGPv9xHGP/SEUD/l4JTMlsr5c/9Mv/2PU/Y9c/6e4/DUCDv5fgjPntu8q331Mt8Jar8iH7H5P+/yTdD/5/26a/17dk/K08pc8AzWP3g0vFwz84mf5QwtSf5c4/zDGMb83zd5siJI7+dYUJ/KepMP0tovy3S5gw/L8+/11W/r3K9NPKvxLv7yXwvz1P8783LwP+rXmZdkvyf53m8XwH7d3p8vXvFe3/85Mx/+X1rb89V/Or6uW8Oz9L739VBvv8V7gu/k/juhDxn3Tdz38ba/y9aGav8bowf0EWf9VE/2h+L1ynyw8h8AzX1r+Q83LMeJq6x6I/i+f/0BT0d4z591LTP5Qt/q0E9I9li78X1fg5LuIh/lVVi5cqGeM5+3+YN/xGBf+RtEG1NFEeZrrF/TL9sIvRitXfnIz9d+b5Y8az7n+si/gTUZUXII1xqX9EIswk47qO/d9Hzmcsix8/fxeV42X6uV6jqM/XH6gflyT/aAX+aHm2s3h9/ernLsRNb62drj1Ktw5A5suRfH40261Yt3y2xHeflmgyfD6Zm0UZ+T2BDijRD9RnA2me3U08OGRyCOjZLkm2Y03PQgYdJOjgcL0k+ReIytDtu2HGLkNLWXEgxJqm50cygiwxcot3gorHjdU0YyzZmevPjWljSwGDwGYdKBRfxmg2mRNpWi55rbVaGlDE0FlytIC/85OyORgDvd02WVT/ANIhPC0C9ly2B4iHVigzCDrD87PrW2jZ9aBGVehxsUleshsAUwskLHr61j1W4SSWMtm7dCmR4eFT7aqBZTzTTtksZcvEAY1Rdjw0EOUFgYawowTawCQ7KnZU+r0nRY3ByN0JnarQx+LQxx4EZFpyKtWRhkiypBoeg4pQBytOUFlRqi2EjFwVc/cF0bYM79UfdH2T0q3vOmQMAoGiGlWvbjpkwUNHaJdqhTubOdQL48aUIVSvx+o0jL1vl4fjcAkA6enqOp8zfR0c4vRrRM9GzF0BUVthpyWGo8sUf6KSajIS9izT1ZjUIVbEmk7P4nosW6FwmeWLB1blOlOzbFnS3TeKMEhBHSnUHniVVcRZg4unk4fhcMNncwNiJ7qcELbaMDAEATr/+xxde3OrHb0RdC4zn0ijaEKebDxQHijU+kIfhA8hItZzABAjB/nyQtd9yuVun15F0XnaCetoaFflSy2V2uQBQQpf+oF4jvsMMR3+s+HJtcKheM6U5V5+SnpkwmKl21RUo41YJDcnitWfPWzHwkP4MD9s87W0/o6MZe3TnZAE5SZcTDGSJHgu8PPfNs3wk0U54+fFngPctwxOqsNzqTCO77gflOhNODztz9klpTIFXU7+rJeGkk/3WJ/Rsttwx0uf4Y7rd6y0MFgxXA66Tx4A6Koj97ieL0RRhiRNxl208xlJOIDoN+1/NWvXO6N4o9QZ0Lb+JpnxnKo/Hg95P437/IOq4Ny0TZJsRq9XHBOFYwdtz/LONGBLi6VUh7FY17VYizRr8TLIoVppUtw5eiyY670tyO+h4wEvCh5RhmJY0jTN5tqDvse+W2gIzIQ+OYpDH9SkvhCjTkbhiKWtPW2OXAew+T2lDs3kLg6/tGNSfPUcWVNMmcPGMeMmOlje6qNLneXjW1niTS2+QfcuC2pCiORzymxwsJoemCp+lFQZ861kdg5rRkN8Ebvx1od526Qs3my5NEiqK5YSJOkiOSPnk+uuSOo6UJKDIxhxgc5KoJWzxbCBVbaGCZUJwwIaCn8xYR2iA39CdnWC5+4ppjxcFUm/JksUAfiVtxlBrMqtR3eUv/IX/uDNDTaATopWJUkk44uT5pZAtFwIAM/HEFXBcwtPzDydUxIvkIX4XIWUFSYNTp55xrcMWYaCBKqc21D1RTbstSZ0FBDebeMdTp+Gsov6bFE/c/uTXJMVwrfOd3FxxMstEBSu8WHmBFCZM12RGjmSY6oolGyWqeVvc0Ci+lwRACcqu+jR9mSJ+OkWcz/tyl03kNd9zDdMUySoex7BDog8RDrsqab9PJbob6pUhRtR8YI7R6fiGlyIQUKx6wcCEPT5rTrVAeTIxHPfem1fjOnOxmy/aIzL3BvSjipmvTXWDSlS4lGnshhWtkR0belR9mnYP05q/bbOD+lz+1xjP+F64TkZVjsZldG16den+yCIe6o9Q+ctolrT/JqHzZ0DscRUF0jyTL2XBcmx2TrD5+533Ua3w7Paqe5Lu7zSsiq3m7545WRL27xoV59rv04HCn0yKDVQoOXVIr5WTXO5qMBIzWszyhQiHWzPXsuI89TRSO1zJ2iJoTj4Y5T6RyvpracxLpT1O37ViBDVEA58wisV6tYmKQpgywr1v0LsyBKf9/YoAH3JsiwjSQckV7L6RncDWLJ+HcjHffKvf+ZE1sMW4FEaMg/ekzzEtmiscduxJpgn2j78l5S8CezzYkuFr+XpDIgR2g0HKJyDOe0TFZAtUqShm5ZOsYh8u8mkcOzrDowRwrdcOm2tqxaiZtHK2NOTixYlXP39BbLPYn6MAPD1e1I/bZ0aArfc3OM61N6JjgtIjm1grDXG/CAAKMctsFJGmAX6DPU6AP8OaPQBWNuKPxtMyUwLsK7Os6yASoc4d16bWh5r7JnAt2YybhLL4Oon8vmGLg3DoMiBtiPIOFm/rnUNHq5IO68nqCoeSqDnQslQVHhwZDDWsNfC8ZpMsWKyaV53XrSxSRV8c3UtimRt8VLhhCIsLN96sq1rClMRDybSUyvTLfEF33zFWko7jtbRpqSIfTBkB0fWjjS9xlsOLiGEp1f2lrtQxFnXbN1oWs1SlOjJNOXJpyVytGyZnquKk7ysZ78maYoVK3keXdckK0qe5PFrN3cYZIk0KNcuBcnKVp9WUvW3lVvKVKDMSFOGA/gAkSozNqAU4Nt9F0YVtK6YHtFRpiqYRMij3cS0iH9gzpSdKvE+qD9iTKFVnYHsI/2NjWWCNqKx0mqv6K+/qan9yRur3XE6ZfvjMQymKk14qafwEJ60WNsHgXIIRqdH03BVKg+hBwUctmk+1tO2xYKkK0W1xCjSuCtDL7c1K4+fMa77fqpY0T8blq95EuFL18BC9RrYiVKwIHxC7LKUhh3UzrWunS4C6NKfwO4vgv/IeLTlAq1WoinA8L4qk2y2r3uZR3QGi1IFOwad1Aj4vJukH1rZrbFJ0NCSaFwvAydzj8y0eHEKrpuYd0Q/qwL7Ao7NXm1dqypiOoxkq+m3C+WwRxY7IzjF+eZFU02OOp0WgaWDPIPhNM/zO9quXa2Qx60PdyuOXoU/o/yEXBVH9arjY4HkfT01qFs4hSeCqjxfxTf/AaQvE9iTJXAkeeKrDIgDGBYMA5JQQLfrh+soonj1LMK0pJHMDAg+dNOO9EF7EuHnBFVt8RK6h3FmV5JbBMyOLNK+rlj1EqVkXse9Cz36s2a8dPaHmW+Ia1XnwNbD54kgb3qiIMazT5JgJcRRfm+1wYvuw/DBN9c6iig45O+wvi4uf+km4wcq0OhPhUdaILafBcU4WeMb8+5Z/hkRhbl70k/sBTouc4Lbsf8SzTszRSFOIoJXljJQA1tn394UKL4k6LlY/UJKKD6md14itObBpYosBerTIRAbYNqPhOC27pFVzmIfb+qPCcgwEmg5uZjIt/wmXxoFZ8GDjaHKcXlh8N1DgS8jqxPNn11ZO++bgAhud5wXnuhAebmjmQTcSdz7XTkkaLq2JotP6po1x6Px0yMVsWFu1RpZ30eNiTtGbisBWkm/sm11kkTByf66PGso1NDfS+VsWjShra4Oe7Z9JYMyWSeFtZs/lKkfKfd+zla66/Lk1IWOIsW3bMNSGyYOyeihndQ9zP5ksMdo3cgk90j3ykK5FYQam0Lc30swivXlvIEWjMeYaw8LosavfcLvbRg9DBQrxO2bo0XD5xOlSXRkKGOy1pQE3k5WpPMlh9Cnq0p7IPJ8+blV+o8upsBWqYD7tNwW29EAP0sdZfOpufSRl+1cM1pNz8I8+dRqBfPRyWNxjNsTmXWT9WZr4KzXl+YBgEXVCZEnyp7xBSlePP4O9gDDRoKJsfCy7QAldUFI8K9cja94aHgeZp5IoASuGl6qDVYA8xDFh7WKswdjeX1uSp0MdW+tl1NQigWv7QGijPE5Kpa+jFIZOuoHecQw26NwtD0KfCstSq0FEkbe5Pg8Ijd+/U6OqrmbQB05wZ8cJt/fkXuJWyEUR31tCmejOI2Lj+YuA1ecDKki02Kn8OjJvBRGha7fzN8l6XHW6Bpu4Vpam0ns5sIBQ+Hd7LiPHxkHxb+pNTzSpBHDTJJFo2H7cUMKIKzr4tVNffZpJL5Sq7ClaTZXBatmK3tSU87yvVyHQJAQLMAS1ANguTKP10AZ7BlyPuqN5SiR79rB5534Jf1bQwm5ykVzYGLcMxAEXAuA49yXpnmLzxnA+qgeIvb7oNu/OVbJXQoai1bQ7/jvJ/+ybFd6HXme0WKWJ7F2uBKeKuRWUYmIaFa015mW0Z9jNowv2SoP2uICCXmzX6HOUZGrFse6+fTJN7sqF6VP/ETrqWzoylx/reBUxhHRE9DvA0bOkKRA2PTpDf/bGE2cor5vIbYrgfDReImleZmz56sApMs4V5cdpVrNUpcjZAzIRDtXQdnKnOiUI4E+4cUduq+KyTH4A0IPWFB58e5zlrnUqsmtZUubS6DUpoXqHIvfpU580nLPoD2sGKbYP0zRQwmtpU6z8CcJ0hPd9rNewJbfM0JKEEGYMbeJig154s4YIWdCb8LYCO5zLdEDSDNv7utqWyg0GpNPgYQLH1CSeXQYSGSS7eHhRYshk2C6dJQcndpLv26vPsUGAiAQtsaOpo2OG+lsaKZXgr5i4lb7QLNGZeEb0fDeehKfHBPRhHBbcVzQb+v3qxtMnkFEDYUFPGqBzRshpArnOZLv+f6gpq/umtaA+iK9Goyu0+V1LCCFns/h2owyvecVs/aTTvNftw2Nq5TTSHjBwZdDSxmWZdlxr+Xws+ssUdnSSy3Tlvl8MdHiImIeVhKFq5N4Y7g0lRfjnBljUM1mCenMro/p4iOF2AYzH1wXf1NRQfnNJKk1i9BCwXaTOPDXQdBj/fLLEWLCupr1YVejPIbRLDkGAn2/FUfmASraTjK/HMQhsvdCpHWU8sAhH+OUzcp1eTjMsbTGuBW6Exdc4zTtHemFyYLgKtex0CxVlRU89jaw67hRvZgsaIiMgA/8Oowmc8KbrD4vIJE9cB/4CxUpV1SbGTMHwiwIqDRwxzzST6UFGIOEEPaFgwh5a7yCrHGrq6ZGpnniK1caXdfZ1NrWW0AbeTrkM6NY9n3WtM3Gpp2fcSNA2HxQQm+AAyY/dqSancS0QvQOCeRI46cBP/B56PqSZ0XHllPY+0ENAfcSYw3BPujqji43Xu23q6eOgAEePa0FirJaddhSCBWWsgV0SOrxeCP9hX2B37OT1zOwvOk5wdNSeNIpdQln+g4qstrioNBcp/TN2hpKvpLWNivM0QIpnYxeW5QNar39HPZnmoOilvH9IQ7slgwnzy1Rbdri5eZgVcoSnQ6In8HwiAVvKsCxcJMdNtps034iGhh4hD8Fs5dl8f3ORLZlNmBsOCtnITmE2Qlsuz4cVfVfjiBYKF8BPs5f/utmD5P0lE/HfmfyTT6Yez8S4GmXWppnTFip4JgHUvsny1qM9vmd7OYJAaAv083t77KoXPBlg89Zcf99b+/T+x706Fy9RgwfM1oKNV+4Y/CEnq9xTgx0Y92LFhHarIa98qbvFpUIgAFf5kk+FXg0ZaNUR9qErsOe3He4aMz5ro4Vpm/y0z8h/7ldSuRRUWxK2tfUPezrz3w2ZTbxdmB88SKSKTqWSrYCwu4LLN5iwTHIjp6NKFHDL91d4Kw/xpHUEDELb+Sa0hrdicdYD5JTNimwIxo/Fy8ZbrrOBvVdUDLPP1DWKb9jwM2OnFQ/VFws8FsgIuYdecQHM13nobFV6owAlZs6dw4HwfES8V3V76FzFseTdmKRqdjz39bX0oIjH9vnxvaDk7Sz2dv9WcHZkD+C9vukqLDv4B0AADW8ZHimyr2rGumdGqKE5hfre/5TqbPAHqk+CbzpieytKlJN0AsryqKxTmtYVECZrsbV4/yjaZlQmHtvvqcU/duGhQo4gok/SstpjwSEyeS04pU7RzbmD8yDct5eU2pZcZPM4iJnxx+Em6P2afjOi/tlkfDB6DPx0f0l4h+DvL/osOHtWNVa06h6l3xpyW+M/k6D9soMhvcKVGbCtyyoevSy50fE929d9PPgKlO6vLn8JPWUFTDvw9L7OZ41EA7662ILOnvfjSpgrCiVI4tZujUNP/bwHX+zq/8KX7q8ZM2m9pw++gnwHl7y4FepYeUQJ8ei9j1+bKBQiLqWLKb4xMc2PaGkW1QkjCwyHJYaSkkt2EQEfsrD/+6JBqW3u5o067pjgTP4oyOxBSIuu+ao0A+swUx8J0pc/U513+veXDqqjy4P7BRC9CDbCbdWxcbnHYAykLBERI0YUGE8yEc8GMt8sWORpmEDTmu26QETPj00PMbKrKnURSUSdc2EwQFJPns0x3Jz0iZ4X7GS6/z7qYnOihfFeBjh6FNgOlGvlTTTJ+06zXniw1CfN4khU2jmpQoyQWFGcfKmXC0zmRtbbpcVaAQ8IdO0hzsBVtijKDu5pAO/Fba3c3aCd2tC/cWkKM4d87Ky7i/1aHaoO+GfvMYICYVbX9Tdq2NC7xraC3nJ/ClddK+yjouOlQ4G6TwKgvngSzwVWBTYZNr5TLAH7yihrP5Md1MJ6l3fsQsrVToSweCe9a83oDI66UyxAa5ysa097doLofRGx1+2na5EJl9lg3vo2IoxGe16HfLkkemNbQbFQadAvuuFiMb15YXxltw8JlvY62pH4jnzEohHrulbBinQ+cCJp1Er83HdwnUNZZVVz3xDTYVczzgBf2eqA4qTRp0g3lvl+6eSKLX10c0WVbK8uFfiryo1JmwNz3gexcDn/RNL89NES79ycangDnFrhFkM4YTUCZz9PvBxiaqvG8nZfp7ntCP0dQbWpD8XoNJUmLZKtGGI1a0ZIrAamUoczTz6d9Dkcb7g6ewMzN+hVjSroFaQKy3Zhwepx+ioNnmJjHvF4JxjLnd2R8b5EjTHgdPEn5YWglzgF481Nb2jYOuTjaX3kkG0FNDdQHLk9Av/I+HG1aaVjFRhh3HdflidhSZWSaAjmjnHNntyBhVOSXhYBaCe5DN5O+TfeofP8qO2hxUDpsGTmAEEoHE8PCpIBZwdfuDt2Tzahyz91cbnna2PsBrpFyh6ZpHFt7bONUFWn/33uLIuiHDj4dRUNNx+zOq2mdMQs+ugRU5kRVNIskmS3PKcdUtHHXNYXvpF4tzJhhoNLKvx+hKgY/W5sYMA4iQYWEwaUkZEIBfL+sJkiqX70vclX+ST8pAQbMZGrsG944g/YWjJOJfAYQpJwcBN/uC8StzKhyXl9WJPEZjuAr/oiy1FjJKx127wKxDaODuQwBjgYy8tiMELagxIJynsPBdmjpIaV/6EjKoXmu1Vc6SgVAU316NifPfxMDYBghnzazMPAFGSZEzLYiM7nLfxW4W8LP0gzFRbrKNuRUonNI8AWnG1m3y97NdHvwvSxApqYBrmaSfz8Ym4XMg3ChAt3zqQKQnrYVihY7d0P2kDbidMheQDpJilLZWPbBTV+o077UZQnHSMFaFyX2iAEMe0DWH6D8CmufDVtAHYKqw5QjBa2TOYSVn6Hni57zCX0RFz/CAgrWf1EVqD9jX9WcrR7OIY4DEZS5eu0esJ4vFbNAjIZ+yrlcshkWZd/SDz/PA3q9pGnUNQ7HFaiZxEDRpo/8Tvb3nG8PV9iyMUwJMnFB39z+6/TnVYgdkwcwcVjOaJSA+91eeXtBpOrvmcB6l+5q5edto24eugzz+OlrG2hFxTpMCDKjpldopSC/VOtPPmCTNkyIq85KKC004ASL2ipkbG27gv9lIclSmG6ss8tpOJ7mos1uGRMEmCFS9RNMDfiXWKn71m2y7xpm+Vsy4KN8ubZcR3km1K+xw6Ew7i4DhfeAquTwOLezgEt7kU6g2OJybVPsYYafFtWN7ox9bGoMPyytMiIi/SXoHC84mOTPnUTDtcxBNO2pVY9G/XQvdkz9W+zqrWPqCinZSvAKynKY04g7KHa1jHZrKpeIQihx9KItkN4Ayd48cR8pXrWCckCP7Y2slpbCfyM0cFM4MFDZyrma5N7YkiEIfeZlLfBvIQ44YwUtd/FClhv4jqkw8Z+6BoI3junb91qitBvlVTq+jRDZ3VKp9ezI56SexmFLq2ILB3Ng6bPZjQD1LemhXUbNNcqpSUb54uiRGZ4SpTlHdYjQPmiYFyQclhKbwQCEujzTxdKpiT/N2COd6oQ20Ih5p/KyDu7bM5gT/a/1zcMM1yTs6uCrSg8ItdCCieHXRux+O51F1/ARSVkEOy6ZTBUetG9af5KhrNDbA0lCIM8dsRZXGVWOpXsbAlkwMOKaYZ8Rbog7X4nNv4zquC6HE9ChlDt0JI41rIEUyiztF9XHmovEc2IAvj3uIHCigdNx6K8+Fj3DUQF41vaJQXGUIer7QVnCTwn6k4wd4SHd1VAeyY4NdbB97WBDtRmflN1kkp5Uaf8VU8vqAr0upFAJBb5IcqDa6b0aGQGUGdFwwo2jvmrHGW5kjIgRx8PACWNY4gADjIZcckH3IBQRKXzc05p1/0GdBVJCeLDlwEs8TZScev3JSd47aWF7JtnHIulkMrjkYqry8dXfOIPDHY7XylzlyMDgZsWkF3KAy9/pNTbtlaYenAXTs9KeMzwyL+kiusdQrshiTzFHUBPO0hZGDaqtIYT+LvgcB6xdrJ4qwWsAn8ecakU1VuK+7ga6Rik1jbpeXDXZJpVGzxAPOvy1MTXUuvyChuyDdpo7H0xuGOAempAs/xGaBlO7dsO6McHZK1scNC51KBGqQVzezz1yvO3KJGUTmSsVkCL54PqZ9tytuU+QsACZ5D0JyM37GzPC7zqhV2a/2zh1nSp0Knh6wg/WIMXkDDD2Hm+G78inU04exDuEl6JdoIygV8QIQmhwUNEF1GmsNJx4b20SE1X0PrKw6wbwcOEjkDdZIZBJbQSEcjbzXjwEeklL4RJ/LUQ8Ht1hK67zHBSt3S9WosXVTJPKPOwlnfkV6tOnewky2hLYoYz887hxqqj5hYY/sbL7RiYdo11Ff8feUIiTSnhjPVW7njXq3qNlkoGurKfS4s877XgumIpskS8kO4CCbK6en3S7mjlBKtMLFqa+enQHivBPXJYT9CaSJD1cVFq+to8p241bvKfCGCNdWpiHmWQ7xtHd/+SE1LKL5eZf68EbjG7lPeVVoSEja9eJuxpZqR5Vpc3M3gI6EvyqytWezLe9vXFtgZrjOBYO5vZUGKd1sXQ0UZS37iD+wtYa1MzEdwY3eWb/M+E1jKak1wlrKeiX6dnxuN2ygAMyjkII8KPL1yQD2oUx5+KHdZXX4GqBJQi9zJljRSq+3J+mLNsJItjqdIjawIqe2T8G3Wh8sPJExjxPHlXh6PF+d0NjALRr1zoVhE1qq8YJegdBceTCyUaGWwMGQ34IJViHeglxJh2Km8fi6/vGOuTOFW6PprOgMuUvBhLE8pSkHZ7TUwZ+ti90sVSFrXKocqd6laDEuf7c3RpFwk5dqEhBlvsiq2Cdsj5HuCTETLh+EPEpfOj5UrLrSSKx3xVktJ/OBw+veLLtyV88e7DkQTDAZQq/Xz4btKrZf5rlni0gK28aIdNBJ46CqIqeyLvE9JCQLM6Ua4WRyoTU+CXbQuKxGKY0/Z0i5Rv2HQq+D8q4PU5D1Ua/tcT8LGWSNn4q+8lJ/JarpIT5Uv7HatXxW6YyZr5noOZKVxw5FxJ/OgcNRlYcqgMyv7D/pPlrnk4OidWoR89pK02A+1cwQ7rA3O8Kpl33G/N9LvQgTkObZoeBPZO/vBxdmXVzgMp7mFxJsXljLI8nF08hwkXccFvIWHXK6xt4dv8BEgWm3G8R2QTcotWTWrxB7F7KvKtD9q0XsnkI9tWgcxt3UMFhpWZ9UYK5TD4btcShM9FOiAN/vVqU3OTwc+Uz79ZhnG4Xju+g35Yx2Mn/AdPGIMY2IPRZKbbvEotDHom87qcnqJImL7GlEpYEd+YL3MDAyip54/it1F5fFh77xhX2OppJqnAsPR799DXAVJIuWPa4kWx/LAuwhHu1TTB2hqT4xiRM7Y4aHWNdcGUrRZeucym1SrF09gPzS9OcmcQYtA6X7EZ9kWgSnxXSp5vFavX9VEUYgQzFwqP5CEl1qVbO25ksgV6mDeni0/wNgErPs87rOl8j3FzLHJCVL0a73kbQ/zdiHidZ9zvzus2qmf0KCOqnxcBaMw9q3GctAu2o1wggCIDpSmXO8NCmudpOsXyD2DOhdoBSsnGHpn384CgpUexFmi5JBOB+4r9V/+XLPhKRmSjtCGhr5DKcvPr9ndvXEdVOYLmaMKvjDuE+vYfkThEYUTeSWjGuT16CDzalWtWD5wa/NtJo9KvtfBPnO8C2lhOL5LtUieyvJr5VCbfGKBH/BTPnFZlWUzobQG8Psrp8uEtS+55mID+paLsSUmLryJH2mXTNM1pQMsGlMtVCQIJ04Ll6AM4MdhrWSgAcSxGgaPSu4VsiHDjRfAb2XMeMVyByiHbzFmb2C/DHHj3RYiwIndjXM2uMmg1kbzHGl0DYBv+aM7NPvO/1Drh6Dhd0xxBBAfSjhWNB4RTNuNtfvJE8x5tLGBXrFcVCQdab1Gc5ewMN4PPyH05uyY8KaIlLFzT9MsMJ8fl9SWjYc5NVGdqBu8CAZBZKKdBDsTYLDXiZAlifk5q0L5WLPvfUbHoVMGjWQGJ5SyC6IdFmLb+rFZpG1JYLQRsK/fdlWy63nQ8X0xscV/kNpssjQSVMgbU8+ZUnoizjUYYuNgw+ymWVhcAjZ4U3zyrcD1vlnFm/Ino8MnM9uPyEWShJbpBoBo5ZWB2feEYa2UUXbUkJ+cLBP5XIkCgoyQfVDhR+P2Cr/FzfmtIEL2YnR9XzLgZuQqQustNHtvEQfjB51XRFagu3EyFnm4VIJ21D5kKkDyIjXvEKTjZIiCoYnMAXibawCI+RE6yesZxegrzpb7QrH8rUmkiAWHPGlQCRgLtZ9W5Rob1Dn32TYppQDShxE1uLC4ruFaeIHYErhJVITTadgQbv85TP4DnzKnPawyryhbDasfM7JOXkJhTqwnH3ZwGL1Ow13ij8VowaBwI5IOvVLPn22QfONGghDfAX1m6ACFWarIGMl/9Il3g9lG50ZieZ+Ljd/5NIeZ9/SjxOm7xmarLfd5usuU1CH81jU12w3PjbLe5BAB9Y0a5rGb2HxC8k5NhHbHxOMXv5htXxLuHUsF+WKKVMbT46TfRKCrffkauqIab0GTgejmhuoK3FGoltsvZpPeFVAdcL45gsYQVryQxqdh/AEvTZnPYzMCyesg3RSjK6FW34MduLZlG0sYtBmnI/kIIYdly2YUIpSNOqXMJa6bEfUaN3DKxeqW1URdBNoceodTcFz7PheOL07BUc5cV/JKVudGQPh70r6tRwyloBm2nHd34JahregPhYcAg2faxMNjyf2ddkn6YTaTkMvvVrK9bs++VLa2co7UEtvWi929mca2eEfiKWAre+Pz1s1Yzzfq1ooaU3Ra35SXBQIp/V24i30rB7HLq9hvWMCzQcBvF34Iqyi5zx1M0t7+qgkb6dHar+3pKWsezK/d8K2SO3WUNDwDLvaMd/W5L3GEY9L0xu13+maQYND1DgyzwkicKFE49kTudH+Y2SOYN7JhY5vaVwhXSwVgP+aVIBeng55jY1dq4A2EiC7w5ng7FU7q1jKLOiZvb18P3PWdRIt6onQl0SJPUeuZ5dut2Rw7jqJr83KIrT+haUzu7JPrcJE1cJKkwtZ2LcrTH1YDfgKSqOdzPhoZgNeDdcfvl+HGk/F9MCQqXN+Z/jPF+ccriXjSN8qcaZaQpgWT4+WhDsZLFcqsbxSNlPeoJgp2YLT4q3supAftBBAjJD3e9H2L2zcAt8ABAa6R0XKmcKfxCpaHYAA7q4kVrU3XRonVtw7MKrKiKg58cI5yEZm49Usxavq+e85xvX4O+Fh4pfyo97Dt73aGYqURJ166vgmmkMiXi7cLKNaN0/O62IoHE6k/SzC+MVxN9tW/70KKd42J030c/7azjoQ28NA5znnS2yxYZGPZmlzr8qvkKMNjBf/FcwBqAkCa5Fpyj5wd/Vm6iroeh/ORWlfMr9NwWxzC7zi9zuaWGBh8fMOeH2o81xYmN1Ff/whnxssmraMlEEF4beI52FbCDB/Kp3YyE3j4Scp4opAeIK/YH5YwXXjg11OBZ9EOA0M4VSl3w4sJUJD/FJ+6p9bZ1+M7sMxuPuYvRyYVb8FrHS4Me5n5uwImyg4D7xCqnC/8oWq9fIo1ps+gi+fl1XBin4L0k4+jc3eyKJ4Z+GDQ4+ewVZ3YChqprK204bqMqi9hbzvWFLp61arjs1CF5rjQOFa2sDLKEA5KTArB0lGtRmLSBbSPplK/Yi4AX7F6KwZZk4v2aUVPTpAm0cNPjgpOIl2ZR1dtuhJ83uJBfQ+3GoLqAfQOz6RqpsIoQ11EiLV98W60B1OAEKMQig+twrwJse/66rlWvd5RiAliGBTvG1bMxJIy3037jpMfaPuKTRnboEl1YVSarT1NLAo9DhuIC2/lDnnkW65Q20Yblx87CfRjCU1t5G+NlupXofDEejlJ82vB7BN/YZP0TILbrie2A431xBfzKrpZAzKm3VWZdBQAuJTr3mSpmvwZ6O12Eq+snFG9v9+VXponAcWj3YMH4c5R3ZAF5v3xe12QK+8xsjFlpdez3D/2VQRvfyvDCxYmwf7WURX4O548+XBEGVc4DifoUebdMuTr22uF8ayZBNqDECwyrnce4uqbG8eGvXPkyC0NYoOgVJapO6UHDQZwUQgIrnedbLLMe+t57yRCZ8TxvI1y87Uz379b3+3A4sDUj2NmpOQJRp0LFujZvqzBIOU+Ms1rbhxRoNwILnGcD7kWEKeK+GZPgIyzAkH1z/DN3Ka84QBI9DPLW9JHF0FnzTUTgZA00WoNdPOUpy6vil4I9kUMgFheVFmrcfTohc1FUkcZath4k4uxyLxgL8N+/wDqx+DIvcyWuiKKBrLOA820er9YBzZeFMmiOvA8613HjhuwJh5jmqiHgvdACeI9PdPLLONGrIXOoLkJOWbVu/Z9AUr+gY9UwdedMoxkGFP+eBU9JRcKD5ae8NHpqHjV0kHje1FWDtVerMJsCCdwzFRPS2hWMdrjSBqLKECFWYBxdZUGMH0QFrAq9iw5tnKon75PXHDMQf9TCkSRB8XYKC8fQVhmCfgEQgcYHGrN4s8VRt6K2Izd99WEOFW1w2c3rMYOHOygHPCTlNQ7c4+XMX/K0adagwUOeP37QeW61w+/fouJidZKFfYuLkG8zklVNNUD5y2g1RLfTQuF7BkCQPEdFjOA43p4PELY28RTGwB+rj4fDpTU2Lk1s4s6nTt9peaBhewpBweD/sXNMsDet60eZXk5TmN5oysCoY3UQsXysm4K0sbLV5SVH3Y6PqQ9mHYTqtSnvKIyNeRPXzyDzTQHNq9Lktt+KSq1nP6w367d+gUzKSGhRrgEyg3YFz9PyWsxDJsfYYLHGpEsVRkzllu6UMJuGuYfAhQdma59/LLJ4RmgpvhsVZVvH49wwLyoYiO/bhvwfLdENqEJEw944463+P4uVezTLqdNcu8tONWMq5Af7vdQvpBpwdGkToCWw6My47t0vcDhZJOyV3bLzN9HEiXacClPNGf6BwzOc4LFt6oED+IMry+66syYKtGnEEfgfDQjyicvShQWGDDv2nQ/IvIpKI7aXx/B/1hM9rD3Raf6ldqEJnWQ8H0wJCQ8YKDtecl1oRx5Hf8sYKBsMBbBgQxBvtd9AuOJ1tGJ7fzr9Myr+jzVZE8DtVNFXkcj9AscbLpNRzaNUS7Nd1SahOgjtGSorS9L7Jlq1LZQ3U+Jpzsb9FKhXnYJYeE8cpK0qHzXm/fXxwT/yBK31u2QszSVGlFjv8KNI7PA1Yz9PhCI8R3oSajPcEJqbYn3cmK0XXeS761T7HUFfrW0er+FdOVALEntGTJnLb66iz0CQVoFppBgaYaW2ahMfiTrRH5drlQAEEbfFwHG83ueTcNatlkdVa2lpj6ERHxsaSXfbFRbtPQkPi/qMU/lUqImSwnycliY4NUfoRez88WMxputU8gSIeIOQk1pKyvLkT1qRrLDOkl6vO6nLoJXMNtfZSXJtMY3cjkr9IW2bOsXr5/o/KLGWXEjtI+FVSV2d7Kek9oPcOSQ6CyXhyczJgJii54cO8HkwZirG32yPC1a7Mkbod02ahCMiW4b+4lGbjyE7vvcGccg1ENyOtX08y+xbitEmU8+6j/4vXh3jk5EjWbOCmFa9bl6b3UQDXwA5H6XZRgvpH1tGecAMOldtF9pH0aOpHZ5q3YKBlFVle70YfjYIB5atuF735uf3+ckqNeNM4Jb0sRpTLTIlPdFc46174L4scQzK6qdn0KIn4VauOt73LRHjmn0CaN6+whgEXwZx4V279KcQRTHq4rJ4oW+MY4aUQcRD6pqY1btc+OLr4BRyKfDz4SgQPdxTiPul3MG5XeBRfFjLBKv5gnsdWxS2hn+/kaClEs8VOCMuj1pkSuCwpx16ciEDpeNNLu+Wf9y+5lrTKejsnPTjnpgnGTK7impxjD34XoUczowzZGO+qgLl82uQtXK1/fPqlDsZFAbTDY94G7GjOhTCGOu1kEcD0HeUsp4fEFHlIMe5sCYaKYBI9bCWfQ3qwOyFfF2srgL6no+8E7xHXsGTgkV30KzbG8JnJ9kkGCiZ2TLtChEp0Y26V3jRBCSXAyWiuM/1qzyk9aUnRuT61bg7wImajQvqouoWJ3aq2jAdf1xr9KWQ4FQPAj93LIqDGaHfFX0JfbRmiV287C9V5joGazglGRZ9UfBg9mC+M79nDpvhg+9vVGLPQq90gInAaxxa0AJU14tNq8PXZgSPIuSn1SSUbBq08i30P0hztCdmWotV23K8WrkdEQhxEt91BiE4TmUOAsNJ16iaeu8xMniONeJphnyZrPMWbQqMaKV63PpAJMmwaVsNpagO16Qpm/+CpUZcW2L+ZgTj70CpCjBXApw1cHeyTnuyDdtvCvfxjQYM4pkgSxVtaN6XfdIhT4X1H0+KFq395SVKSr2UQsVztSCY3ws44qeyarscc6LP15MW6MfswLnsM/fDq0m4/gKM/vWM5fQnVpIvVI9X959ZvXvB50MCaFZp12zdZxAUZ8lFC9fTWxgLrPVa4eEKVAA/rQadkaLWYdjS1oMoDQPFP/Zkja5MUvLDhR21voITc3ygtnW5+qx35rwr1Y2+SbbBPbqEeUqlk0ht12CEU8Ge9/oLX6F141XlpfaHsAWv6sAL1UUAFTkTYtWWyBkIijoQ30o+aouOpFmazLxVi0hUSHC7GC5jLESpQ3A04PZ3yW1pNp5Zn6E7FskdHV4fr2ouhAgWeBk2S+Jfd+t9bIN26+h/ejOCwj7YFyaxZGuFz35SDtiVKY9x3wYRkLyo+pr7+s+GCdkeG6m8ivhZPBdJI1tHkcLeVonNHLL5Snghf5DZz4+wsGARtmlw0YARLviRLpDmCyrwofGdtYBg236bMPepiH+6cN+duX+F2wplvMAWHgMlW+jNgcC/hbgKO1dpLFbn0WK750hkxOs1n03eHV8KzeqRbVnOFcEls1gwgWfkUvIpP9sR3hE3OeAjm0CekYY1akqKxXszGgj3He233/cxGEgsP14H6V/k1qzwOB+uJbKS7LLhPpoOyzU3Vrz4Y6FlwCjjr9j3KB6NmbcOwbC7e+TDjMGzAE8OL91JStMhFJ5q5bEoapiWZMy/YEJ+OPOA4GVJxYPb8E9QMJdqCdvIQOqYuObmoDawQk5hwm+EW8xkaYUxHCNp0CYHnOHYMNRLpULnVwePvk73C/bCVHNknpJ/DDF0CQt1Vp6Kv6U2fj7BwMoYZ1BjOGuoCjJ0eaogjvbXGEpPGtVZkkxSvKpiF4EI5h7WinzY5S5plrPnWgo5X1xj3sZaewPH49V8LdHkHl/E4EHFjfx6IFVMFvJqApZ2fmCsrWu0kXNjo4uEh8hWN4VePFtODSthRAkjUWliQ5+uGLg49e9O02o3DDwOu+LMBc7SPnZUVpl2VtT32ynCxqCWH8+IMRbmeqHntLzHJrL17KxZztFAKHS9WLjhqT8eGGQLE2NiizCfKQaviuX0ivJkZ0/K0iSOTMhghPdVh2XsTDBC2MC4/YubjA19eGjPI+8zaoh7wtT6Ks5Ke5Snaln5JJdDinX7RY+punJV0hqEAn0LsvH7E8YkKjYctXGS+mVSC/sSgO/OHVah6kDzFqPpNnXaB3jRV0oeJe61i85JpZp1oVmnGgaIOWHXXj0ga4eSZQVOZbM/f6VAwqeQQhLHq50Gri8N5/yrgGo4/X7NrAy1DVd4O+0GaaZMdHDz43vHdZe2JDjYa1BZ9lZ6feMssjAYWWPzJxMaCjF6lHf+BzFfhnOiVqqUKH0aWUHySh4sGid5JXTUfrNEqe6JtLirHxhmDa5SqeFNbiYakqI5gDbWbuyRtWlgES+8Glj4ktDMPUrDtTRShEcSXW1wssmHdNZzy8hW9IyhwaiZqspai4WzS49Q3Kt24GQZr+UpV7Y6B5PWgvBzFNTwokm/+u96pPjpi5emj570vgPioJ7++Ep8WHwTATiYGnuYJb4SDEMASIuGyJvBLCIRRHk+bsu4eMsDEalhnqGS+G745SHoklJnnH1WLnynaP6PzNkwFvtJOQY425dLhSFPJwZEbe+jT7grDwI1UHyMYEoCenXQ8Nurg61XhyqK9dKtW88Zjt1Gh8T5svjX8vhomg7FljEAlg3m8B519vVF1lb/Y9c1HOgWr40iW9srWimXcKF9OoJsgs/4QbHWfbmysLZCrHs7Yc2kdt8laHm3kV/Yuwo1vSi21LyM+pb7y3ihqblkQLAQAjzllPwUCmiqokDPiV2H5ltcHuABo5oCRnhuAUSwiU9kWIl9cjShlv2QBzFAejYaUOWrNHywPj1Mpys4DJay18aWsYXHIKXZjNdLt4I2mRDxmIQ730IwgS3zQhJuRbab+fPQ4c4GM3TwkuCNn8PAOPgcTgy/OWmCeLXK8uIZJ2D6zgclD5sde1iK+O40HtwMshk+aVUpLzk0cK4Xn+2hoASKRnkL0NbG4c6jmHEXV7k8fgI4Ps8W63a8KgFqgE3JjGcMLMS2RA1ELHG9wz2qjBj664K3Qcfw7UXvNqaWuYVyLkcXvade4thhJS+WRCGzIb/rfLpbtMk8g7M2Syarx8tZjhtSjw8GCm+tPToZclsCDaOw5l6WPmAlwqr9cv/V9x1Nb1qJNFfZJcAkR7JIgqEiG/kzAAi//plvlt23WvfXa+3XLWlR0AaDT3dfTqcJsd2Ni0xUHjySQW54lY8K/YpTJGshXYuJlnDHCJdHngwWw6koGRteY0poZZvYUPfioYOCnA3/cReI3TWXgmLc4XL8u95wxA6rQnyHdwM/lXcnixbyYd85/24+kx5CVY1L3Uxm25F52pOJy07IOVA2igfzL7NaoHzdpQbV6lG8yqQgD27Oll5sBrrS5UHqUFnOqxcfdeVT37B0FSeLwna98sCgzHqlKEuIpG6jeq0eSKZrRrVQ62Yf1Cgoo5/7Jkprdr2AFHF3O904Kt2Nyym+UR3LpQXmF81ct3SbH+TFPON2qWDxGS9XdDtzmqu2Gc2bTPdSD1ZldPKeeixuzwqyOWkgjUfcXOErYIrkCaRpCXYP79n6qbShAXs2B0RQh35z4nopSLYbHucU4YMlxfXI7bTKhKfzNLN5/QLquHce+DcR7DcFna+bHGA1Yd6OWYxGy5d75AsL8Hogxhch/ebPcHk46tB7R7VAWqYoVPxDM4CzfCm9sB15D1ze/va9O64VEGRF6gQlp1ecZaLF32+2c3dahJHBN0qPAkmODoYtsJMNxm+So0XxTL4FMTGcZei2c/3gQM4PEDuZSUu3W3NtdS8JBEGNdfGJ49tUjEKF7lL4aqh4lhDAkHJWCrDVFZdJzqIz45NjbSXycz5ouaIeXvaOMKrRJhW6NI/ByyN5awIsEOtwJjkb+C4hA3G46Mplhau2ORCJEJAeHp/yaTCe/WtwK+dVdquYzKAgVD+5JpqivmGLSqXpdsxN9atUJdmd8Lad4c4tEdHgMok1rdxjPsCVxJo54QWgeEu0cmej2enUMRGelzjDjV2nqL+2OhH92FsXSwtanZ0i2Cdz4NzGvawiYnSBysd3wVte+DFOJWbbBfc56Rzv+ntXsRxHpJGtQd7ksFGqE158Pu9eVC+NSgBI7nSIlRcwkpikPlq63hOU3gfWRBATayqurxXtlJmErMa5HJzB16PuEFkF4KMJI6LYPH79EJmwTnGkPg8MPI9XsAq4H2/X9KJYLyxtbweHcX8uAPVKPwRGGhJss/+JM/TeJJ+Pw0xnuRPX7uHNneXsuy9CUNsVyDRFIho5bLZlV22YOT+KWyoLQkdyhsREI21SMNFYmHblcdsdH/qLaL3MUORimrLZ7kDEPrnacdnfpKHXlvufJc+htHoZZykkD/qwh89UsSfSuNDNUUq+DYMqPGhRSn4iWWRRSdZrfEKpjgUbb7EqWXXSGbGpWN6ty5dLAzLuQGY0QxSyqvZ6jdu2QXSXIHgINuuwSL9wSTmY9b6UKotEMJ3vTWC3I50NGeqeX9f34yNhmrwcgPrwkG8yU2kwHX4NEfP742SEyB9SMEYG0kKep6Tp6hmbISafQclWrHRvMtNe3E8juFJYUj8G8ZlLeeWPUrno7hdt08R6rZHNI3jVDnAxM2bVsDcjPBQt88lpFTH7ynR2oVu48my3IwXYqQkHb0I1MMve8sksuWMpVD5xtAArxJ5PNR6aDTM1dJJ5pMClTR9JJI+X2kFvMMjdByJ9T7Kk4jHSjlHwOeXs50jPvcnBqIOnNS04QPhazdZ5lIioGYccbyTsGsXi8D8USz7QuYQuyHshyM38FqDSsqdNCZhenQ1Ujz3SE6/jdyUHV+sNRDfpN8UFdDmPJUEs5JU6bb4GkC9MAeHnL0feRrBaDzc736sph4+Nn+2FMKuWOpyofCTPEZgLmix3ydsFEkzMSFhtYoIZMgEISInEo09dEAa853RJ5iqPdQ7ezyatt6MBl2pnpfvL2vB4zmbBljbNbPeWRO8+2DiMOWInC+btX/UrZ9YVYe/XJrDQcLh6x7Gbzk0qCNMqO6R+R2e5tiwrB/XiYeKytTW3Mwt2/j0hXK5hLGWXeMEPnRf65lMl36uL39nt2DyfeFed6dJWBB+VeflJHO2kDvOtPpTvVE9Xpb49oWPx2rRvupOclWe6zA2KjbnT4ZkuxQRPVGl3kJhCpE8EmoBdL7bD5yupyenaneg2YimXaZ3QckZnfyw4CtFkE/fQU4yomHxttsRZM1yhPJOMPrgWlTh8h5bvSkySKSUWvU6EB6eRp4G49BM67WToXGv4FWkNQ2Q/vr9EBSjujPkeK0VCVLCSmeifDyzUb1//BNPunC1YWt7ojeHX/hDXFw2sc6eNVWisKRo9ARKVqO3M9tNKSePHVcMs4SVjvIQpvtZ8q37qO93J1vYpEBKjns4o72/kjKswDdXUO7J6EmcVkfIMSKPXg8az1qKzJ4E2jhzDPprJ4VJBp3cPbrAnpmq4cHaHzhihN2jIeSncsHF0LDnPkgkZ5NNz034xXErtJnl5KVX4tiPQHWdGers8HIr59EwtVFemScPvcF45VCGiG9wU1nIS8YIrfhu7MXqOO7/RyKJ3X69UT/Q1SE0+We6utuvyE8I69B/gEnyF6oo9ym7LRcS7qiPR/Ga9B+mhw2Q7PaCVp8fuEwH5Hvq0p+QnXbRBEfufHEU3iDR6e3bnd9IUOcpm5Pyt+kuUxe1v98AB3f98tt//v76H5lS/zbfKvU35uD99Ri84d/Srs7fKBL/+0F3XZWm8OGfkjT+OFHvD0KLEvDz13SN/wz96f1HsUXvfxZb5CfcmP+D0MLw/u/jGr+ufTf0EhP+BQ== \ No newline at end of file diff --git a/experience-kits/images/dek-component-diagram.png b/experience-kits/images/dek-component-diagram.png new file mode 100644 index 0000000..8152c61 Binary files /dev/null and b/experience-kits/images/dek-component-diagram.png differ diff --git a/experience-kits/images/dek-workflow-diagram.drawio b/experience-kits/images/dek-workflow-diagram.drawio new file mode 100644 index 0000000..7773a9c --- /dev/null +++ b/experience-kits/images/dek-workflow-diagram.drawio @@ -0,0 +1 @@ +7Vxbc5tIFv41qtrZKlHQ3B9l+RLv2jNJlGSz++JC0JIYA800jS3Pr9++gAR0I8mxsB3HcpURp6GBPt+5HzQyp+n6Agf56hpFMBkBPVqPzNMRAKZj2HTDKA81RfcEZYnjSND0LWEW/w0F0aipZRzBoqIJEkEoIXHeJoYoy2BIWrQAY3TfPmyBkqhFyIMllAizMEhk6n/iiKwE1bP1Lf0DjJer+sqGXo2kQX1wRShWQYTuGyTzbGROMUJEfEvXU5iw1Wuvy3nP6ObGMMzIISd8Moswm/+eXf/v25+5bp78y43KsWOJae6CpKyeeEZgTinVQEEe6qXAqMwiyGYzRubJ/SomcJYHIRu9p9yntBVJk2o4CeYwOQnC2yU/bYoShOlQhjJ6/MkCZeQ8SOOEweIyIzD5g9KB/gWuSTVcQQGY1X49wwiY5/xD6QXB6BaqRhZxkjTo3snkzHIofYmDKIbb2ar7CZJ4mdHdkA5BzAg4rG7Apnt3EJOYYmJSHUYQe1qZARVP2OFw3SBVDLmAKIUEP9BDqlHLtTXd1befSlwqaRkbuu0Kyv0WfMC3NQcI8qqBPacCWlBBfrm53BYV9EsFjMeAxOwDifmWQKLrvj2dvD6QGK6v2ba//biHgMR2tJr8PCABfSABbwkk3rl9OgGvDyRjgzLc8w5SH75Wm+XnQYbehwzjLSHDMlzgvUL1MXYdS9N1a6s/DgCJ7erPiBDLkxDyNGQcznsDqFgjkIVwBHGXixumARU4TvjfkZwDx1VItCEzyzQUzLIMZyhuue/cUllpoHmWtcNKKzgHVGJmgcHkzHnnnMp0OkAzZdf7YEkbjF9Atpzv/KIGzTV3GzQF5+xakz4P5wwV55ykXmSWiRA8otS/SpYCOOmyYzPQ4ndNZLOMC77iE+ZJgXzdPMNZVlt+ySIPMuUs843zNA4Fs9hkeDn/B7CpPNCHmDIHvv39N7bDb6tzjultDmt9/a3/zuY1YfrxK59IkOmaz7eH6iaO6P8LmEEckBixZ+GrNZqC0eSM7n2HlFjvXLAcD9Ad0/R+p9uPGIWwKOiNbicXC9K5FfwTLts1TBGFa+/Ksfn+yWYDF2z39PL6+i0uw4eAIyTC8R1LFk746fKCMImcTb6w4dnstBBXQFzkFgwf+hwhEmdLphEyNmHAnhuGiO+IQ6KAMGoYhCt+5NtbzBkKbyEp+kHFlpHK6zhHeZkEBLLFKaqT3uB6/A7JPcK3DA5RkFNDuGdp2qrpjKwgziBT+duJJvVEdNwz9PH3799YVuENLt4fOdfZXKaKh4LAVCxekDLXJJsXeeOszmJSzX0XF1Tfi9Nzoci5XRcWOCtIwL99nZcZKdlj6BrLXO9cR0oWNrgmdzyqOhK/Ys7OR1TE3OSYp2kcRQn3dYpclBoW8Zp5Xk0PqxvGb0+qCAlcsKuikiRxRl2kumzBlr5YBTm7iXS9ZOUULYwJjteaUd6A8qaA+I47YZ28wyYfkXTuFwtHh6EkpCt4xS98aunH8sJMSzMbWcmOF+bajuSEuZZmKHKSwB/KCzP2eWFPdKw2aL2IyYdyziQ7DKnYEVlm6RH9gtwG/mEwjYJixf1+vQ9ODVh20NcHlb3Q74V3F7r3cJ6gJSq0ZUxWdGX2xh5tlNpHQymwNdNoQ9NxPDk+cGRc2kNlUAy5oDIQLvf5hxN2sQQGBdta3Ef8PLnuB6rS4zpwelO/2GHLNqd9nfXZg0VCIb+5qDQTN1eQxaoN929z/lWclWt2ckyD03heijhCeZ2iDFeC7bJlYZZvSi+B2KCr+cyBZG7h5w9nV3TjaQc84SnzlbD64ntP/vhAVjwCMjWnvjhzwnrmm3KYE/Y81BFhRvXhv5NrdqspisoEbs0oc+P2Xp3JbeXNBKGwxj+uwXIUZ4TLln0ysk/3WNaOCtuU05mmaOnCOOU1/Xp7GqfUezhPYnp/5/Q5YxjdUOUDi5je/HmI0rykz3RDHX5KxmmcBQkb+EjdM0S/30yrI7TibrlfjQ2gWYVi/MLqAcJLGMieKzSlXxMamtJxNcOSlaVvDKUs5RaFa8gR+Bn+VcYYppChqAsuIvjRAFA7KVVXiRrOVF+hppc1qmzbNh+n70dLswhVH96qSDveiX0k/tqepVlmh7++Iidt6JqnqA8Ol+U8oPrDgpm8j4ftFOg2E6nvkJ5WorLu7AGPWupN01Ewr29T380CAziaCxous9Nmh6cqEZiGZityl8Zg1TggVwmmV5cSS9pLZ+xUyZJINcWy4zmm1DqWuUbXlgTUpcWFdh9nfN6uCVCXTNvq0WuDBtSC5YWLRXvIFEOhxf5kWXT4h9JTeldCfE/bEkyZt68m3UbqPo+4m0Sf8o8So7sFa7+SeFnAqQqKwoXIlb4wC3XG9b0xt4+3C9pKf1gkGmrXTnJoPpd9XmDezj4UIY5zdQi35xInZcwT0qrrHO0ihz5Hb+Il73XTnm5Je+R1EDvKC1d7tP7x5aceBZrtNvS7Z7UUvKVL6t3wFBXFTRPp8UXNP9TWDmgGbYc6mo2mR9B1OuXMkU8NpykvlGkPtE72/mX6ofxLu5LaiAMkZ7MbKMwRISjtt3pPzb8cKhN+Dwx62e0YlHUNbtedxc2yusLdHIy14ADewmgJZ9UuwjTgXrJQ8GxL7ain7TFXiOscxvM/ISEPlVYKSoJGTynFP5I9BSpxVV/uObB6aBLgJdw1Y7VabEV2chvDJCA8GdTQDMdnHuh1FTY2b5rwkp7SGIocyCm8gwnKWc5EP1vTbQyzkA38O+aleUwFTbLBQpCoX5qyukIaYDJmizKm82RvXTl0Ug7uIPripRSCKScYJEydi9SjElNV/qzuTBBpJ2WuC+KzOyhSXrvjpcf4WjycsSaeMwBwJGQeVLa6gzBItbKYPzbi6Sq9bictf1Tg2ZPjpESAY2lNb83vtID5Cn/NczTdV4BzKIfNlK3VZVbQRxxt8uAUgNS4iGYFnjzXc4SrHLjw9sV/VdiA+8KGDnz34ucxNdGnYOtpJdHNteQkwTHwRB1bpwsiz1Q4/TrQFN6sNRSILJWGe2avX7k4LlAvTjNH5ityEfZg4qbqkO2G2pC1CfUYA1nmqKXdymjVAvFzi5fjnICni9dupO4XuocaLi8vXgdEFO0EdjPhrDTwAwqi5Zua03mtwNflivjmdY9WZsL3taH8MEvu1njPO/+MeeeNPDwu76zEm2tpvm4Zum97bDtUXGn1d2Rs48oVDG85FgNS9hS+dbSQIoK96dTDC44YFvHflaAzfCsq6CVBRYWLR2Rqu573cZJFjwBCrZzaislwPZY8MkzH9xy2letjhqOIFQ1vKJz09/WXyUhVq5ijNWvb4clnVqqY87cnxpQ8qtst8yCK6PhYqBdez2A+QjWaBngZZ2OuZNiYnq+7Y7WbwBuDnMZ4t/mTvXFZN3yyMKb+brjb5k+puGLYvcWVRFRjHv3gfVO9wLsRIYrg4bNUa3Q4UyueaYKjumZJnJW5Ws3Ggt+y6mZ3Dur2iln+Icdoifd35ojn7pSXqrLQuBEqPq0hLYJ5gh5S0ZPFGm70vntsdwrtIzPMPRmFfQKmAZtxaViQSmLmCimrbuqXxbBAzLYZ7Vnh26cCHVepAVvovuf9gkXJ2+IW3BqM5XqrHiZlwbv1hDhgGEQPv7AY/LI4XwRx8kIo794bxJi3kaIwLDHmr/NEJRYdCaJg0gT60aBKiUxK3k7XQU8MRiMo8dk85gCOs6lZ/vaNWM9vN3KajubacoRvK7IkBhjsdwPk3oOqVKfImvF+lqND4o21dBoG2LRFvKqWTlsOk6oKmoLVl/VrXO/s3sluGqdpltHfPfo6OC8n8b5BHC8etow+NC3ya3PbNlzNsu2+JilDr3896SW57RyQ+X6vfL/CyrdtaU7nhU1PUe0GnqZ4X9MYrPp2wC8LFbeQhCsZLVKFTAmfFvD2Kg4BhT7IdZm/DHOgBe5Nju75bK8bAL6mm/2/VuQqqtZAkZ8/hmb5Pp2vo9n35afizPE/G+7lpykZy2ak8zb4TF1J/UnCBpXR4BConkTvQMI6ksfosp866r7k5ShKqHUA8SzMBhKzv4i2xHc2/yCbLU/zOmx2Fe6hoSsiwB9gM93d/pAxH2v8HrR59n8= \ No newline at end of file diff --git a/experience-kits/images/dek-workflow-diagram.png b/experience-kits/images/dek-workflow-diagram.png new file mode 100644 index 0000000..40cdccc Binary files /dev/null and b/experience-kits/images/dek-workflow-diagram.png differ diff --git a/images/dek-component-diagram.drawio b/images/dek-component-diagram.drawio new file mode 100644 index 0000000..5419b19 --- /dev/null +++ b/images/dek-component-diagram.drawio @@ -0,0 +1 @@ +1VnZbts4FP0aAzMPHmjzksdUTpMgK+JpO32kJMoiTJEaivIyX997Jcob3TQFbI8NxBF1SZHSOYfn6todP8wXt4oU2ZNMKO94TrLo+KOO53k9z+vgn5Msm8hVzwQmiiVNyF0Hxuw/aoKOiVYsoeXWQC0l16zYDsZSCBrrrRhRSs63h6WSb69akAm1AuOYcDv6jSU6M1HXcdYdd5RNMrP0sGc6ctIONoEyI4mcb4T8m44fKil108oXIeUIXotLc93nn/SubkxRoT9yQTkffP8nKe6/dZ/vqM6nb9XtddfMMiO8Mg/c8foc5vtUFkTgXeulgaL/b4W3+imVQndTkjMOlF53cNU+yQvo6Xi+H8Axo3xGNYuJ1VOPDnFaIspuSRVLMVR31POWtQBw1qtCr9eE1gSP45woCDs3CbDmOS8FxXv8Y8zEhGPgGfQHh5BXpaYKWt32cSKFyJhpRnRGuSzqETcLODIqYrxwyvSf7RUwvsGgucaQuILDU7ISCUVw8QHmGdN0XJAYe+ewFxAHnXM4c6FJOJsIaHOa4mQzqhAffm3CWuL4lHEeSi5VvYCfEDpMY6RCKzmlGz39eEijdHVTmxpoCYUF6GIjZDRxS2VOtVrCENPbDVopmx3qtoKdr/V+NTSxbEPqQbtHidljk9XkaxlCwyjxN1Tp7VNloyQRlXi4IyqSSN8bnbCyXnPNc0NYCHoiTNQkq9Uoh4gE/t9RnqNQMtBTWQ8opMUw7NgCm/GSM6Ba+b/mOWpE8RitAiSeTmqpvFQapqEm3gh95PY2xBEDi3C7h6HVH+7Q2rNpXVG9SavrDo5Eq2/R+iWqhK4g5jl/OQEcH/8eN9t/RFNScdzrOsOdyYmmJZ5OqRKYY9BFrkXJItz4ePK61JkUxl6WsP+RYLCYGYtpWaeN2pUFqa3jWuN8RLPtS5LmJJHxFIUT7uruMC5wVKKDYc/ev8Eeon3/SDwHFs/jt+79y1c0aKrnUk0b71ZEo6Mh3asBCUW+EBReTZh4B2/3t133EGh7wQ7avo320NmH9rF2Vc9C+/nzqEHVpMNHEsE8kCMRyqooJObQQ+B6UB3vIguvih9E9lhpqG8hG8KTx3LXocLn+wvA09/jC6fFc2Dh+QQAVmWDJ7ZZ0bzGrVziHgFJSe0IlyNdP7Bz7WmhHtrSlZxDgZSsrNXArimnZs0Pp7pzwdlzfAtnr9c7Yaq7+nmqQ1eoEa7tAcobtZnnvkIx4jyYd5lNmZ8b6G5/J+F5to34V6fUdlt27Be3aHIeXaBXUGVUHr5+ad7snmgu68e7IN23iL+n+sEpX/Bc+2uDkCQzVravcxcNt+98AO+TuoxrF8SvCufOaGXbuIM1J8Uy99yRtg09GP7fUNtF6sMQy0f8OkFJXqdPIt4z6nMpBH3P1u3+ij84Vm3i2qVgg6Zx6S58plVU13tKLvBpHiqo6PUF1t3ewE6Me+FuK8bDo22XgmNNdDm6McnQ1Np1DBq4KouRDngb1KrKER7MoaQooM4xX46cm2nsZsN9lnHabGjXibeKpEQQUyiSMoskUcn5YWmluiP6L5yuf+yo+zZ+MvJvfgA= \ No newline at end of file diff --git a/images/dek-component-diagram.png b/images/dek-component-diagram.png new file mode 100644 index 0000000..19e4a10 Binary files /dev/null and b/images/dek-component-diagram.png differ diff --git a/images/overview1.png b/images/overview1.png new file mode 100644 index 0000000..feb6e5f Binary files /dev/null and b/images/overview1.png differ diff --git a/images/pwek-aio.drawio b/images/pwek-aio.drawio new file mode 100644 index 0000000..e540680 --- /dev/null +++ b/images/pwek-aio.drawio @@ -0,0 +1 @@ +7LzHsuRIliX4NSkys8gRMANZgnPOsRkB58TAga8fwN0jsjIjMrO6u6pnFvMi/LlBAVOoXXLuORdq/heY7k9+jqdKHbO8+wsEZOdfYOYvEIQ+f97D6+chiH/gnyPlXGe/xv42YNd3/msQ+DW61Vm+/N2F6zh2az39/WA6DkOern83Fs/zePz9ZcXY/f1dp7jM/zBgp3H3x1G/ztbq1ygIAH87IeR1Wf26Nf75daKPf7v418BSxdl4/IchmP0LTM/juP581Z903r2m+80uP9/H/ZOzvy9szof1P/OG/7veUv3DkFR+spxXZhOtDH/9Ncsed9uvD/xrsev1mwWOql5ze4rT9/h4fPwXmKrWvnuOwOdlvEw/7V7UZ/7cilrWeWxzeuzG+RkdxuF5H/XHxf5253xe8/M/DP1aPJ+Pfb7O13PJb2d/j4lfsfRX/DfTHn/zDPHbRdV/cMrvg/GvaCh/n/1vBnte/LLZn9vva6A2g+UA6ybTqhiL73vnXz/oH+yVZ08A/Toc57Uay3GIO/Zvo9Q8bkP22ooBnqO/XaOM4/TLqk2+rtevbIi3dfx7m+dnvQbv2/+vz6+j8D+cYc5fM/84uH4dFOOwcnFfd++AOKx5pz+egQAnP9dfp3/dD4R+Hf/mw79AMPfj5/VuFU/vRV09tL87+7fMgP/g/uetAICQOPqvYmAZtznN/4WVf0HGGs9lvv6r6365+XXBvwypOe/itd7/Psf/y4MD/pPkQrvX2ssUD38XNeh3G39zw1+XH34gnwtAfDr/dvJ5Vb5///DeX2joLyT7XGP38bw+f78R9vylT/k7tfbg8PPXX5/3xP2bskOyTD/n6H7Oxf1cxM8p/+cWBvx4J/Ak8PrXuKvL4efJ+Ufm/WHZH/45Z8z1Hq/v0vz68UK+LD/y9wkfCHg/Vzk/Z7PnwCK15zf93HYeX0sbXfzG6z9Z/T/k4N8y7E2Kf4dgP1fOdHnxTvYiUv0UAPLX8Pom5X8mff40Xf4xG34/U9Rd9x/GcZSCafKnB9J6KJ2fUAD8bUT5sTzmh9HjOf2VrMh/EbgiBPYP4Ar8EVw/6J+AK4z8d4Er9GfV6WcATH8apA8o5X/9bW0/wpR41vb5Yyj+hzia/nEsq/c/nfwfozx9rJ3P/3zu/xigMr785Ci/gnn6l8H8+/CPpfz3rC6Z//X9HuAASEP8I3wASz7vP2b+h3n++rd07fJ5+fML7LTKs637Z+/P1zT7ZzPPb7oZ9TA8ufBPJqcd413gNk3jC4l/ds2TV0/SlW/oq/HwkLffV/KHQPj74PgvRJh/gyVdnOQdFadt+eMm/8Ck/gE3fq+uf4MEEPxPA9KvOf8eYdCfZf4nWQX/i/DlA+F/jy8YiP4BX0DoT/AFxP/b8AX6A758ePrPqg7wf8DzW5SMp9Re/+f/iyD0L2f5E7j43xDUv5Dmvz2s0f/1qP7z0v53FRf57wh+FIP+ffBDwP/e4P8jOyVV7o3xMftfC4x/4+h/EBoP4cvqx7bMQwfTtR5fv7xDcfdnMfaPHuzrLOvyfx98/xVexMB/8CKK/ich7L+PISF/cKL9/zvxf8yJAPH/thM/f0Jz/8F3rxemfwp0v+HVKw2yeKl+d/J7YMTrY/bhx8hTjP6H7Ph7Xy1OflsJ8K/tixP/36vzf0yS36rhG3D/oUL+qVL+G38U4jl57P62D8p6+bGyf0Gm/37uf/Dnb22U9HroQpbP8L/PyORn+irJ7wO/Z6e+rS/r+DW+/ErMz/9Ekf5DGYbwD/mfLrzPGQJ5//tXSf73Ahf7x3o75XP9uD2f7Z/X/Qrrv280/XkU/+vs+vcY8e8jFPpvilCQ+PcQ8FvI1P2PlvXvBlZepDbGpf6Fu8m4rmP/zyH8cVLx4+efEqI/tHR/3JL8bRT4beR5ncVr/LDcn4cPtXylGV17lG4dgMyXI/n8aLZbsW75vBLfY1qiyfD5m7lZlJHfC+iAEv1AfV4gzXO4iQeHTA4BPa9Lku1Y07OQQQcJOjhcL0n+AlEZun03zNhlaCkrDoRY0/T8SEaQJUZu8U5Q8bixmmaMJTtz/VmYNrYUMAhs1oFC8WWMZpM5kablktdaq6UBRQydJUcL+Ds/uczBGOjttsmi+geQDuEZEbDntj1AQCCszCDoDM/Prm+hZdeDGlWhx8UmecluAEwtkLDo6Vv3WIWTWMpk79KlRIaHT7WrBpbxTDtls5QtEwc0Rtnx0ECUFwQawo4SaAOT7KjYUen3nhQ1BiN3J3SqQh+LQx97EJBpyalU94hzkiXV8BhUhDpYcYLKilJtIWTkqpi7L4i2ZXiv/qDrm5RufdchYxAIFNWoenXTIQseOkK7VCvc2cyhXhg3pgyhej1Wp2Hsfbs8sMklAKSnq+t8zvSNcIjTrxE9GzF3BURthZ2WGI4uU/zJRqrJSNizTFdjUodYEWs6PYvrsWyFwmWWLx5YletMzbJlSXffKMIgBXWkUHvgVVYRZw0unknO5x7DZ3MDYie6nBC22jAwBAE6//ucXXtzqx29EXQuM59Uo2hCnmw8UJYD0/pCH4QPISLWcwIQIwf58kLXfcrlbp9ZRdF5xgnraGhX5UstldrkIUkUvvTDk36czxDT4T8vPLlWOBTPmbLcy09Jj0xYrHSbimq0EYvk5kSx+rOH7Vh4CB/mh2++ltbfkbGsfboTkqDchIspRpIEzw1+/r9NM7w8sxs/b/ac4L5lcFIdnkuFcXzH/aBEb8Lh6RFxVEmpTEGXkz/rpaHk0z3WZ7TsNtzx0me44/q1lRYGK4bLQffJAwBddeQe1xN+eD1lSNJk3EU7n5GEA4h+0/5Xs3a9M4o3S50BbetvkhnPpfoT8ZD307nP/1AVnJu2SZLN6PWKY6Jw7KDtWd6ZBmxpsZTqMBbruhZrkWYtXgY5VCtNijtHjwVzvcuC/B46HvCi4BFlKIYlTdNsrj3oe+y7hYbATC9v5tAHNqkvxKiTUThiaWvPmCPXAWx+T6lDM7mLwy/tmBRfPWfWFFPmsHHMuIkOlrf66FJn+fhWlnhTi2/QvcuCmhAi+ZwyGxyspgemih8lVcZ8K5mdw5rREF/Ebrz1Yd42KYs3Wy4NkuqKpQRJukjOyPnkuiuSug6U5OAIRlygsxJo5WwxbGCVrWFCZcKwgIbCX0xYh+jAn5RdneBZPcWUh6si6ddkiSIAv/I2I4hVufXojvJX/sIfvLnBBtBJ0aokiWR8cdLcEoiWCwHg+RiiKniW8OTMMzkl8QJZiM9dSFlh0uDkmce+ZcgyFCQ8fLsNVV9kw15rQkcB4d02XnP6NJRd1GeL+pnbn6qdrBC+db6LiyNeboGgcI0PMyeAypzpitTIkRxTRaFks0wtf5sDEtXnjgA4UdlFj7YnS8TPsJj7aVfuuoG87mO+aZoiQd3zCHZA5CHSYU817efxRH9TpSrciIoX3Dk6FdfgQgwSil0/EICgz2/VqQ4gRyae+9Zr+2JMdzZm+0VjXObelHZUMeutsW5IkRKPOpXFsLIlomtLj7JPw/5xUeu3dX5In9vnGvtJ1wvPybDayaiMrk2/Pt0HQdxT7Rk6bxHVmubXPWzuHIglprpAkmfqJWM8y7HZOsPn7nfdRrfDs9qp7ku7vNKyKrebvnjlZEvbvGhXn2u/TgcKfSooNVCg5dUivlZNc7mowEjN6zPKFCIdbM9ey4jz1NFI7XMnaImhOPhjlPouL/TW0xgXyvodv2pEiGoIBz7hlQp1a5MUBbBlhfpfIXZkic97exSAvmRZlpGkA5IrWX2zuwEsWb8O5OM+9dc/cyLrYQvwKA2ZB+8pHmJbNNa47VgTzBNtH/7LSt4C9nmxpcLX8nQGxAjthgMUzsGc9skKyBYp0tBNS6dYRL4feqJw7BsOjBHCt1w6ba2rFqJm0crY01OLFiVc/f0Fss9ifowA8PV7Uj9tnRoCt9zcEzrU3omOC0iObWCsNcb8IAAoxy2wUkaYBfoM9QYA/xo0+gCsbcWfDaZkpn3okc6zrIBKhzh3XptaHmvsmcC35sODJZbB1U/k8w1dGoZBkQNtR5Bxsn5d6xo8XJF2Xk9SVTyUQM+NkqGo8ODIYKxhr4XjNZlixWTTvO68aGOTKvjm6loUydripcIJRVhYvvVkW9cUpiIeTKSnVqZb4gu++Yq1lHYcraNNSRH7YMgOjqwdaXqNtxxcQghPr+wtd6GIs67ZutG0mqUo0ZNpypNPS+Ro2TI9VxUneVnPfk3SFCtW8jy6rklWlDzJ49du7jDIEmlQrl0KkpWtPq2k6m8rt5SpQJmRpgwH8AEiVWZsQCnAt/sujCpoXTE9oqNMVTCJkEe7iWkR/8CcKTtV4n1Qf8SYQqs6A9lH+hsbywRtRGOl1V7RX39TU/uTN1a743TK9sfjGExVmvBST+EhPGmxtg8C5RCMTg8/5qpUHkIPCjhs03ysp22LBUlXimqJUaRxV4ZebmtWHj9jXPf9VLGifzYsX/MkwpeugYXqNbATpWBB+KTYZSkNO6ida107XQTQpT+J3V8E/5HxaMsFWq1EU4DhfVUm2Wzf8DKP6AwWpQp2DDqpEfB5N0k/tLJbY5OgoSXRuF4GTuYemWnx4hRcNzHviH5WBfYFHJu92rpWVcR0GMlW028XymGPLHZGcIrzzYummhx1Oi0CSwd5BsNpnufX2q5drZDHrQ93K45ehT+j/KRcFUf1quNjgeR9PTWoWziFJ4KqPF/FN/8BpC8T2JMlcCR54qsMiAMYFgwDklBAt+uH6yiiePUswrSkkcwMCD500470QXsK4ecEVW3xErqHcWZXklsEzI4s0r6uWPUSpWRex70LPfqzZrx09oeZb4hrVefA1sPnySBverIgxrNPkmAlxFF+b7XBi+7D8ME31zqKKDjk77C+IS5/6SbjByrQ6E+FR1ogtp8FxThZ4xvz7ln+sYjC3D3pJ/YCHZc5we3Yf4nm3WtBIU4igleWMlADW2ff3hQoviTouVn9QkooPq53XiK05sGliiwF6tMhEBtg2o+E4Lbu0VXOYh9v6Y8JyDASaDm5mMi3/CZfGgVnwYONocpxeWHw3UOBLyOrE82fXVk775uACG53nBee6EB5uaOZBNxJ3PtdOSRourYmi0/pmjXHo/HTIxWxYW7VGlnfR42JO0ZuKwFaSb+ybXWSRMHJ/oY8ayjU0N9L5WxaNKGtrg57tn0lgzJZJ4W1mz+UqR8p936uVrrr8uTUhY4ixbdsw1IbJg7J6KGd1D3M/mSwx2jdyCT3SPfKQrkVhBqbQtzfSzCK9eW8gRaMx5hrDwuixq99wu8yjB4GihXi9s3RouHzidIkOjKUMVlrSgJvJyvS+ZJD6NNVpT0Qeb783Cr9R/lSYKtUwH1abovtaICfpY6y+dRc+sjLdq4ZraZnYZ58arWC+ejksTjG7YnMusl6qzVw1utL8wDAouqEyBNlz/iCFC8ef409wLCRYGIsvGw7QEldEBL8K1fjKx4anoeZJxMogauGl2qDFcA8RPFhreLswVhen5tSJ0PdW+vlFJRiwWt7gChjfI6KpS+jVIaO+kEeMcz2KBxtjwLfSotSa4GEkbc4Ph+RG79+J0fV3E2gjpzgTw6T76/lXuJWCMVRX5vC2ShO4+KjucvAFSdDqsi02Ck8eiovhVGh6zfzd0l6nDW6hlu4ltZmEru5cMBQeDc77uNHxkHxb2kNjzRpxDCTZNFo2H7ckAII67p4dVOffRqJr9QqbGmazVXBqtnKntSUs3wv1yEQJAQLsAT1AFiuzOM1UAZ7hpyPemM5SuS7dvB5J35J/9ZQQq5y0RyYGPcMBAHXAuA496Vp3uJzBrA+qoeI/T7o9m+OVXKXgsaiFfRr//3kX5btSm8gzzNazPIk1g5XwlOF3CoqERHNivY60zL602bD+JKt8qAtLpCQt/oV6hwVuWpxrJtPn3yzq3JR+sRPtJ7Khq7M9dcLTmUcET0B/T5g5AxJCoRNn97wv43RxCnq+xZiuxIIH42XWJqXOXu+CkC6jHN12VGq1Sx1OULGgEy0cxWUrcyJTjkS6BNe3KH7qpgcgz8g9IAFlRfvMWeZS62a3Fq2tLkESm1aqM6x+F3qxCct9wzaw4phiv3DFD2U0FrqNAt/kiA90W0/6wVs+T0jpAQRhBlzm6jYkCfujBFyJvQmjI3gPvcSPYA08+a+rraFQqMx+RRIuPABJZlHh4FEJtkeHl60GDIJpktHydGpvfTr9upTbCAAAmFr7Gja6LiRzoZmeiXoKyZutQ80a1QWvhEN720o8ckxEU0ItxXHBf22fr+6weQZRNRQWMCjFti8EUKqcJ4j+V7vD2r66q5pDagv0qvB6DpdXscCUuj5HK7NKNN7XjFrP+k0/3Xb0LhKOY2EFxx8ObSUYVmWHfdaDj+7zhKVLb3UMm2ZzxcTLS4i5mElUbg6iTeHS1N5Mc6ZMQbVbJaQzuz6mC4+UohtMPPBdfE3FRWU30ySWrMILRRsN4kDfwMEPdYvvxwhJqyrWR92NcpjGM2SYyDQ91txZB6gou0k88tBHCJ7b0RaRykPHPIxTtmsXJeHwxxLa4xboTtxwTVO096RXpgsCK5yHQvNUlVZwWNvA7uOG9WLyYKGyAh4N6FynCZzwlusPi8gkT1wH/gLFSlXVJsZMwfCLAioNHDHPNJPpQUYg4QQ9oWDCHlrvIKscaurpkamefIrVxpd19nU2tZbQBt5OuQzo1j2/axpm41NOz92I0DYfFBCb4ADJj92pJqdxLRC9JoEcqTx04Af+Dx0fcmzomPLKez9oIaAe4mxhmAfdHVHlxuv9tvVU0fAAI+e1gJFWa06bCmECkvZAjok9Xi8mf7CvsDv2cnrGVje9JzgaSk85ZS6hPPdasEhqy0OCs11St+sraHkK2lts8IcLZDSyei1Rdmg1jvPYX+mOShqGd8f4sBuyXDy3BLVpi1ebg5WpSzR6YD4GQyPWPCWAhwLN9lho8027SejgYFH+FMwe1kW3/dMZFtmA8aGs3IWkkOYncC268NRVf/lCIKF8hXg4/zlv2H2MElP+XTsdybf4oO59yMBnnGppXnGhJUKjnkgtX+yrMVon9/Jbp4QAPoy3dz+LovKBV82+FwV9993eZ/e96BH5+o1YviY0VKo+cIdgyf0fI1zYqAb6160iNBmNeyVN323qEQADPgyT/GpwKMpG6U60iZ0HfbkvsNFY853dawwfYuf/gn5z+1SIo+KYlPSvqbuYV9/5rMps4m3A+OLF5FM0bFUshUQdl9g8RYLjkF29GxEiRp+6e4CZ/0xjqSGiFl4I9eU1uhOPMZ6kJyySYEd0fi5eMlw03U2qO+Cknn+gbJO+R0DbnbkpPqh4mKB3wIRMe/IIz6Y6ToPja1SZwSo3NS5czgIjpeI76p+D52zOJ60E4tMxZ7/tr6WFhz5+D43th+cpJ3N3u7PCs6G/BG036dEhX0H7wAAqOElwzNV7l3VSOPwBmDzi/U9f6jUWWCPVJ8C3vRE9nYVqSbohRVl0VinNSwqoExX4+oJ/tG0TCjMvbfeU4r+bcNCBRzBxB+l5bRHAsJkclrxyp0jG/MH5kE5b68ptay4SWZxkbPjD8LNUfs0fOfF/bJI+GD0mfjo/hLxj0HeX3TY8Hasaq1pVL1LvrTkN0Z/p0F7ZQbDewUqM+HbFlQ9etnzI+L7ty/6eXCVKV3eXH6SesoKmPfD0vs5njUQDvobYgs6e9+NKmCsKJUji1m6NQ0/9vAdf6ur/wpfurxkzab2nD76CfAeXvLgV6lh5RAnx6L2PX5soFCIupYspvjkxzY9qaRbVCSMLDIclhpKSS3YRAR+ysP/7okGpbe7mjTrumOBM/ijI7EFIi675qjQD6zBTHwnSlz9TnXf695aOqqPLg/sFEL0INsJt1bFxucdgDKQsEREjRhQYTzIRzwYy3yxY5GmYQNOa7bpARM+MzQ8xsqsqdRFJRJ1zYTBAUk+ezTHcnPSJnhfsZLr/Pupic6KF8V4GOHoU2A6Ua+XNNMn7TrNeeLDUJ+3iCFTaOalCjJBYUZx8pZcLTOZG1tulxVoBDwh07SHOwFW2KMoO7mkA78VtrdzdoJ3a0L9xaQozh3zsrLuL/Vodqg74Z+8xggJhVtf1N2rY0LvGtoLecn8KV10r7KOi46VDgbpPAqC+eBLPBVYFNhk2vlMsAfvKKGs/kx3UwnqXd+xCytVOhLB4J71bzSgMjrpTLEBrnKxrT3t2guh9EbHX7adrkQmX2WDe+jYijEZ7Xod8uSR6Y1tBsVBp0C+64WIxvXlhfGW3DwmW9gbakfiOfMSiEeu6VsGKdD5wImnUSvzcd3CdQ1llVXPfFNNhVzPOAF/Z6oDipNGnSDeW+X7p5IotfXRzRZVsry4V+KvLjUmbA3PeB7FwOf9E0vz00RLv3JxqeAOcWuEWQzhhNQJnP0+8HGJqq8bydl+ns9pR+gbDKxJfy5ApakwbZVowxCrWzNEYDUylTiaefTvoMnjfMHT2RmYv0OtaFZBrSBXWrIPD1KP0VFt8hIZ94rBOcdc7uyOjPMlaI4Dp4k/LS0EucAvHmtqekfB1icbS+8lg2gpoLuB5MjpF/5Hwo2rTSsZqcIO47r9sDoLTaySQEc0c45t9uQMKpyS8LAKQD3JZ/J2yL/1Dp/lR20PKwZMgycxAwhA43h4VJAKODv8wNuzebQPWfqrjc87Wx9hNdIvUPTMIotvb51rgqw+++9xZV0Q4cbDqalouP2Y1W0zpyFm10GLnMiKppBkkyS55Tnrlo465rC89IvEuZMNNRpYVuP1JUDH6nNjBwHESTCwmDSkjIhALpb1hckUS/el70u+yCflISHYjI1cg3vHEX/C0JJxLoHDFJKCgZv8wXmVuJUPS8rrxZ4iMN0FftEXW4oYJWOv3eBXILRxdiCBMcDHXloQgxfUGJBOUth5LswcJTWu/AkZVS8026vmSEGpCm6uR8X47uNhbAIEM+bXZh4AoiTJmJbFRnY4b+O3C3lZ+kGYqbZYR92KlE5oHgG04mo3+XrZb4x+F6SJFdTANMzTTubjE3G5kG8WIFq+dSBTEtbDsELHbul+0gbcTpgKyQdIMUtbKh/ZKKr1m3fajaA46RgrQuW+0AAhjmkbwvQfgE1z4atpA7BVWHOEYLSyZzCTsvQ98HLfYS6jI+b4QUBaz+ojtAbta/q9laPZxTHAYzKWLl2j15PE47doEJDP2Fcrl0Mizbr6Qeb54W9WtY06h6DYE7QSOYkaNND+id/f8ozh6/s2RyiAJ08oOvqf03+d6rACs2HmDioYzRORHnq7zy9pNZxc8zkPUv3MXb3stG3C10GffwItY20JuaZIgQdVdMrsFKUW6p1o580TZsiQFXnJRQWnnQCQekVNjYy3cV/spTgqUwzVl3l8JxPd1ViswyNhkgQrXqJogL+Pzil+9pptu8SbvlXOuijcLG+WEd+HbFPa59CZcBAHx/nCU3B9GljcwyG4zaVQb3A8Man2McZIi2/D8kY/tjYGHZZXnhYReZH2ChSeT3Rkyqdm2uEinnDSrsSif7sWuid7rvZ1VrX2ARXtpHwFYD1NacQZlD1cwzo2k03FIxQ5/FASyW4AZ+gcP46Qr1zHOiFB8MfWTk5jO5GfOSqYGSxo4FzNdG1qTxSBOPQ2k/o2kIcYN4SRuv6jSAn7RVSffMjYB0UbwXPv/O1TXQnyrZpaRY9u6KxW+fRidtRLYjej0LUFgb1P47DZgwn9IOWtWUHNNs2lSkn55umSGJEZrjJFec1qHDBPDJQLSg5L4YVAWBpt5ulSwZzk7xbM8UYdakM41PzbAXFvn80J/NH+5+KGaZZzcnZVoAWFX+xCQPHsoHM7nsil7voLoKiEHJJNpwyOWjeqP8NX0WhugKWhFGGI344oi6vEUr+KhS2ZHHBIMc2It0EfrMXn3Mb3uSqIHtejkDF0K4Q0roUcwSTqHN0nlIfKe2QDsjDuLX6ggNJx46E4Hz7GXQNx0fiGRnmRIeSJSlvByXej0FuKE+xt0dFdFcCOCX69deBtTbATlZnfYp2UUm70GV/F4wu6Iq1eBAC5RX6o0uC6GR0KmRHUecGAor1jzhpnaY6EHMjBxwNgWeMIAoCDXHZM8iEXECRx2dycc/pFH4OuIjlZdOAimCXOTjp+5absHLe1vJBt45RzsRxacTRSeX3p6JpH5InBbucrdeZidDBg0wq6Q2Ho9Z+ccsvWCksH7trpKRmfGRbxl1xhrVNgNySZp6gL4GkPIQPTVpXGeBJ/DwTWK9ZOFme1gE3gzzMmnapyW3EHXycVm8TaLi0f7pJMo2KLB5h/XZ6a6Fp6RUZxQ75JG42lNw53DEhPFXiOzwAt27ll2xnl6JCsjR0WOpcK1CCtaGafv1Fx5hY1isqRjM0SePF8SP1sU96mzF8ASPAcguZk/I6d5XGZV62wW+ufPcySPhU6PWQF6Rdj8AIafggzx3fjV6yjCWcfwk3SK9FGUC7gAyI0OSxogOgy0hxOOja0jw6p+RpaX3GAfTtwkMgZqJPMILCERjoaebsZBz4ipfSNOJGnHgput5bQfY8JVuqWrldj6aJK5hl1Fs76jvRq1bmDnWwJbVHEeH7eZ6ih+oiJNba/8UIrFqZdQ33F31eOkEhzajhTvZ077tWqbpOFoqGu3OfCMu97LZiOaJosIT+Ei2CinJ5+v5Q7SinRChOrtnZ+CoT3SlCfHPYjlCYyVF1ctLqOJt8Ht3pXmS9EsKY6FTHPcoi3reM7H6lpCcXXq8yfNwLX2H3Ku0pLQsKmF28ztlQzslyLi7sZfCT0RZm1NYt9eW/72gI7w3UmEMz9rSxI8W7rYqgoY8lP/IG9JayVifkIbuzO8m3eZwJLWa0JzlLWM9Gv87PQuI0CMINCDvKowNMrB9SDOuXhh3KX1eVngCoBtcidbEkjtdqerC/WDCvZ4niK1MiKkNo+Bd9mfbj8QMI0Rhxf7uXxRHFOZwOzYNT7LBSLyFqVF+wSlO7Cg4mFEq0MFobsBlywCvEO9FIiDDuV18/ll3fMlSncCl1/TWfARQo+jOUpRSkou70G5mxd7H6pAknrWuVQ5S5Vi2Hps705mpSLpFybkDDjTVbFNmF7hHxPkIlo+TD8QeLS+bFzxYVWcqUj3mopiR8cTv9+0YW7cv5494FogsEAarV+PnxXqfUy3zVLXFrANl60g0YCD10FMZV9kfcpKUGAOd0IN4sDtelJsIvWZSVCcewpW9ol6jcMehWcf3WQmryHam2f6ynYOGvkTPyVl/IzWU0X6anyhd2u9atCd8xkzVzPgaw0bjgy7mQeFI66LEwZdGZl/0H/yTKXHBy9U4uQz16SFvuhdo5gh7XBGV617Dvu90b6XYiAPMcWDW8ie59+cHH25RUOw2luIfHmhaUMsnwcnTwHSddxAW/hIZdr7O3hm3wEiFabcXwHZJNyS1bNKrFHMfuqMu2PWvSuBPKxTesg5raOwULD6qwaY4VyOHy3S2mihwId8Fa/OrXJ+ZnAZ8pn3izDOBzPXb8hf+yD8RO+g0eMYUzsoUhy0y0ehTYGfdNZXU4vUURsXyMqBezID6yXmYFB9NTzR7G7qDw+7J037GsslVTzVGA4+v17iKsgSaT8cS3R4lgeeDfhaJdq+gBN7YlRjMgZOzzUuubaQIo2S++zzCbV6sUT2A9Nb04yZ9AiULof8Vm2RWBKfJdKHq/V61c1URQiBDOXyg8k4aVWJVt7riRyhTqYt2fLDzA2Aes+j/tsqXxPMXNscoIU/VovedvDvF2IeN3n3O8Oq3bqJzWooyqfUMEojH27sRy0i3YjnCAAogOlKde7QGGtk3T9ArlnUOcCrWDlBEPv7NtZQLDSgzhLlBzS6cB9pf7Ln2s2PCVD0hHa0NDXlLL8/Jrd3RvXQWW+kDmq4AvjPrGO7UcUHlE4kVcyqkFejw4yr1bViuUDtzbfZvKo5Hsd7DPHu5AWhuO7VYvkqSy/Vg61yScX+AE/5ROXVVk2E0prAL+/crpMWPuSay42oG+5GFti4sJb+JF2yTRdUzrAojHVQkWCcOK0cAnKAH6c1koGGkAcq2HwqOReIRsy3HgB/FbGjFcsd4By+DZj9gb2yxA33tdCBDixu3HOBjcZ1NponiONrgHwLX90h2bf5z/U+iFo+LUpjgDiQwnHisYjgmm7sXY/eYI5jzY20CuWi4qkI63XaO4SFsb7ESeE3pwdE94UkTJ27mmaBebzE5LasvEwpyaqE3WDF8EgiEy0k2BnAgz2OhGyJDE/n6pQPtbse5/RceiUQSOZwQml7IJoh4XYtn5sFmlbEhhtBOzrt12V7HoedHxfTGzxH6Q2myyNBBXyxtRzppSeiHMNhtg42DC7aRYWl4AN3hKffCtwvW9W8ab8qejwycz2I3KRJKFlugEgWnllYPY9YVgrZZQdNeQnJ8tEPleigCAjZB9U+NG4vcJvcXN+K4iQvRhd81eQzshVhNbbaPbeJg7GDzqviKxAd+NkLPJwqQTtqH3IVIDkRWreIUjHyRAFQxOZA/A21wAQ8yN0ktdjxegrzpb7QrH8rUmkiAWHPGlQCRgLtZ9R5Rob1Dn32TYppQDShxE1uLC4ruFaeIHYErhJVITTadgQbv85TP4DnzKnPawyryhbDasfT2SdvITCnFhPPuzgMHqDhrvEH5vRgkHhRiQdeqWeP9sg+caNBCG+A/rM0AEKs1SRMZL/6BPvBrONzo3E8j4XG7/P0xxm3tOPEqfvHputttzn012mpA7ht66p2W54bpT1JocIqG/UMI/dxOYTkndqIrQ7Jh6/+MVs+5Jwry0V5IspUhlPT5B+E4Gu9uVr6IpqvA1NBqKbG6orcEehWm6/mE16V0B1wPnWCBpDWPFCGp+G8Qe8NGU+j80IJK+DdFOMroRafQ924NqWbSxh0GacjuQjhByWLZtRiFA26pQyl7huRtTr3MApF6tbVhN1EWhz6B1OwXHt+1w4vjgFRzlzXckrWZ0bAeHvSfu2HjGUgmbYct7dgVuGtqI/FB4CDJ5pEw+PJff3sUvSD7OZhFx+t5LtdXv2pbK1lXOklti2XuzurTS2xTsSTwFb2Ruft2/Ger5Rt1bUmKLT+qa8LBBI6e/GXexbOYhdXsV+wwKeDQJ+u/BDWEXJfVYwSXv7qydspEdrv76np6x5ML92w7dL7tRR0vAMuNgz3tXnvsQRjknTm7ff6ZtBgkHXOzDMCiNxokTh2JO50/1hZo9g3syGjW1qXyFcLRWA/XiuBLk4HfQcG7tSA28gRHSBN8fbqXBSt5ZZ1DF5e/t64K7vQ7SoJ0pXEi3yFLWeWb7dms2x4yi6Ni+H2PoTmsbkzj61DhdZAydJKmxt16I8/WE14CcgiXo+56ORAXg9WHf8fhluPBnfB0OiwvWd6T9TnH+8kognfaPMmWYJaVowOV4e6mC8VKHM+kbRSHmPaqJgB0aLv7rnQnrQTgAxQtITTd+3uX0DcAscEOAaGS1nCncar2B5CAaws5pY0dp0bZRYfevArCIrquLAB+coF5GJW78Uo6bay5K5Xj8HfCy8Un7Ue9j2dztDsdKIEy9d3wRTSOTLxdsFFOvG6XldbMWDidTvLRjfGK4m++rfdyPFu8fE6T6Of9tZR0IbeOgc5zzlbRYssrFsTa51+VVylOGxgv/iOQA1ASBNci25R86O/ixdRV2Pw/lIrSvm12m4LQ7hd5xeZ3NLDAw+vmHPDzWeawuTm6ivf4Qz42WT1tESiCC8NvEcbCthhg/lUzuZCTz8JGU8UUgPkFfsD0uYLjzw66nAs2iHgSGcqpS74cUEKMh/ik/dU+vs6/EdWGY3H/OXI5OKt+C1DheGvcz83QETZYeBdwhVzhf+ULVePsUa02fQxfPyajixT0H6qcfRuTtZFM8MfDDo8dNsVSe2gkYqayttuC6j6kvY2441ha5eter4LFShOS40jpUtrIwyhIMSk0KwdFSrkZh0Ae2jqdSvmAvAV6zejkHW5KJ9WtFTE6RJ9PCTo4KTSFfm0VWbrgSft3lQ38OthqB6AL3DM6maqTDKUBcRYm1fvC/agylAiFEIxYdWYd6E2Hd99VyrXu8oxAQxDIr3DStmYkmZ76Z9x8kPtH3Fpoxt0KS6MCrN1p4mFoUehw3EhbdzhzzyLVeobaONy4+dBPqxhaY28rdHS/WrUHhivZyk+bVg9sm/sEl6JsFt1xPbgcZ64ot5Fd2sARnT7qpMOgoAXMp1b7FUTf4M9HY7iVdWzqje3+9OL82TgOLR7sGDcOeobsgC8/74vS7IlfcY2Ziy0utZ7h//KoK3v53hBQuTYH/7qAr8HU+efDiijCschxP0KPNuGfL17bXCeNZMAu1BCBYZ1zsPcfXNjWPD3jly5JYGsUFQKsvUndKDBgO4KAQE17tONlnmvfW89yFCZ8TxvI1y87Uz379b3+3A4sDUj2NmpOQJRp0LFujZvqzBIOU+Ms1rbhxRoNwILnGcD7kWEKeK+GZPgIyzAkH1z/Ct3Ka84QBI9DPLW9JHF0FnzTUTgZA00WoNdPOUpy6vil4I9kUMgFheVFmrcfTohc1FUkcZath4k4uxyLxgL8N+/wDqx+DIvcyWuiKKBrLOA820er9YB37/kQIui+rA86x3HztuwJp4jGmiHgreAyWI9/RML7OMG7EWOoPmJuSYVe/e9wUo+Qc+UgVfd8owkmFM+eNV9JRcKDxYesJHp6PiVUsHje9FWTlUe7EKsyGcwDFTPS2hWcVojyNpLKIAFWYBxtVVGsD0QVjAqtiz5NjKoX76PnHBMQf9TykQRR4UY6O8fARhmSXgEwgdYHCoNYs/Vxh5O2Izdt9XE+JU1Q6f3bAaO3Cwg3LAT1JS75N7vIz5U44+1RoscMDr3w8q171++PXbTEy0Vqqwd3MJ4nVOqqKpHjhvA62W+G5aKGTPEACK77CYARzXw+MRwt4mntoA8HP1+XCgpMbOrZld1Onc6Ss1DyxkTzk4GPQvbpYB9n6f6lGWl+M0lje6IhDaSC1ULC/rpiBtvHxFWflhp+ND2oNpN6FKfcorKlND/vTFY2ymObB5XZLc9ktRqeX0h/927dYvmEkJCTXCJVBuwL74eUpej2HY/AgTPNaIZKnKmLHc0oUSdtMw/xCg6Mh07eOXTQ7PADXFZ6uqfPtEhAPmRRUb+XXbgOe7JbIJTZh4wJt3vMX3d6lin3Y5bZJ7l+BUM65CfrjfQ/lCpgVHkzoBWg6Pyozv0vUCh5NNyl7ZLTN/H0mUaMOlPNmc6R8wOM8JFt+uEjyIM7y+6KozY6pEn0IcgfPRjCifvChRWGDAvHvT/YjIp6A4an99BP/jMdnD3i861a/UJjSpg4TvgyEh4QEDbc9LrgvlyOv4ZwEDZYOxCA5kCPK97hMYT7aOTmznX6dnXtXnqSZ7GqidKvI6GqFf4GDTbTqyaYxyab6j0iREH6ElQ219WWLPVKO2hep+Sjzd2aCXCvWySwgL55GTpEXlu968vzEm+EeWuLVuh5ylqdSIGvsVbhyZBa5m7PeBQIzvQE9BfcwJqbUl3suJ0XbdSb63TrHXFfjV0ur9NtKVA7EktWfInLX46i72CARpFZhCgqUZWmajMvlRrBP5DblSAUAYfb8IMJ7f82wa1rLN6qhqLTX1ISTiY0sr+Waj2qKlp/B5UY95KpcSNVlKkJfDwgSv/gi9mJ0vZjTebJ1ClggRdxBqSltZWY7sUTOSHdZJ0hN1P3URvILZ/iorSaY1vpHLWaEvtGVbv3jjROcXNc6KG6F9LKwqsbuT9ZzUfoAjh0RnuTw8mTEREFv05NgJJg/GXN3ok+Vp0WJP3gjttlGDYEx029hPNHLjIXTfz51xDEI9JKdTTT//Euu2QpT51KP+g9+Ld+foRNRo5qwQplWfq/dWB9HAB0Dud1uG8ULa15ZxDgCT3kX7lfZh5Ehql7dqp2AQVVXpTh+Gjw3ioWUbvve9+fn9nAT1hnFGcEuaOI2JFplyvsKZte+C+LHFMyuqnZ9CiJ+FWrjre9y0R45p9AmjevsIYBF8GceFdu/WnEEUx6uKyeKFvjGOGlEHEQ+qamNW7XPji6+AUcinw8+EoED3CU4j7pdzBuV3g0XxwxaJV/ME9gY2Ke0Mf38jQcolHipwRt2essgVQWHOunRkQofLRppd36x/uf3MNabTUdm5aUc9ME4yZfeUVGOY+3A9ijkdmOZIR33UhctmV6Fq5Sv0xhg7GdQGk00PuJsxI/oUwpirdRDHQ5C3lDIeX9AR5aCHOTAmmmnAiLVwFv3N6oBsRbydLO6Cup4PvI/4jj0Dp4SKb6FZtrcFzk8ySDDRY9kyLQrRqZFNevc4EYQkF4Ol4viPPav8pDVl58bkuhX4u4GJGs2L6iIqVqf2KhpwXX+sVdpyKBCKB6GfJavCYHbIV0VfYh+tWWI3D9t7hYmewQpOSZZVfxQ8mC2I79zPqfNm+NDbG7XYo9ArLXASwBq3BpQw5dVi8/rQhSnBsyj5SSUZBas2jXwb3R/iDN2ZqdZy1aYcr0ZORxRCvNRHjUEYnkOJs9Bw4iWats5LnCyOc51omiFvNcucRasSI1q5PpcOMGkSXMpmYwm64wVp+uavUJkR17aYjznx2CtAihLMpQBXHex9OMcd+aaNd+XbmAZjRpEskKWqdlSv6x6p0OeCus8HRev2nrIyRcU+aqHCmVpwjI9lXNEzWZU9znnxxxfT1ujHU4Fz2Odvh1aTcXyFmX37mUvoTi2kXqmeL+8xs/r3g06GhNCs067ZOk6gqM8SipevJjYwl9nqtUPCFCgAf1oNO6PFrMOxJS0GUJoHiv9sSZvcmKVlBwo7a32EpmZ5wWzrc/X4b034VyubfJNtAnv1iHIVy6aQ2y7BiCeDvW/0Fr/C68Yry0ttD2CL312AlyoKACrypkWrLRAyERT0oT6UfFUXnUizNZl4q5aQqBBhdrBcxliJ0gbg6cHs75ZaUu08Mz9C9m0Sujo8v1FUXQiQLHCy7JfEvt+t9bIN26+h/ejOCwj7YFyaxZGuFz31SDtiVKY9x3wYRkLyo+pr79d9ME7I8NxM5VfCyeC7SRrbPI4W8rROaOSWy1PAC/2HznxihIMBjbJLh40AiHbFiXSHMFlWhQ+N7awDBtv02Ya9TUP804f97Mr9L9hSLOcBsPA4Kt9GbQ4E/G3AUdq7SWO3PosU3ztDJidYrftu8Or4dm5Ui2rPcK4ILJvBhAs+I5eQSf/ZjvCIuM8BHdsE9IwwqlNVVirYmdFGuO/Tfv8JE4eBwPbjfZT+LWrNAoP74VoqL8kuE+qj7bBQd2vNhzsWXgKMOv6OcYPq2Zhxrw2E298nHWYMmAN4cH77SlaYCKXydi2JQ1XFsiZl+gMT8MedBwIrTywe3oZ7gIS7UE/eQgZUxcY3NQG1gxNyDhN8I95iIk0piOEaT4EwPeYOwYajXCoXOrk8fPJ3uF+2E6KaJfWS+GGKoUlaqrX0VPwps/H33z+ihHUGMYa7gqIkR5ujCu5sc4Wl8KxVmSXFKMmnInoRjGDuaaXMj1Hmmmo9d6KhlPeLe+8/PkTF/vDxWAV/ZwSZ9zcReGBxE48eWAWzlYyqkJWdLyhb6ypd1Ozo6CLxEYLlXYEX34ZD01oIQdJYVJro4IcrBj5x3bvThMoNA6/zvghzsYOUnx2lVZa9NfXNdrqgIYj15wNCvJWpfugpPc+huXwtG3u2UwQQKl0vNm5Iyo8vDJKlqVGRRZiPVMN35VJ6JTmy82cHSTJnJkRwotuq4zIWJnhhTGDc3sUNpqY+fJTnkbdZNeR9YQp9NSfFXaoz9Yxcsssh5brdwsc0PfUKSQ0igd5t+Zj9CQMSFVuu2ngpvRLphV1p4BenTuswdYBZ65E0+xqtY7yoCwXvVtf6JcfEMs260IwTTQOk/LALjz7Q1SOJsiLHkrk/bxTBMwhhycOVTgOX9+ZT3jUAdbx+3wZWhrqmC/ydNsM0MyZ6+LnxvcPaCxtyPKw16Cw7K/2eURYZOKzskZmTCQ2lWD3qG5+j2C/DOVFLFSqUPq3sIBkFDxatk7xyOkq/WeJU10RanJUvDNMmV+m0sAYXU00J0RxgO2tX1qi6FJDIFz5tTHxpCKZ+xYE6WimCI6muVnjZpGM66/klZEta5tBA1Gw1Rc3FotmlZ0iudTsQ0uyXstQLG93jKWshmHlqSjjR5H+9V31y3NTFS9NnTxn/QVFwbz88JT4MnolAHCzNHcwSHymGIUDEZUPkjQAWsSiCPH/3JXychcGo1FDPcCl8d5zyUDQpyTOuHitXvnNU/2eFDHirnYQcY9yty4WikIczI+LWt9EHnJUHoTpIPiYQJSH9emjYzdWh1otDdeVaqfaNx2ynTuNjwnx5/Gs5XBRtxwKLWADrZhM47367+iJrq/9Ri3oOVMuXJvGNrRXNtEu4kF49QXbhJ9zgOMveWlk4WyGWvf3QJnKbrzLU3LvoT4wdxZpedFtKfmZ9671N3NC0PFIAGAhh3nYKHipFVDVxwKfE7iOzDW4P0MARLSEjHLdAQrikJ1KspB5Z2nDLHoijOAAdO23IkjVaHhi/vgwnK7iM1vKXhpbxBYfgpdlMl4s3gjbZkLEYxHsfgjDBbTNCUq6F9tv589AhDkbztPCSoM3fA8A4eByODH+5aYL49ZVlRLLOwXUcDkoftrp2sZVxXOg9OBlksvxSKlJe8mhhXK8/W0NAiZQM8pehrY1DHccw4i4v8nh8BPB9nq1WbXjUAtWAG5MYTphZiWyIGohY43sGe1WYsXVXhe6Dj+HaC15tTS3zCuRcDi/7zr3FMEJK3ywIQ2bD/1b5dLdpEnkH5mwWzdePFjOcNiUeHowUX1p69LJkNgQbx+FMPax8wEuF1foF+3arYXKhxGA6HhaSVzP/T3vX1pwoE21/zan6zsOkuIOPAoqI4gVR5A25iVyaq6i//tBekihMkpmJyUwdM1UzE0FDeu3evVbv3Xuz3ciCIZKtOzgUY3oDY4it9Z5c5hMVFsVlFXG7YjobEdH9FuL6raULiLFmKlsDywdTkyU5d87ys7zE0Za1oejZEpH5qYuMWNYT9yLBaysvS5012ErOeti1U8QN5wM1FIodoMWlUDIayDWFHSzVmdpHOE+S/amLLtlDuDG3PNjK26kkxoKP5S3dmw93w37mVDLUEvPKgna7agUGiRH2441rdBkkkdJy0aXt7YCJoFd0MgxImKrtd/ZY2A7KHjC8NkG0lpqkhHExHo+wHaeLBYyvys5wMlC0UuiPZ5iyVtEVvSkr6Uawg3k3spWW0g4TZsRK3GCdxxFOiEkfrUgq2DoJOU7gUcEtENIu3RLg+fmdLZVSi5oAZTVPUEpK+OyADtf9jsIG+0Nqo3HF4iJUUYO+wJu5gGjcsJJqJDeLuXlvWSAFm1dr8RLf7KWKmK1YvQgjlWZ5Ae4+dJfV5D2tJ7i4Px5QI4zNEpPHuurxbZIFA3mRBntyiM5ybhdMy2G4r1yB67hYR1+HQ4+bzEk3ckrFJya+qXZBuO2MqPZyH8JtK3w8N+NjqnHRn8i8BVbynhCMXHN2MQdIOIHm1SpR+e5JvhH8qdCFm5pbX6P3ZSrhDNnlKocr6X11EptQlCTrfpyuvTDsqqjGJv4GDaol0+HdDUfl5UghUV6idMvDimgU49ZKtN0lvpc8kJjODKhzSgHJPhv0JwN9i6dzqEQoKE+JqUj3+cUGcclqZPtBGLZtgANdzJyBNO46JV5InG2V+9yfIK5U+DtV32jzeKUridqBzmQ1LJNkFblk34TrXCeARXXYrmqPeqOwz1AlveD8ebzBD4fusFe2emHWVobd9YTJ1eGEYtWsx6k+u1eolBnGEyuZuS1lAaZt1ZubZSX3OeGwQ4bBzl2tHJ2Wvd1yZ9rwIFTZ7/E7wu8x2iTuL9vCXCg6HmeyQndpa1KgLlTfXWRipwM21FaSitmW9fo5jU98tKK5MT80uLjLFhRtCBxnwOT3dIrmHXWf6FTWw+lZUgmrJa9pUWGlVHuRBJNFhCVdZ08ASXa1BMjYmmZH0YE+HOQRrUVpvCJNZ6QNCF3hCMG2Z2UnXikeMAd9qGjFtb/r78QJ3LkfdUpMETohxssG6Mpb19ILgYXHrhbtshUdhgE6jFZthu5LinhY7wDQtcNBWR2cA70fbibznBAyWfaH65Vpwfpylf6IUHeVeQNelyzUg2jI0OPDFcXlU5ZFiyHNDvyF23b3/UFemVPAbg2xnRRhO5pv1nNc19e5D3DZjwWLl+yt5s/X4VLIPbDc00Ho48aw1zbHvXwQ6cJmAnSIdel3xCBpGbktjYlZ9cl4IksyL/owLxysStE3+vA5tBbXymclI5pAyOiOnPiCsIx4TkyNTVtBmVxTMSro+oNFRdOmHE/ipOnKAj+D+7ITFbF7azXrz8NwlxrYPNgbaZKkngrG5BgZuDA20+lJZVYZKRPyO4sKFHeokGZRIPIUlS26ZUwpbEFW623bFCdqsu54mhz7YOF1eVIfRHDRGG8nQ7qdWUCixxpqCNkxrECGpIElCbUlEjHtkqv+OjeAxheHIDd4R0vbUHWQ9GAQZ1C+hulkXKxRsGmr3YSg4ald3AB51p8olTKH2g1lM44uwXS79ARHtVY0DI9uZYt0FjQ3RBIutffHqjVQ31gnRwUGuWMJnbEnSAJSaAOALXQH7EV71nMsA+7Gw/GOEi+N4NvyrLSg7FoJodNxNdNZoTAWVCizAzwoYtldk4bZKl0gwkoQXfSAGkkECYg/ntmtA0i9nT4MlWTvB5tS9rEtE/EiMZ0U5Cq30xjmduXs4rCh+HmvvdItjnL4tb+NeptAMydeSE7nLY4EJkdud/pqJuoys9dNJuzZWkhaDh4X22yurmKPsaVgXI5Lu+StKcY5As5OlA1JkXF4fJ50PG+NtlNtx5bLVNM6xCY8jKkJlF/eoSLJnNJxVDXdaummZCJyvSbLoz5OvGJwzDtxJDHf6CvZYx3+0KbZ0EK7i67EzDruuGOICSW5YMiHuz3Z2qQjThoQYKCgg0G19BYYnWOppru81++IB01FD7TRgsnb85CiNyxH9Wcm3tpzAdbnnAjfLlJDptG1EEjVhFiQlrEYwH3odrAIUnnATZdT19q0ABpVP18HbiLt2nRSPSu6tKiJlVPr3shOJCLTDqQZ6lsFHm03h/5ec7V45VZr4sYebZg1BlOKkkWHESVjpuaKvxbN3o7sy+M1zHQUY93aHdZ8MO9tCEK1C9Z00TXH9dRE2U3Nte6BExUUI9oYUYdJSIkrVEwWEfAXk8K1lbTTkg8ODvnagcEFuWUSi5aLj2xpQC630Z5EZT3s+ZQ46ldyUZeVPFqaglqK48Xc5At17mF+LprToddNogRIcxVWn2f1ilbmiTweJOK2PeIhG1xtOaxNrRA4qCysS9buBN2ZrxSTkON+pQbe85XPL1G3u9SifGKI1ssXelVzkUKeaKqF4cj5b7pWgBGnnioWSuMUQ2AoQl66tFyVdafuVe6Ser/W3XWHjHcKI75dE/NuBazJhsKWRKupQP7dCofS9eqvwMlLA5ZKvxnR/FTf8b1he13/8fzSx8u3NuF0jeRHmpCcn7mhv81vgEa0bkHD6qA1YIbdDTOmhhlX/YIjpV5+9NMR/EmDi8/D7zMgo267/JAI80TWS8g2oYaSd0LtMoW/uZUL8hstVO7YMOXU5+Wv75hSb5hyvj49Ww6OfJLtYvSN7RINHaropjWCuFvxY7ze5MCFFgVbedTrF//XDezdqfXOkewjp74H8Zf2PXg0NvhXGhsQ+A0pwsk6KUIJrMFVU/fil3i9sQH4wauPovhvwEje+C3sUun6vZLYd+O2eL1o+6sWLNW/4Af3wPSXMMXrOvCLMa1z3zew+2uUH97QCY3GWl85cPXy8A+hd43RZVfhbO11wfClKo+o64WHyrvCi7luN0h/u8S7kJRXkKkZ7A733APsuNQ0cebv6A324Mj/CkcmW8jFuN9kyXjrctvX8GSizpPVcfdBpho3pRD0FsNGiozSTRjS92owTNRJ8vWGEg/bFHmr4ji4FWd2zrtKc1EewRECIPC9hk6Z/7Xj+OHKHq7stnkacaMpmuV+07J9PzdW757GgSgrwtPa/ZCHP4GSxJ6w33dpd5M6RD06OE6BVZgPOH8ZTgL9fjjrIcoagM+9DV+3Mfz4oPx6j8JnQf9i+B/sUdi6lxdDm4Li9+tH3tDrsLH3YS1s9Je0Jm9+2lPHxQ+E4tCmUJxfrOwfcQrg+bYGlQfDb1J1SwArn7zXKfJyMy8rP/+tfr1r5B2bjn+AUqA1LkJ+Ah1ifpkOvc1q3nZDf94E8m4u4OKtfyXGbJ6GBhp16q7+w8jKt8MG5q//87/HYUKOc8A5G0D7ZOInKz5/lgdtAhxt4rgJdXP51ae8mklYNZNeXTg9ILwSgTQ8LpOXa+UrqUAgJxePVJOpcgU/skurz9t3Hl0HbJAand+JXH7e8UqeGlHmVPdf3nk0EAgiSK3rT31+o+VlcWCcx8CLTn1Tj08ZACO/+aBbHyEx2fEyDMh/LLB+3s97Z67eEo/fjJ3XKMnnT4/L1ds9Dby2gLa+coecqIcW3kqtqOtdtxqh7CF5H5K31s/6L5S89XAQNOsj2XiIozqGLQRt0LofFkf0vZzWJRfxFY6zil9eHr8Su9nXzvM/6/599/jrBTuUqgdkG7kacS/kGtXalSZQpj/E0RzyDXvrmXCM46BwvehrAb32o/RfCzGD3UBM1cOBGPqVCNf3LeQuf2GHx3zMo7cMIMGs8CziGEB+8Zit1CUz4nm2NqRcowzxhVg2pFxXY22CE5yW7RhFALkhBwsSPhCsI9jkb78UwTrhGVaQeXEABzOy80rv+Y27NlDUpo5xdMCPSfoqb/MjSyrToODuBvHFuzeToQdkdcia/GpjbvndIKtnEEzas+/mPH8nfDWn2sBwvtanovXkgTiPiWwfmY2uFF5YpyDyDsYpneCfA/VlQ+J0SuT+QgWtb4s1ChWGuRfIdR776tBQtwP3iPlvna0/AfBngH+Ts6WI7/a19ejAs8SEvPVIZY8EFnGqocGQ56vzbvWXb6eRffwhL4ToH56/9JeI0CZS1KhCL0Tp01EnP5Axb1uufcmlrujtGrggMoLOy6s32b0v9wzAc1xvY+f5/jy4RpGD/6mFJ97L5r6JXnzoaN/zVvHpyuI8zHj1SoVjutfgIz+RCHF5YXl8AUHIywv87vxbnb7bv/5ubKdeBQM0z7dTCjJQpKb9FgznG3Mjde23Aink2d9DTN60ucoFV6vo1r56jibzOb91DLxjbPslfxq7cVC3u82nZz2/78YMnx/kDyyz7o+uAy4vaYa3RIIL4LbxI9TSOJe+Ppz+bqjleav+KtrSuguDgnHF65MBP5iG4zeNARfqbru9dab8/8QL/4HLxD7oMi/nkz/ZZbbT1Ni/uiGGjjB706NeF1H5gZ9t7MVyTp/5uW4Uq5nW9THH6VR9eMp/21Myd/OU1wZLot/uKJtSmB6O8m1HiX+UW366o/wzaVLf/Kth/QqTy1QZwFk7Bpl3zhhYgTwH4c+nczXszvGrYbqdc0Bepw1bRrZ+Np1rmNDTvTF8tnDnVv5n/eSBjH4Kvcx88mLDQnEvMtdNyJsI/HNlm59oaD8rbPJmhsXl4rWqRon2ZzkYAn26bF89iwyGeGo4WkszT5fjnFcJMCj1dK/TmmSdkp1Pa3aSwovDo8BAKsfQtJO59bKiWm0Of+vh6avF4v2knE+wwE8xGLLO3Z93vV9ZC9WwIv3GQezq2xRAgvNCx+CUHgLLhnf8Hw== \ No newline at end of file diff --git a/images/pwek-aio.drawio.png b/images/pwek-aio.drawio.png new file mode 100644 index 0000000..ace06df Binary files /dev/null and b/images/pwek-aio.drawio.png differ diff --git a/images/resource.drawio b/images/resource.drawio new file mode 100644 index 0000000..7b86e4f --- /dev/null +++ b/images/resource.drawio @@ -0,0 +1 @@ +5Vffb9owEP5r8ohk4oTCIzA6Kq3aDx62t8lNjsST48sch0D/+tmxIURZ2SZNbVElpNjfnX13330XIKDLYv9esTK/xxREEJJ0H9B3QRiOozAM7IekB4fMJtQBmeKpd+qADX8EDxKP1jyFqueoEYXmZR9MUEpIdA9jSmHTd9ui6EctWQYDYJMwMUS/8lTnDp3GpMPXwLP8GHlMvKVgR2cPVDlLsTmD6CqgS4Wo3arYL0FY8o68uHO3T1hPiSmQ+m8OlKx51PO79WzxTZHv6/jz9FM28rfsmKh9wZuCKW2gVWqICcnHEqR5LEVdaVC+En040qOwlinYCCSgiybnGjYlS6y1MYIwWK4LYXZjs/SxQGnYP1nE+ESN0RRgAVodjMtJULE74uV0YrfpmhNPPZafNSbyGPN6yE5Xd5SZhWftHxi8GTA4IAkMlRu/RaVzzFAyserQRUej5anz+YBYevAHaH3w88FqjX1qQaZzq3azlSjBIbfcltJ2xuVkE7lMvMkba5XAhYJ9fZqpDPSfpDVspALBNN/18/jvXQkHXfkCvrKQ3KWmaL7liUkE5ctrmpKepmk01PT45jk1PX1rmqbXoGl6SdPzIJwIk/qiKpnstWrys7ZfMYstSj2qWrKNM5mV9i3f0kha05YVXByc0ZxiRdkaKY1sW0DsQJuJGVha76WNyGQ1qkDxbRfTrDL7FAKPw+aPy4fK3eKyNoS4xJ3/i88kjV/bTM7e2kxG1zCT0YWZ/L3SX+V83qPkGhWX2bUMaBw934CabfczvbWd/dmhq18= \ No newline at end of file diff --git a/images/resource.png b/images/resource.png new file mode 100644 index 0000000..d3ce342 Binary files /dev/null and b/images/resource.png differ diff --git a/images/seo-node.drawio b/images/seo-node.drawio new file mode 100644 index 0000000..5ce4aa2 --- /dev/null +++ b/images/seo-node.drawio @@ -0,0 +1 @@ +7Vxbd6o4FP41PraLO/jordOzeua0a+yZeY4SlREIE0O1/fWTQFAwUbyAcmZo12pxJ4Swv2/vZO8kdvRBsPkNg2jxO3Kh39EUd9PRhx1N6xoW/csEn6nAcpRUMMeem4rUnWDsfUEuzKrFngtXhYoEIZ94UVE4RWEIp6QgAxijdbHaDPnFp0ZgDgXBeAp8UfqX55IFl6qKsit4ht58wR/tmLwgAFllLlgtgIvWOZE+6ugDjBBJr4LNAPpMd5le0vueDpRuO4ZhSE654X3y/BWR5ct69GG//xgun+ejvx94Kx/Aj/kL886Sz0wDGMWhC1kjSkfvrxcegeMITFnpmkJOZQsS+PSTSi+3L8nq+mAC/T6YLudJGwPkI0yLQhTSm/szFJInEHg+Y8a3kED/lco15R1uCC/mdFA1/jlroaPpT8kPeyLBaAn32k6FGWCsYzPP92W3zzFwParBoYcpfTwU0nImAvSN+sD35okgxbf/ATHxKDd6XB54rsvU1OdapMVwcxAedQs6NRaIAkjwJ63Cb9BszhNuKGrGm/WOdk7GukWOcWZWEXCqz7dt79hALzghziCHJiGH5WfgJEaXKdT6J2Y8porQocV+86IcmzIha+BhlcDboxVUK9rk77Dm7H9Cis5A6/RGtM44AJg9dOTOGU1eIxjuPv2gfifrHH3XtH9pKwKhKUcjdjmNJ7Cc0ZOU/t8nW8GW0K8x8b0QcrkL8PKVNuMRhqDyqJhFoZZIEyPhtFZEXiqK2XdMgdZJidG1+uV2k3HWhzMZZQli77diXQrn35M6w+6euVm7Gu+s+tAuNeat3atZW1zfShWm0d0zDVM0DdORmIZh12UaetV+U+rF8kqtQI+6VtSjaZmiHnWJHvW61GgIanwG2F3TNxLUSVJ6l+ksb09clFnElKoJ4mNuXAZSEcZTBi3eZ7UKxPRuETFbZL4hAay2IcEUABu8/Uz6Q7vDnDIbQgFrZgek0ptOoQ8xSIbX6nE94OmqQ7UCIA21HEj9pkha/wkPpt3bg9mCGsdoRloPdtCD6Xf2YI7oweirvI7pdBEETF3hZBWl09ZTRPbgwhtF0c9JHJL416RNJS5SKWWKelOqdEXb/lwRGFDZG3JX1/nLEoWWYkxDAkO1Nad3LDau2f3qXQlEmgQiqy6IMs7kMHrzAZkh3KKUoWSY9r1REtNMA6oMQINnzIJ3SNYIL2m42XywZPkiCFYkV1QI2jXH7NWAsWGeYIkyZ1kfxmK2KI/xH3Rs8wKYH/bSv+LAmGZxJjhL4PzpYRLTseyLBxDKt3CGqcpxPCWxZJ7VcuY0zkj9wm05I6ZR3mmkuGu8hbUcVktpnCsQ0zov8QQ+RBht7o/rFogLcK0bu5NNUnPqwk7M8DDsfEha4I4B53TvDZyY0GHA4RASuBIHVTY2Y8RapfPlsB1ETwXePmXiJVumU7NUVfXIizmoXhT5NNDn8yUaBl2O76NWDcDOkznsHV3IzSN1wSqWvZe9Kl+BroAO+7HWdt9Cjg7bgbZAh9oykqokw7WbiN/MzCm0tm2MHFuemyplgXIZC8yqcLWLuBonZjqc2mAVs1EtrOfCaph6w2DNHlZ03rfMWlWJaHmmWYQ1v/NAqch6defRLAItWW/rqlmlm2STNTEJNh6+1Ix0XSAdoVCRL1t5JbAW15K0rrj6d2tQxaxXa75Xe2lrb/BtgPGKmarWeM8Fdd8nN8B8xUxVhaFTRaHxTSKnChA2u42LhTTJfqV20nw2rk2LhTQx19XCei6stta4WEiayGonU1dmqpsYC4k5qnY6dSaszYuFxAxVa75Xe2m9cbGQLqasWuM9F9TmxUK67CxgumwYFYDNjmmxs08PWdeSU1yMGeZF57jK9gDxysdWNEfhAoRTGFAMVocapIqJDp4La9ySZ9Vrl85egGZIjhiqsgCttp0iupg+O5dzhpxz/HjgGMV4ynedCSvgQURxSdnSPHY0abeDQBzZovdtiSM7gFcJcfhmVFrjKQ4TvTaTH5dlhc4dMiuaxewTyLTuTqBjRw8V1nbu3Bq6MqtQfjYqPy1Rj8FT/zmM/aVvXZYXkG5t0urCSsziDeEMxP6Ve9J+KVgMWy+HRbbxqD5YxCxca0Kpe+s2zoTE1Nr/z4RswyyHpSIToh93X+2SlOW+H0cf/Qs= \ No newline at end of file diff --git a/images/seo-node.png b/images/seo-node.png new file mode 100644 index 0000000..7b0562f Binary files /dev/null and b/images/seo-node.png differ diff --git a/product-overview.md b/product-overview.md new file mode 100644 index 0000000..ecddb57 --- /dev/null +++ b/product-overview.md @@ -0,0 +1,92 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` +- [Intel® Smart Edge Open Developer Guide](#intel-smart-edge-open-developer-guide) + - [What is Intel Smart Edge Open?](#what-is-intel-smart-edge-open) + - [How It Works](#how-it-works) + - [Control Plane](#control-plane) + - [Edge Node](#edge-node) + - [Next Steps](#next-steps) + +# Intel® Smart Edge Open Developer Guide +Version 21.09 + +## What is Intel® Smart Edge Open? +Intel® Smart Edge Open is an edge computing software toolkit that lets you build platforms optimized for the edge. +Platforms created with Intel® Smart Edge Open are capable of hosting a wide range of services, from network functions such as 5G RAN and 5G core, to AI, media processing, and security workloads. + +Compared to cloud platforms, edge platforms are resource constrained, yet require higher network performance and more autonomy. Edge platforms have strong hardware affinity and face significantly more threat vectors. Intel® Smart Edge Open addresses the challenges of creating edge platforms. It provides a toolkit of functionality selected from across the cloud native landscape, tuned for the edge. + +As cloud architectures disaggregate in the era of 5G, opportunities emerge for new types of applications and services delivered from new locations such as enterprise premises, telco infrastructures, and hyperscaler points of presence. These locations include: +- **The on-premises edge:** Typically located in an enterprise. +- **The access edge:** Located at or near a 5G base station. +- **The near edge:** Aggregation point hosting a distributed UPF (user plane function). +- **The regional data center:** Hosting a next generation central office with wireless wireline convergence. + +Intel® Smart Edge Open experience kits offer you a starting point to create platforms for edge locations. Experience kits simplify the deployment of complex network architectures, significantly reducing development time and cost. Experience kits combine 5G capabilities and cloud-native components to simplify the deployment of complex network architectures, significantly reducing development time and cost. + +[![Edge Locations](images/overview1.png)](images/overview1.png) + +The Intel® Smart Edge Open portfolio of experience kits includes: +- **Developer Experience Kit**: Our only experience kit not designed for a specific edge location, this may be your starting point if you are new to building edge platforms. It provides the base capabilities you need to run typical containerized edge services, including networking, security, and telemetry. +- **5G Private Wireless Experience Kit with Integrated RAN:** Adds 5G capabilities for creating enterprise private wireless solutions with a containerized 5G radio access network and core. +- **uCPE Experience Kit**: Provides SD-WAN and firewall capabilities to enable a secure access service edge (SASE) deployment for applications. +- **Access Edge Experience Kit**: The starting point for building O-RAN compliant edge platforms. +- **Near Edge Experience Kit**: The starting point for building edge platforms that reside in a telco cloud. + +The Developer Experience Kit is available under the Apache 2.0 license. All other experience kits require a royalty-free Intel proprietary license. [Request a license](https://smart-edge-open.github.io/intel-smart-edge-open/request-license/). + +Experience kits are composed of sets of building blocks from the open community or from Intel. Building blocks are units of functionality that have been carefully selected and optimized to address the services targeted for use cases at a specific edge location. You can consume experience kits in their entirety, or use only the building blocks required by your own use case. + +Common building blocks used by most or all experience kits include: +- **Resource management:** Provides identification, configuration, allocation, and continuous monitoring of the hardware and software resources on the edge cluster. The resource management building block allows edge service providers to offer differentiating and/or revenue-generating services that require leveraging specific hardware features. +- **Accelerator support:** Support for accelerator resource allocation. Enables AI inferencing for applications, high-performance and low-latency packet pre-processing on network cards, and offloading for network functions such as eNB/gNB forward error correction (FEC). +- **Container network interfaces:** Enables highly-coupled communication between containers, as well as communications between pods on the same node or across nodes. +- **Telemetry and Monitoring:** Combines application telemetry, hardware telemetry, and events to create a heat-map across the edge cluster and enables the orchestrator to make scheduling decisions. +- **Software Development Kits:** Intel® Smart Edge Open supports SDKs that enable the development of edge services and network functions optimized for the edge. + +## How It Works + +Experience kits are built on top of Kubernetes, a production-grade platform for managing containerized workloads and services. Experience kits customize and extend the Kubernetes control plane and edge node with microservices, third-party applications, extensions, and optimizations. The control plane node and one or more edge nodes form an Intel® Smart Edge Open edge cluster. + +[![Smart Edge Open Logical](images/seo-node.png)](images/seo-node.png) + +The Intel® Smart Edge Open node architecture is specialized for each experience kit, to enable developers to create solutions for specific use cases at a given edge location. + +The edge node for the Developer Experience Kit: +[![Developer Experience Kit edge node diagram](/experience-kits/images/dek-component-diagram.png)](images/dek-component-diagram.png) + +The edge node for the 5G Private Wireless Experience Kit with Integrated RAN: +[![5G Private Wireless edge node diagram](images/pwek-aio.drawio.png)](images/pwek-aio.drawio.png) + +### Control Plane + +The Intel® Smart Edge Open control plane is used to configure edge nodes and the services that run on them. Functions of the control plane include: + +- Configuring the hardware platform that hosts applications and network functions +- Configuring 4G, 5G, and Wi-Fi network functions +- Detecting hardware and software capabilities of the edge cluster and using that information to schedule applications and network functions +- Setting up network and DNS policies for applications and network functions +- Enabling collection of hardware infrastructure, software, and application monitoring +- Exposing edge cluster capabilities northbound to a controller + + +In a single node cluster deployment, control plane services co-exist on the same physical node as the edge node. + +### Edge Node +The Intel® Smart Edge Open edge node manages the edge services, including the APIs used to discover those services. Features of the edge node include: + +- Support for the Docker container runtime and virtualization infrastructure (libvirt*, Open vSwitch (OVS), etc.) to support VMs. +- Platform pods consisting of services that enable the configuration of hardware resources on the node for a particular deployment, operators for accelerators and device plugins enabling hardware resource allocation to an application pod. +- System pods consisting of services that enable reporting the hardware and software features of each node to the control plane, providing resource isolation service for pods and DNS service to the cluster. +- Agents that expose edge node configuration to the control plane services (DNS, 4G, 5G, Wi-Fi and Telemetry) +- Telemetry for the edge node at the hardware, operating system, infrastructure, and application levels. +- Support for real-time kernel for low latency applications and network functions like 4G and 5G base station and non-real-time kernel. + +Intel® Smart Edge Open (formerly known as OpenNESS) is a [CNCF Certified Kubernetes*](https://landscape.cncf.io/card-mode?organization=intel&selected=open-ness) product, ensuring a consistent, updated, confirmable Kubernetes-conformant implementation. + +## Next Steps +- Get started with the [Developer Experience Kit](/experience-kits/developer-experience-kit.md). +- To get started with our licensed experience kits, [request a license](https://smart-edge-open.github.io/intel-smart-edge-open/request-license/). + diff --git a/release-notes/release-notes-se-open-DEK-21-09.md b/release-notes/release-notes-se-open-DEK-21-09.md new file mode 100644 index 0000000..2d3574e --- /dev/null +++ b/release-notes/release-notes-se-open-DEK-21-09.md @@ -0,0 +1,112 @@ +```text +SPDX-License-Identifier: Apache-2.0 +Copyright (c) 2021 Intel Corporation +``` +- [Overview](#overview) + - [Operating System](#operating-system) + - [Package Versions](#package-versions) + - [Hardware](#hardware) + - [Server](#server) + - [Configuration](#configuration) + - [Building Blocks](#building-blocks) + - [Known Issues](#known-issues) + - [Disclaimer](#disclaimer) + +# Overview + +The 21.09 release of Intel® Smart Edge Open introduces the Developer Experience Kit. Edge developers can use the Developer Experience Kit as a cloud-native platform for testing edge applications. + +This document describes the hardware and software configuration used to test the experience kit. For architecture and installation information, see the [Developer Experience Kit documentation](/experience-kits/developer-experience-kit.md). + +## Operating System + +Ubuntu 20.04.2 LTS (Focal Fossa) + +## Package Versions + + | Package | Version | + | ----------- |:-----------------:| + | Kubernetes | 1.21.1 | + | Docker | 20.10.2 | + | Calico | 3.19 | + | Harbor | 2.3.2 | + | Multus | 3.7.1 | + | NFD | 0.8.2 | + | StatsD | 0.22.0 | + | Prometheus | 2.30.0 | + | Grafana | 8.1.5 | + | cAdvisor | 0.40.0 | + | SR-IOV Network Operator | c608feee2dd74b4b99e44d4950b3651040e68a65 | + + +## Hardware + +### Server + +Dell PowerEdge R750 Server R750 motherboard + +### Configuration + +2 Intel® Xeon® Gold 6338N Processors: 2.2G, 32C/64T, 11.2GT/s, 48M Cache, Turbo, HT (185W) DDR4-2666 +1 2.5 Chassis +1 N9 +1 No Rear Storage +1 Additional Processor Selected +1 NVMe Backplane +1 GPU Enablement +1 iDRAC,Legacy Password +1 iDRAC Group Manager, Disabled +1 2.5" Chassis with up to 16 NVMe Drives +1 PowerEdge 2U LCD Bezel +1 Riser Config 7, 2x8, 2x16 slots +1 Dell EMC Luggage Tag +1 No Quick Sync +1 Performance Optimized +1 3200MT/s RDIMMs +16 32GB RDIMM, 3200MT/s, Dual Rank +1 iDRAC9, Enterprise 15G +1 No Hard Drive +1 1.6TB Enterprise NVMe Mixed Use AG Drive U.2 Gen4 with carrier +1 BOSS-S2 controller card + with 2 M.2 480GB (RAID 1) +1 No Controller +1 Heatsink for 2 CPU with GPU configuration +1 Dual, Hot-Plug,Power Supply Redundant (1+1), 1400W, Mixed Mode +2 C13 to C14, PDU Style, 10 AMP, 6.5 Feet (2m), Power Cord +1 Trusted Platform Module 2.0 V3 +1 Order Configuration Shipbox Label (Ship Date, Model, Processor Speed, HDD Size, RAM) +1 Asset Tag - ProSupport (Website, barcode, Onboard MacAddress) +1 BOSS Cables and Bracket for R750 (Riser 1) +1 PowerEdge R750 Shipping Material +1 GPU Ready Configuration Cable Install Kit R750 +1 Intel E810-XXV Dual Port 10/25GbE SFP28, OCP NIC 3.0 +1 Intel E810-XXV Dual Port 10/25GbE SFP28 Adapter, PCIe Full Height +1 Very High Performance Fan x6 V3 +1 Fan Foam, HDD 2U +1 Power Saving Dell Active Power Controller +1 C30, No RAID for NVME chassis Software +1 TPM Module + +## Building Blocks +The Developer Experience Kit uses the following building blocks: +- **SR-IOV Network Operator** - Operator for provisioning and configuring the SR-IOV CNI and device plugins supporting Intel network adapters. Use this feature to allocate dedicated high performance SR-IOV virtual interfaces to the application pods on the cluster. +- **Multus**: A CNI that enables attaching multiple network interfaces to a Kubernetes pod. Typically, in Kubernetes each pod only has one network interface, apart from a loopback. With Multus you can create a multi-homed pod with multiple interfaces. +- **Harbor**: An open source registry that secures artifacts with policies and role-based access control. Use Harbor to store application container images and Helm charts that can easily be deployed on the node. +- **Prometheus, Grafana, cAdvisor, StatsD**: Cloud native telemetry, observability, logging, monitoring and dashboard component, used to create an observability framework for application and resource utilization. +- **Node Feature Discovery (NFD)**: Software that detects hardware features available on each node in a Kubernetes cluster, and advertises those features using node labels. Use the labels created by NFD to deploy applications on nodes that meet required criteria for reliable service. +- **Calico**: An open source networking and network security solution for containers, virtual machines, and native host-based workloads. Calico supports network policy and high-performance data plane. It is the default CNI on the edge node cluster created by the Developer Experience Kit. + +## Known Issues + +- Edge Software provisioner - Occasionally the USB image is not built and SE-O DEK provision exits with an error. + - Mitigation: Retry the ESP based provisioning. +- Edge Software provisioner - sometimes builds an incorrect image and machine fails to boot using the image. + - Mitigation: Retry the ESP based provisioning. +- Edge Software provisioner - Occasionally when running ESP's build.sh the /dev/null node of the provisioning server machine is corrupted. + - Mitigation: run command: `rm -f /dev/null; mknod -m 666 /dev/null c 1 3` or reboot the server +- Edge Software provisioner - Cannot boot USB images using legacy BIOS. + - Mitigation: Use UEFI BIOS + + +## Disclaimer + +Smart Edge Open 21.09 does not include the latest functional and security updates. Smart Edge Open is targeted to be released in Q4'2021 and will include additional functional and security updates. Customers should update to the latest version as it becomes available. \ No newline at end of file