Rewrite readme

README.md

@@ -1,35 +1,65 @@
 ## Description
-`make-argocd-fly` is a tool to generate Kubernetes manifests that can be deployed with ArgoCD in a multi cluster environments. It makes it easier to develop and maintain ArgoCD applications by providing a way to render Jinja2 and Kustomize files, run yaml and kube linters and automatically generate ArgoCD `Application` resources.
+`make-argocd-fly` is a tool designed to simplify the generation of Kubernetes manifests for deployment in complex, multi-cluster, and multi-application environments. By leveraging YAML, Jinja2 templates, Kustomize files, and Helm charts via Kustomize. This tool streamlines the process of creating and maintaining Kubernetes resources. While the generated manifests can be deployed using any external deployment tool, such as kubectl, `make-argocd-fly` offers native integration with ArgoCD, enhancing the deployment experience.
 
-The idea is that you write your resources in YAML, Jinja2 or Kustomize (including helmCharts), write a configuration file that describes where to deploy them and then run `make-argocd-fly` to generate the final manifests. This way it is transparent what is being deployed and where, it is easier to maintain and develop the resources and it is easier to debug issues.
+With `make-argocd-fly`, you can develop your resources in various formats, write a configuration file to specify deployment details, generate the final manifests effortlessly, and then push them to a repository for ArgoCD to deploy them in a Kubernetes cluster. This approach ensures transparency in what is being deployed and where, simplifies maintenance and development, and aids in debugging issues.
 
 ## Features
-- Jinja2 and Kustomize rendering
-- ArgoCD `Application` resources are generated automatically
-- `App-of-apps` pattern is natively supported
-- Use `include` or `include_raw` in Jinja2 templates to render content of external files
-- Global, per environment and per application Jinja2 variables
-- Variables referencing other variables are supported
-- Source resources can reside in subdirectories
+- **Jinja2 and Kustomize Rendering:** Seamlessly render Kubernetes manifests using Jinja2 templates and Kustomize, including Helm charts.
+- **ArgoCD App-of-Apps Pattern Support:** Native support of the app-of-apps pattern for organizing complex applications with automatic generation of ArgoCD `Application` resources for streamlined deployment.
+- **External File Inclusion:** Use `include` or `include_raw` in Jinja2 templates to render content from external files.
+- **Flexible Variable Management:** Support for global, per-environment, and per-application Jinja2 variables, with the ability to reference other variables.
+- **Subdirectory Resource Management:** Source resources can reside in subdirectories for better organization and management.
+
+## How to start
+### Prerequisites
+- `libyaml` is expected to be installed locally for speeding up YAMLs generation.
+- `kustomize` (IF USED) is expected to be installed locally.
+- `helm` (IF USED) is expected to be installed locally.
+- `yamllint` (IF USED) is expected to be installed locally (https://github.com/adrienverge/yamllint).
+- `kube-linter` (IF USED) is expected to be installed locally (https://github.com/stackrox/kube-linter).
+
+### Configuration file (`config.yml`)
+The following structure is expected:
+```
+envs:
+  <environment_name_1>:
+    apps:
+      <application_name_1>: {}
+      #<application_name_N>: {}  ## additional applications for the environment
+  #<environment_name_N>:  ## additional environments
+
+vars:
+  <variable_name_1>: <variable_value_1>
+  #<variable_name_N>: <variable_value_N>  ## additional variables
+```
+
+Application names shall correspond to the relative paths from the source directory to the application directory, e.g., ```grafana```, ```path/to/grafana``` .
+
+Example configuration file:
+```tests/examples/app_types/config.yml```
+
+### Source directory structure
+Example directory structure:
+```tests/examples/app_types/source```
 
-## Usage
+## Execution
 ```
 python3 -m venv .venv
 . .venv/bin/activate
 pip install make-argocd-fly
 
 make-argocd-fly -h
 usage: main.py [-h] [--root-dir ROOT_DIR] [--config-file CONFIG_FILE] [--source-dir SOURCE_DIR] [--output-dir OUTPUT_DIR] [--tmp-dir TMP_DIR] [--render-apps RENDER_APPS]
-               [--render-envs RENDER_ENVS] [--skip-generate] [--preserve-tmp-dir] [--clean] [--print-vars] [--var-identifier VAR_IDENTIFIER] [--yaml-linter] [--kube-linter]
-               [--loglevel LOGLEVEL]
+               [--render-envs RENDER_ENVS] [--skip-generate] [--preserve-tmp-dir] [--clean] [--print-vars] [--var-identifier VAR_IDENTIFIER] [--skip-latest-version-check]
+               [--yaml-linter] [--kube-linter] [--loglevel LOGLEVEL]
 
 Render ArgoCD Applications.
 
 options:
   -h, --help            show this help message and exit
   --root-dir ROOT_DIR   Root directory (default: current directory)
   --config-file CONFIG_FILE
-                        Configuration file (default: config.yaml)
+                        Configuration file (default: config.yml)
   --source-dir SOURCE_DIR
                         Source files directory (default: source)
   --output-dir OUTPUT_DIR
@@ -45,46 +75,96 @@ options:
   --print-vars          Print variables for each application
   --var-identifier VAR_IDENTIFIER
                         Variable prefix in config.yml file (default: $)
+  --skip-latest-version-check
+                        Skip latest version check
   --yaml-linter         Run yamllint against output directory (https://github.com/adrienverge/yamllint)
   --kube-linter         Run kube-linter against output directory (https://github.com/stackrox/kube-linter)
   --loglevel LOGLEVEL   DEBUG, INFO, WARNING, ERROR, CRITICAL
 ```
 
-## Configuration
-### config.yml
-Example configuration file:
-```tests/manual/config.yml```
+## Advanced usage
+### ArgoCD integration
+ArgoCD app-of-apps pattern (https://argo-cd.readthedocs.io/en/stable/operator-manual/cluster-bootstrapping/) is one of the ways to deploy applications in a Kubernetes cluster. `make-argocd-fly` supports this pattern by generating ArgoCD `Application` resources for each application in the configuration file. The generated resources can be deployed in the cluster using ArgoCD, which will automatically deploy the applications specified in the `Application` resources. The app-of-apps pattern can be nested, which allows for organizing applications in a hierarchical structure, where only the top-level application is deployed externally.
+
+To make use of ArgoCD app-of-apps pattern in `make-argocd-fly`, specify the following variables and application parameters:
+```
+envs:
+  <environment_name>:
+    apps:
+      <bootstrap_application>: {}  ## application that shall be deployed externally that will deploy other applications
+      <application_name>:
+        app_deployer: <bootstrap_application>  ## application that will deploy this application
+        app_deployer_env: <environment_name>  ## (OPTIONAL) for multi-environments with single ArgoCD deployment
+        project: <project_name>  ## ArgoCD project name
+        destination_namespace: <namespace>  ## default namespace where the application resources will be deployed
+vars:
+  argocd:
+    api_server: <argocd_api_server>  ## kube-apiserver address
+    namespace: <argocd_namespace>  ## namespace for ArgoCD `Application` resources
+    repo_url: <argocd_repo_url>  ## URL to the Git repository
+    target_revision: <argocd_target_revision>  ## target revision for the Git repository
+    sync_policy: <argocd_sync_policy>  ## (OPTIONAL) default: {}
+    finalizers: <argocd_finalizers>  ## (OPTIONAL) default: []
+    ignoreDifferences: <argocd_ignoreDifferences>  ## (OPTIONAL) default: []
+```
+
+### Magic variables
+The folloving variable names are reserved for internal purposes and shall not be used in the configuration file:
+- __application
 
-### Source directory structure
-Example directory structure:
-```tests/manual/source```
+The folloving variable are automatically populated and can be referenced without explicit definition:
+- env_name
+- app_name
 
-When kustomization overlays are used, kustomization base directory shall be named `base`, overlay directories shall be named after the corresponding environments names.
+### Variables scopes
+Variables can be defined at the root level, per-environment, and per-application. Variables defined at the root level are global and can be referenced in any application. Variables defined per-environment are specific to the environment and can be referenced in any application within that environment. Variables defined per-application are specific to the application and can be referenced only in that application.
 
-### app parameters
-- `app_deployer` - name of the application that will deploy this application
-- `project` - ArgoCD project name
-- `destination_namespace` - namespace where the application will be deployed
-- `app_deployer_env` - (OPTIONAL) environment of the application that will deploy this application
-- `vars` - (OPTIONAL) application specific jinja2 variables
+When a variable is defined in multiple scopes the following priority rules apply: global variable < environment variable < application variable
 
-### Jinja2 extensions
-To render a template in the current jinja2 template, use the following block:
+```
+envs:
+  <environment_name>:
+    apps:
+      <application_name>:
+        vars:
+          <variable_name>: <variable_value>
+    vars:
+      <variable_name>: <variable_value>
+
+vars:
+  <variable_name>: <variable_value>
+```
+
+### Variables in `config.yml`
+Variables can be referenced in the configuration file (including in the application parameters section) using the following syntax:
+```${var_name}``` and  ```${var_name[dict_key][...]}```.
+
+Variables can also be used as substring values:
+```prefix-${var_name}-suffix```.
+
+### Jinja2 templates
+To include file content in the current Jinja2 template, use the following block:
 
 ```
 {%- filter indent(width=4) %}
-{% include 'app_4/files/file.json.j2' %}
+{% include_raw 'app_4/files/file.json' %}
 {% endfilter %}
 ```
 
-To include file content in the current jinja2 template, use the following block:
+Example:
+```tests/manual/source/app_4```
+
+To render a template in the current jinja2 template, use the following block:
 
 ```
 {%- filter indent(width=4) %}
-{% include_raw 'app_4/files/file.json' %}
+{% include 'app_5/files/file.json.j2' %}
 {% endfilter %}
 ```
 
+Example:
+```tests/manual/source/app_5```
+
 To perform a DNS lookup, use the following filter:
 
 ```
@@ -93,46 +173,24 @@ To perform a DNS lookup, use the following filter:
 
 Ansible filters are supported as well: https://pypi.org/project/jinja2-ansible-filters/
 
-### kustomization.yml
-Files referenced in the `resources` section shall be named after Kubernetes resource type + `_` + resource name. Example:
+### Kustomize
+Local files referenced in the `resources` section shall be named after Kubernetes resource type + `_` + resource name:
 
 ```
 resources:
   - deployment_nginx.yml
   - serviceaccount_nginx-prod.yml
 ```
-### Variable names
-The folloving variable names are reserved (at the root level) and shall not be used in the configuration file:
-- __application
-- env_name
-- app_name
 
-### Referencing variables in config.yml
-Variables can be referenced in the configuration file (including in *app parameters*) using the following syntax:
-```${var_name}``` and  ```${var_name[dict_key][...]}```.
+Example:
+```tests/manual/source/app_1```
 
-Variables can also be used as substring values:
-```prefix-${var_name}-suffix```.
-
-
-### Expected variables
-The folloving variables are expected to be provided:
-- argocd.api_server
-- argocd.namespace
-- argocd.repo_url
-- argocd.target_revision
-
-### Optional variables
-- argocd.sync_policy
-- argocd.finalizers
-- argocd.ignoreDifferences
+When kustomization overlays are used, kustomization base directory shall be named `base`, overlay directories shall be named after the corresponding environments names.
+Example:
+```tests/manual/source/app_2```
 
 ## Caveats
-- `kustomize` and `helm` are expected to be installed locally.
-- `kube-linter` is expected to be installed locally (https://github.com/stackrox/kube-linter).
-- `libyaml` is expected to be installed locally for speeding up YAMLs generation.
 - Comments are not rendered in the final output manifests.
-- Initial App-of-Apps application (`bootstrap`) shall be deployed externally.
 
 ## For developers
 ### Build instructions

tests/examples/app_types/config.yml

@@ -0,0 +1,11 @@
+envs:
+  example:
+    apps:
+      01_yaml_app: {}
+      02_jinja2_app: {}
+      03_kustomize_app: {}
+      04_kustomize_helm_app: {}
+
+vars:
+  namespace: kube-default
+  version: 0.1.0

tests/examples/app_types/output/example/01_yaml_app/configmap_example-app.yml

@@ -0,0 +1,6 @@
+---
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: example-app
+  namespace: default

tests/examples/app_types/output/example/01_yaml_app/deployment_example-app.yml

@@ -0,0 +1,6 @@
+---
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: example-app
+  namespace: default

tests/examples/app_types/output/example/01_yaml_app/service_example-app.yml

@@ -0,0 +1,6 @@
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: example-app
+  namespace: default

tests/examples/app_types/output/example/02_jinja2_app/deployment_example-app.yml

@@ -0,0 +1,6 @@
+---
+kind: Deployment
+apiVersion: apps/v1
+metadata:
+  name: example-app
+  namespace: kube-default

tests/examples/app_types/output/example/02_jinja2_app/service_example-app.yml

@@ -0,0 +1,6 @@
+---
+kind: Service
+apiVersion: v1
+metadata:
+  name: example-app
+  namespace: kube-default

tests/examples/app_types/output/example/03_kustomize_app/deployment_example-app.yml

@@ -0,0 +1,6 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: example-app
+  namespace: kube-default

tests/examples/app_types/output/example/03_kustomize_app/service_example-app.yml

@@ -0,0 +1,6 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: example-app
+  namespace: kube-default

tests/examples/app_types/output/example/04_kustomize_helm_app/deployment_hello-world.yml

@@ -0,0 +1,40 @@
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app.kubernetes.io/instance: hello-world
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/name: hello-world
+    app.kubernetes.io/version: 1.16.0
+    helm.sh/chart: hello-world-0.1.0
+  name: hello-world
+spec:
+  replicas: 1
+  selector:
+    matchLabels:
+      app.kubernetes.io/instance: hello-world
+      app.kubernetes.io/name: hello-world
+  template:
+    metadata:
+      labels:
+        app.kubernetes.io/instance: hello-world
+        app.kubernetes.io/name: hello-world
+    spec:
+      containers:
+        - image: nginx:1.16.0
+          imagePullPolicy: IfNotPresent
+          livenessProbe:
+            httpGet:
+              path: /
+              port: http
+          name: hello-world
+          ports:
+            - containerPort: 80
+              name: http
+              protocol: TCP
+          readinessProbe:
+            httpGet:
+              path: /
+              port: http
+      serviceAccountName: hello-world

tests/examples/app_types/output/example/04_kustomize_helm_app/service_hello-world.yml

@@ -0,0 +1,21 @@
+---
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    app.kubernetes.io/instance: hello-world
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/name: hello-world
+    app.kubernetes.io/version: 1.16.0
+    helm.sh/chart: hello-world-0.1.0
+  name: hello-world
+spec:
+  ports:
+    - name: http
+      port: 80
+      protocol: TCP
+      targetPort: http
+  selector:
+    app.kubernetes.io/instance: hello-world
+    app.kubernetes.io/name: hello-world
+  type: ClusterIP

tests/examples/app_types/output/example/04_kustomize_helm_app/serviceaccount_hello-world.yml

@@ -0,0 +1,11 @@
+---
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  labels:
+    app.kubernetes.io/instance: hello-world
+    app.kubernetes.io/managed-by: Helm
+    app.kubernetes.io/name: hello-world
+    app.kubernetes.io/version: 1.16.0
+    helm.sh/chart: hello-world-0.1.0
+  name: hello-world

tests/examples/app_types/source/01_yaml_app/configmap_example_app.yml

@@ -0,0 +1,5 @@
+kind: ConfigMap
+apiVersion: v1
+metadata:
+  name: example-app
+  namespace: default