Skip to content

Commit

Permalink
Merge pull request #4 from adberger/main
Browse files Browse the repository at this point in the history 
Experimental changes
  • Loading branch information
@adberger
adberger authored Oct 21, 2024
2 parents 1b3e34e + 6485131 commit 2b419a9
Show file tree
Hide file tree
Showing 13 changed files with 247 additions and 50 deletions.
22 changes: 22 additions & 0 deletions .vscode/launch.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "Launch Package",
"type": "go",
"request": "launch",
"mode": "auto",
"program": "subst/main.go",
"args": [
"render",
"-v",
"debug",
"--skip-decrypt",
"/Users/foo/cluster"
],
}
]
}
41 changes: 10 additions & 31 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,15 +2,13 @@

![subst](./img/subst.png "subst")

__Currently under development and testing. I don't take any responsability for unexpected behavior. You know what you are doing :3__

A simple extension over kustomize, which allows further variable substitution and introduces simplified yet strong secrets management (for multi tenancy use-cases). Extends to functionality of kustomize for ArgoCD users.

# Functionality

The idea for subst is to act as complementary for kustomize. You can reference additional variables for your environment or from different kustomize paths, which are then accesible across your entire kustomize build. The kustomize you are referencing to is resolved (it's paths). In each of these paths you can create new substition files, which contain variables or secrets, which then can be used by your kustomization. The final output is your built kustomization with the substitutions made.
The idea for subst is to act as complementary for kustomize. You can reference additional variables for your environment or from different kustomize paths, which are then accesible across your entire kustomize build. The kustomize you are referencing to is resolved (it's paths). In each of these paths you can create new substitution files, which contain variables or secrets, which then can be used by your kustomization. The final output is your built kustomization with the substitutions made.

By default the all files are considered using this regex `(.*subst\.yaml|.*(ejson|vars))`. You can change the regex using:
By default the all files are considered using this regex `(subst\.yaml|.*(ejson))`. You can change the regex using:

```
subst render . --file-regex "custom-values\\.yaml"
Expand Down Expand Up @@ -133,21 +131,21 @@ Note that directories do not resolve by recursion (eg. `/test/build/` only colle

### Environment

for environment variables which come from an argo application (`^ARGOCD_ENV_`) we remove the `ARGOCD_ENV_` and they are then available in your substitutions without the `ARGOCD_ENV_` prefix. This way they have the same name you have given them on the application ([Read More](https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/#using-environment-variables-in-your-plugin)). All the substions are available as flat key, so where needed you can use environment substitution.
For environment variables which come from an argo application (`^ARGOCD_ENV_`) we remove the `ARGOCD_ENV_` and they are then available in your substitutions without the `ARGOCD_ENV_` prefix. This way they have the same name you have given them on the application ([Read More](https://argo-cd.readthedocs.io/en/stable/operator-manual/config-management-plugins/#using-environment-variables-in-your-plugin)). All the substitutions are available as flat key, so where needed you can use environment substitution.

## Spruce

[Spruce](https://github.com/geofffranks/spruce) is used to access the substition variables, it has more flexability than [envsubst](#environment-substitution). You can grab values from the available substitutions using [Spruce Operators](https://github.com/geofffranks/spruce/blob/main/doc/operators.md). Spurce is greate, because it's operators are valid YAML which allows to build the kustomize without any further hacking.
[Spruce](https://github.com/geofffranks/spruce) is used to access the substitution variables, it has more flexability than [envsubst](#environment-substitution). You can grab values from the available substitutions using [Spruce Operators](https://github.com/geofffranks/spruce/blob/main/doc/operators.md). Spurce is great, because it's operators are valid YAML which allows to build the kustomize without any further hacking.

## Secrets

You can both encrypt files which are part of the kustomize build or which are used for substitution. Currently for secret decryption we support both [ejson](https://github.com/Shopify/ejson) and [sops](https://github.com/mozilla/sops). You can use any combination of these decryption providers together. The principal for all decryption provider is, that they should load the private keys while a substiution build is made instead of having a permanent keystore. This allows for secret tenancy (eg. one secret per argo application). The private keys are loaded from kubernetes secrets, therefor the plugin also creates it's own kubeconfig.
You can both encrypt files which are part of the kustomize build or which are used for substitution. Currently for secret decryption we support [ejson](https://github.com/Shopify/ejson). The principal for the decryption provider is, that it should load the private keys while a substitution build is made instead of having a permanent keystore. This allows for secret tenancy (eg. one secret per argo application). The private keys are loaded from kubernetes secrets, therefor the plugin also creates it's own kubeconfig.

The secrets are loaded based on the environment variables `$ARGOCD_APP_NAME` and `$ARGOCD_APP_NAMESPACE` are used. If an application is in a project, the value of `$ARGOCD_APP_NAME` looks like this: `<project-name>_<application-name>`. For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project_my-app`. All special characters within are converted to `-` (dash). For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project-my-app`. So the secret reference is then `my-project-my-app` in the secret namespace (Assuming `--convert-secret-name=false`).
The secrets are loaded based on how the environment variables `$ARGOCD_APP_NAME` and `$ARGOCD_APP_NAMESPACE` are used. If an application is in a project, the value of `$ARGOCD_APP_NAME` looks like this: `<project-name>_<application-name>`. For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project_my-app`. All special characters within are converted to `-` (dash). For example, if the application `my-app` is in the project `my-project`, the value of `$ARGOCD_APP_NAME` is `my-project-my-app`. So the secret reference is then `my-project-my-app` in the secret namespace (Assuming `--convert-secret-name=false`).

By default the `--convert-secret-name` is enabled. This removes the project prefix from the secret. If you create an application `test` in the namespace `test-reserved` the plugin is looking for private keys in the secret `test` in the namespace `test-reserved`. The is not considered in this approach which helps endusers to keep it simple.
By default the `--convert-secret-name` is enabled. This removes the project prefix from the secret. If you create an application `test` in the namespace `test-reserved` the plugin is looking for private keys in the secret `test` in the namespace `test-reserved`.

The values for the secret name and namespace can also be set constant, however this way you lose the multi-tenancy aspect of the secrets management:
The values for the secret name and namespace can also be set explicitly, however this way you lose the multi-tenancy aspect of the secrets management:

```
subst render --secret-name static-name --secret-namespace static-namespace .
Expand All @@ -169,29 +167,16 @@ See below how to work with the different decryption providers.

### EJSON

[EJSON](https://github.com/Shopify/ejson) allows simple secrets management. I like it, because you can rencrypt secrets without having the private key, which is sometimes useful.

You can encrypt entire files using EJSON. The file must be in json format (which is fun for kustomize). The entire file will be encrypted, which may not bes useful in all cases.


### SOPS

[SOPS](https://github.com/mozilla/sops) is commonly known and also used by [FluxCD](https://fluxcd.io/flux/guides/mozilla-sops/).

[EJSON](https://github.com/Shopify/ejson) allows simple secrets management.

You can encrypt entire files using EJSON. The file must be in json format. The entire file will be encrypted, which may not bes useful in all cases.

### Kubernetes

For all decryptors you can create a kubernetes secret, which contains the private information for secret decryption.






# Running it


## Local installation

**Brew**
Expand All @@ -213,10 +198,4 @@ https://github.com/bedag/subst/releases

## ArgoCD Plugin



## CI/CD

TBD


14 changes: 12 additions & 2 deletions go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,17 +6,19 @@ toolchain go1.23.0

require (
github.com/BurntSushi/toml v1.4.0
github.com/KimMachineGun/automemlimit v0.6.1
github.com/MakeNowJust/heredoc v1.0.0
github.com/Masterminds/sprig/v3 v3.3.0
github.com/Shopify/ejson v1.5.2
github.com/adberger/spruce v0.0.7
github.com/geofffranks/simpleyaml v0.0.0-20161109204137-c9320f076de5
github.com/geofffranks/spruce v1.31.1
github.com/rs/zerolog v1.33.0
github.com/spf13/cobra v1.8.0
github.com/spf13/pflag v1.0.5
github.com/spf13/viper v1.14.0
github.com/starkandwayne/goutils v0.0.0-20190115202530-896b8a6904be
github.com/stretchr/testify v1.9.0
go.uber.org/automaxprocs v1.6.0
gopkg.in/yaml.v2 v2.4.0
gopkg.in/yaml.v3 v3.0.1
k8s.io/apimachinery v0.31.0
Expand All @@ -33,9 +35,13 @@ require (
github.com/Masterminds/semver/v3 v3.3.0 // indirect
github.com/aws/aws-sdk-go v1.55.5 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
github.com/cilium/ebpf v0.9.1 // indirect
github.com/cloudfoundry-community/vaultkv v0.7.0 // indirect
github.com/containerd/cgroups/v3 v3.0.1 // indirect
github.com/coreos/go-systemd/v22 v22.5.0 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.3 // indirect
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
github.com/docker/go-units v0.4.0 // indirect
github.com/dustin/gojson v0.0.0-20160307161227-2e71ec9dd5ad // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
Expand All @@ -46,6 +52,7 @@ require (
github.com/go-openapi/jsonpointer v0.20.0 // indirect
github.com/go-openapi/jsonreference v0.20.2 // indirect
github.com/go-openapi/swag v0.22.4 // indirect
github.com/godbus/dbus/v5 v5.0.4 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/gnostic-models v0.6.8 // indirect
Expand Down Expand Up @@ -77,12 +84,15 @@ require (
github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/onsi/gomega v1.33.1 // indirect
github.com/opencontainers/runtime-spec v1.0.2 // indirect
github.com/pbnjay/memory v0.0.0-20210728143218-7b4eea64cf58 // indirect
github.com/pelletier/go-toml v1.9.5 // indirect
github.com/pelletier/go-toml/v2 v2.0.6 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/shopspring/decimal v1.4.0 // indirect
github.com/sirupsen/logrus v1.8.1 // indirect
github.com/spf13/afero v1.9.3 // indirect
github.com/spf13/cast v1.7.0 // indirect
github.com/spf13/jwalterweatherman v1.1.0 // indirect
Expand All @@ -94,7 +104,7 @@ require (
golang.org/x/crypto v0.27.0 // indirect
golang.org/x/net v0.27.0 // indirect
golang.org/x/oauth2 v0.21.0 // indirect
golang.org/x/sys v0.25.0 // indirect
golang.org/x/sys v0.26.0 // indirect
golang.org/x/term v0.24.0 // indirect
golang.org/x/text v0.18.0 // indirect
golang.org/x/time v0.3.0 // indirect
Expand Down
Loading

0 comments on commit 2b419a9

Please sign in to comment.