Stash Roles, and Bindings Automatically Pruned by ArgoCD

[DreamingRaven]

I am having difficulties with stash under ArgoCD. I notice that whenever a backup is triggered, stash is attempting to create roles and role bindings in a few different places. However, since these are tied to the base application of ArgoCD on creation they are pruned since they are not artefacts of the helm chart. Depending on how quickly these CRs are deleted by argocd, the service accounts for both the trigger and backupconfiguration can be missing permissions to do things like fetch secrets across namespaces, or access stash specific resources.

I have opted to manually define most of these roles, clusterRoles, roleBindings, and clusterRoleBindings, however the CRs should ideally be marked with owner references or the specific labels should be removed since it is the labels which are informing argocd that it is part of the base chart artifacts AFAIK.

I am not sure if this is related but I am also experiencing the following error:

0703 14:55:03.538152       1 log.go:245] FLAG: --add-dir-header="false"
I0703 14:55:03.538186       1 log.go:245] FLAG: --alsologtostderr="false"
I0703 14:55:03.538188       1 log.go:245] FLAG: --appbinding="postgres"
I0703 14:55:03.538190       1 log.go:245] FLAG: --appbinding-namespace="postgresql"
I0703 14:55:03.538192       1 log.go:245] FLAG: --backup-cmd="pg_dumpall"
I0703 14:55:03.538194       1 log.go:245] FLAG: --backupsession="postgres-gcs-1720018502"
I0703 14:55:03.538196       1 log.go:245] FLAG: --bucket="pluto-bucket-prod-stash"
I0703 14:55:03.538198       1 log.go:245] FLAG: --enable-cache="true"
I0703 14:55:03.538199       1 log.go:245] FLAG: --endpoint=""
I0703 14:55:03.538201       1 log.go:245] FLAG: --help="false"
I0703 14:55:03.538203       1 log.go:245] FLAG: --hostname="host-0"
I0703 14:55:03.538204       1 log.go:245] FLAG: --insecure-tls="false"
I0703 14:55:03.538206       1 log.go:245] FLAG: --kubeconfig=""
I0703 14:55:03.538208       1 log.go:245] FLAG: --license-apiservice="v1beta1.admission.stash.appscode.com"
I0703 14:55:03.538210       1 log.go:245] FLAG: --log-backtrace-at=":0"
I0703 14:55:03.538212       1 log.go:245] FLAG: --log-dir=""
I0703 14:55:03.538213       1 log.go:245] FLAG: --log-file=""
I0703 14:55:03.538221       1 log.go:245] FLAG: --log-file-max-size="1800"
I0703 14:55:03.538224       1 log.go:245] FLAG: --log-flush-frequency="5s"
I0703 14:55:03.538226       1 log.go:245] FLAG: --logtostderr="true"
I0703 14:55:03.538228       1 log.go:245] FLAG: --master=""
I0703 14:55:03.538230       1 log.go:245] FLAG: --max-connections="0"
I0703 14:55:03.538232       1 log.go:245] FLAG: --namespace="postgresql"
I0703 14:55:03.538233       1 log.go:245] FLAG: --one-output="false"
I0703 14:55:03.538235       1 log.go:245] FLAG: --output-dir="/stash-tmp/output"
I0703 14:55:03.538237       1 log.go:245] FLAG: --path="/stash/staging/lincoln/horus"
I0703 14:55:03.538240       1 log.go:245] FLAG: --pg-args=""
I0703 14:55:03.538242       1 log.go:245] FLAG: --provider="gcs"
I0703 14:55:03.538244       1 log.go:245] FLAG: --region=""
I0703 14:55:03.538246       1 log.go:245] FLAG: --retention-dry-run="false"
I0703 14:55:03.538247       1 log.go:245] FLAG: --retention-keep-daily="0"
I0703 14:55:03.538249       1 log.go:245] FLAG: --retention-keep-hourly="0"
I0703 14:55:03.538251       1 log.go:245] FLAG: --retention-keep-last="1"
I0703 14:55:03.538252       1 log.go:245] FLAG: --retention-keep-monthly="0"
I0703 14:55:03.538255       1 log.go:245] FLAG: --retention-keep-tags="[]"
I0703 14:55:03.538257       1 log.go:245] FLAG: --retention-keep-weekly="0"
I0703 14:55:03.538259       1 log.go:245] FLAG: --retention-keep-yearly="0"
I0703 14:55:03.538261       1 log.go:245] FLAG: --retention-prune="true"
I0703 14:55:03.538262       1 log.go:245] FLAG: --scratch-dir="/stash-tmp"
I0703 14:55:03.538264       1 log.go:245] FLAG: --skip-headers="false"
I0703 14:55:03.538266       1 log.go:245] FLAG: --skip-log-headers="false"
I0703 14:55:03.538267       1 log.go:245] FLAG: --stderrthreshold="2"
I0703 14:55:03.538269       1 log.go:245] FLAG: --storage-secret-name="***REDACTED***"
I0703 14:55:03.538271       1 log.go:245] FLAG: --storage-secret-namespace="***REDACTED***"
I0703 14:55:03.538273       1 log.go:245] FLAG: --user=""
I0703 14:55:03.538275       1 log.go:245] FLAG: --v="0"
I0703 14:55:03.538276       1 log.go:245] FLAG: --vmodule=""
I0703 14:55:03.538278       1 log.go:245] FLAG: --wait-timeout="300"
W0703 14:55:03.538335       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0703 14:55:03.620944       1 commands.go:109] Checking whether the backend repository exist or not....
[golang-sh]$ /bin/restic snapshots --json --no-lock --cache-dir /stash-tmp/restic-cache
I0703 14:55:04.596613       1 commands.go:447] sh-output: []
I0703 14:55:04.618725       1 backup.go:101] Waiting for the backend repository.....
panic: not found
goroutine 1 [running]:
stash.appscode.dev/postgres/pkg.must(...)
	/src/pkg/util.go:86
stash.appscode.dev/postgres/pkg.(*postgresOptions).setDatabaseCredentials(0xc0002e3208, 0xc000514340, 0xc000561ad8)
	/src/pkg/util.go:117 +0xbb6
stash.appscode.dev/postgres/pkg.(*postgresOptions).backupPostgreSQL(0xc0002e3208, {{0xc0005006a0, 0x20}, {0x1b4949c, 0xa}, {0x7ffdab1fd80d, 0x8}, {0x7ffdab1fd82d, 0xa}})
	/src/pkg/backup.go:201 +0x48a
stash.appscode.dev/postgres/pkg.NewCmdBackup.func1(0xc0002e3508?, {0xc00011ec00?, 0x0?, 0x1f?})
	/src/pkg/backup.go:91 +0x2e7
github.com/spf13/cobra.(*Command).execute(0xc0002e3508, {0xc00011ea00, 0x1f, 0x20})
	/src/vendor/github.com/spf13/cobra/command.go:983 +0xaca
github.com/spf13/cobra.(*Command).ExecuteC(0xc0002e2c08)
	/src/vendor/github.com/spf13/cobra/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(0x20?)
	/src/vendor/github.com/spf13/cobra/command.go:1039 +0x13
main.main()
	/src/cmd/stash-postgres/main.go:41 +0x71

Despite the following conditions on the respective backupsession:

status:
  conditions:
    - lastTransitionTime: '2024-07-03T14:55:04Z'
      message: Repository exist in the backend.
      reason: BackendRepositoryFound
      status: 'True'
      type: BackendRepositoryInitialized

Which I assume could be another potential permission error, where the backupsession was not found due to a lack of permissions. Leading to it not being passed into cobra as an argument.

Any help would be appreciated,

[waza-ari]

Have you ever figured out how to use Stash with ArgoCD?

It also tries to add a tmpdir volume to whatever resource you're backing up (STS, deployment, ...), not just injecting a sidecar pod. ArgoCD auto syncs these resources as well, removing the tmpdir volume, causing config errors when trying to deploy the sidecar pod, due to the volume not being available.

[DreamingRaven]

@waza-ari No, I couldn't get it to work properly for me, or for my needs.
Instead, I switched to using volsync.
Which, while a little more verbose, was exactly what I needed, and works with ArgoCD nicely.
It also has the volume populator, which means I can just delete a PVC and have the backup automatically pull to recreate it automatically.

I even put together a helm chart to abstract some of the volsync boilerplate: https://gitlab.com/GeorgeRaven/raven-helm-charts/-/tree/main/charts/backupd

[waza-ari]

Thanks! I will look into it.

FWIW, documenting some findings from yesterday when trying to make it work:

According to the ArgoCD docs, it should be possible to ignore all changes made by a specific fieldManagers:

data:
    resource.customizations.ignoreDifferences.apps_StatefulSet: |
      managedFieldsManagers:
      - stash

# Or, alternatively 
    resource.customizations.ignoreDifferences.all: |
      managedFieldsManagers:
      - stash

This however does not work at all, most likely due to an open issue in ArgoCD: argoproj/argo-cd#9071. Second attempt was to use jq path expressions like so:

data:
    resource.customizations.ignoreDifferences.apps_StatefulSet: |
      jqPathExpressions:
      - '.spec.template.metadata.annotations.["stash.appscode.com/last-applied-backup-invoker-hash"]'
      - '.spec.template.spec.volumes[] | select(.name == "stash-tmp-dir")'

This does ignore the additional volume and annotation (in this case on STS level), but yields the following Diff result:

During Sync, ArgoCD still tries to remove the respective fields, again leading to ContainerCreateErrors like this:

create Pod emqx-2 in StatefulSet emqx failed error: Pod "emqx-2" is invalid: spec.containers[1].volumeMounts[0].name: Not found: "stash-tmp-dir"

Ultimately, I could only ignore all annotations and/or volumes, but this defeats the purpose of ArgoCD altogether. I'll have a look at VolSync, thank you for the suggestion!