Facing permission denied

[cloudcafetech]

First of all, Thanks for nice project.

My vault running with backend consul with HA and I expose Vault using ingress. export VAULT_ADDR=http://vault-internal.172.31.14.138.nip.io

• Error

2021-07-30 02:35:37.711  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Refresh of secret internal-new-config in namespace kube-vault failed with exception

de.koudingspawn.vault.vault.communication.SecretNotAccessibleException: Couldn't load secret from vault path internal/new/config
        at de.koudingspawn.vault.vault.VaultCommunication.getVersionedSecret(VaultCommunication.java:136) ~[classes!/:0.0.1-SNAPSHOT]
        at de.koudingspawn.vault.vault.VaultCommunication.getVersionedSecret(VaultCommunication.java:111) ~[classes!/:0.0.1-SNAPSHOT]
        at de.koudingspawn.vault.vault.impl.KeyValueV2Generator.getHash(KeyValueV2Generator.java:34) ~[classes!/:0.0.1-SNAPSHOT]
        at de.koudingspawn.vault.kubernetes.scheduler.impl.KeyValueV2Refresh.hashHasChanged(KeyValueV2Refresh.java:33) ~[classes!/:0.0.1-SNAPSHOT]
        at de.koudingspawn.vault.kubernetes.scheduler.impl.KeyValueV2Refresh.refreshIsNeeded(KeyValueV2Refresh.java:28) ~[classes!/:0.0.1-SNAPSHOT]
        at de.koudingspawn.vault.kubernetes.scheduler.ScheduledRefresh.refreshCertificates(ScheduledRefresh.java:43) ~[classes!/:0.0.1-SNAPSHOT]
        at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_275]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_275]
        at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) [spring-context-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_275]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_275]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_275]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_275]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_275]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_275]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_275]
Caused by: org.springframework.vault.VaultException: Status 403 Forbidden [new/config]: 1 error occurred:
        * permission denied

; nested exception is org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: [{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
]
        at org.springframework.vault.client.VaultResponses.buildException(VaultResponses.java:86) ~[spring-vault-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultVersionedKeyValueTemplate.lambda$doRead$0(VaultVersionedKeyValueTemplate.java:118) ~[spring-vault-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultTemplate.doWithSession(VaultTemplate.java:466) ~[spring-vault-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultVersionedKeyValueTemplate.doRead(VaultVersionedKeyValueTemplate.java:100) ~[spring-vault-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at org.springframework.vault.core.VaultVersionedKeyValueTemplate.get(VaultVersionedKeyValueTemplate.java:89) ~[spring-vault-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        at de.koudingspawn.vault.vault.VaultCommunication.getVersionedSecret(VaultCommunication.java:123) ~[classes!/:0.0.1-SNAPSHOT]
        ... 17 common frames omitted
Caused by: org.springframework.web.client.HttpClientErrorException$Forbidden: 403 Forbidden: [{"errors":["1 error occurred:\n\t* permission denied\n\n"]}
]
        at org.springframework.web.client.HttpClientErrorException.create(HttpClientErrorException.java:109) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:170) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:112) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.client.ResponseErrorHandler.handleError(ResponseErrorHandler.java:63) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:785) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:743) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:677) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:586) ~[spring-web-5.2.3.RELEASE.jar!/:5.2.3.RELEASE]
        at org.springframework.vault.core.VaultVersionedKeyValueTemplate.lambda$doRead$0(VaultVersionedKeyValueTemplate.java:103) ~[spring-vault-core-2.2.1.RELEASE.jar!/:2.2.1.RELEASE]
        ... 21 common frames omitted

2021-07-30 02:35:37.713  INFO 1 --- [TaskScheduler-1] d.k.v.k.scheduler.ScheduledRefresh       : Finished refresh of secret...

Steps I followed

• Created Policy

cat <<EOF > policy.hcl
path "testpki/issue/testrole" {
  capabilities = ["create", "read", "update"]
}

path "secret/*" {
  capabilities = ["read"]
}
EOF
vault write sys/policy/testpolicy [email protected]
vault token create -policy=testpolicy -display-name=testtoken

• RBAC

apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-crd-serviceaccount
  namespace: kube-vault
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: vault-crd-clusterrole
rules:
  - apiGroups:
    - apiextensions.k8s.io
    resources:
    - customresourcedefinitions
    verbs:
    - get
  - apiGroups:
      - "koudingspawn.de"
    resources:
      - vault
    verbs:
      - list
      - watch
      - get
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - create
      - patch
      - update
      - delete
  - apiGroups:
      - extensions
      - apps
    resources:
      - deployments
    verbs:
      - update
      - get
      - patch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-crd-clusterrole-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: vault-crd-clusterrole
subjects:
  - kind: ServiceAccount
    name: vault-crd-serviceaccount
    namespace: kube-vault
---
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: vault.koudingspawn.de
spec:
  group: koudingspawn.de
  scope: Namespaced
  names:
    plural: vault
    singular: vault
    kind: Vault
    shortNames:
      - vt
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                path:
                  type: string
                  pattern: '^.*?\/.*?(\/.*?)?$'
                type:
                  type: string
                  enum:
                    - PKI
                    - PKIJKS
                    - CERT
                    - CERTJKS
                    - DOCKERCFG
                    - KEYVALUE
                    - KEYVALUEV2
                    - PROPERTIES
                pkiConfiguration:
                  type: object
                  properties:
                    commonName:
                      type: string
                    altNames:
                      type: string
                    ipSans:
                      type: string
                    ttl:
                      type: string
                      pattern: '^[0-9]{1,}[hm]$'
                jksConfiguration:
                  type: object
                  properties:
                    password:
                      type: string
                    alias:
                      type: string
                    keyName:
                      type: string
                    caAlias:
                      type: string
                versionConfiguration:
                  type: object
                  properties:
                    version:
                      type: integer
                propertiesConfiguration:
                  type: object
                  properties:
                    context:
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                    files:
                      type: object
                      x-kubernetes-preserve-unknown-fields: true
                dockerCfgConfiguration:
                  type: object
                  properties:
                    type:
                      type: string
                      enum:
                        - KEYVALUE
                        - KEYVALUEV2
                    version:
                      type: integer
                changeAdjustmentCallback:
                  type: object
                  properties:
                    type:
                      type: string
                    name:
                      type: string
              required:
                - type
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: vault-crd
  name: vault-crd
  namespace: kube-vault
spec:
  selector:
    matchLabels:
      app: vault-crd
  replicas: 1
  template:
    metadata:
      labels:
        app: vault-crd
    spec:
      serviceAccountName: vault-crd-serviceaccount
      containers:
      - name: vault-crd
        image: daspawnw/vault-crd:1.6.4
        env:
        - name: KUBERNETES_VAULT_URL
          value: "http://vault-internal.172.31.14.138.nip.io/v1/"
        - name: KUBERNETES_VAULT_TOKEN
          value: "s.sOARfxb4nDTsUCKmAVc379ru"
        ports:
          - containerPort: 8080
        livenessProbe:
          httpGet:
            port: 8080
            path: "/actuator/health"
          initialDelaySeconds: 30
          failureThreshold: 3
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
      restartPolicy: Always
---

• Create secrets

vault kv put internal/new/config username="db-readonly-username" 
cat <<EOF > vault-crd-secrets-new.yaml
apiVersion: "koudingspawn.de/v1"
kind: Vault
metadata:
  name: internal-new-config
spec:
  path: "internal/new/config"
  type: "KEYVALUEV2"
  versionConfiguration:
    version: 1
EOF
kubectl create -f vault-crd-secrets-new.yaml

Please let me know if I missing!!

[DaspawnW]

Hi @cloudcafetech,

Maybe its simply a copy paste issue, but in your permissions you grant access to secret path secret:

path "secret/*" {
  capabilities = ["read"]
}

But it should be:

path "internal/*" {
  capabilities = ["read"]
}

[cloudcafetech]

Thanks, Will check & update.

Just one more question if I have multiple path (one for internal and other for external) for two Kubernetes.

Will it be as follows ... ?

path "internal/*" {
  capabilities = ["read"]
}path "external/*" {
  capabilities = ["read"]
}

[cloudcafetech]

It's working, but update not happening.

Holding old value also even I delete & recreate vault (kind) :(