'/var/lib/arangodb3-apps/_db': Permission denied

[pbirkle]

Hello everyone,

I'm facing an issue while setting up an arangodb in cluster mode in kubernetes. In our kubernetes environment it is not permitted to run containers as root, therefore we added securityContext in our ArangoDeployment file for id, agents, dbservers and coordinators. With this configuration the pods can be deployed, but fail during startup. e.g.:

apiVersion: "database.arangodb.com/v1"
kind: "ArangoDeployment"
metadata:
  name: "arangodb-cluster"
spec:
  mode: Cluster
  environment: Production
  image: arangodb/arangodb:3.7.3

id:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 1000

...

At first we installed the two helm charts kube-arangodb (v1.1.2) and kube-arangodb-crd (v1.1.2) before applying the ArangoDeployment. As soon as the arangodb-cluster-id-XXXXX pod is spawned, this pod directly switches to state 'Error' with following log output:

 2020-12-11T08:11:41Z [1] INFO [e52b0] ArangoDB 3.7.3 [linux] 64bit, using jemalloc, build tags/v3.7.3-0-g8ad8edb71f, VPack 0.1.33, RocksDB 6.8.0, ICU 64.2, V8 7.9.317, OpenSSL 1.1.1g  21 Apr 2020
 2020-12-11T08:11:41Z [1] INFO [75ddc] detected operating system: Linux version 5.4.0-52-generic (buildd@lgw01-amd64-060) (gcc version 9.3.0 (Ubuntu 9.3.0-17ubuntu1~20.04)) #57-Ubuntu SMP Thu Oct 15 10:57:00 UTC 2020
 2020-12-11T08:11:41Z [1] INFO [25362] {memory} Available physical memory: 19145113600 bytes, available cores: 2
 2020-12-11T08:11:41Z [1] INFO [43396] {authentication} Jwt secret not specified, generating...
 2020-12-11T08:11:41Z [1] INFO [144fe] using storage engine 'rocksdb'
 2020-12-11T08:11:41Z [1] INFO [3bb7d] {cluster} Starting up with role SINGLE
 2020-12-11T08:11:41Z [1] INFO [6ea38] using endpoint 'http+tcp://[::]:8529' for non-encrypted requests
 2020-12-11T08:11:41Z [1] INFO [a1c60] {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
 2020-12-11T08:11:41Z [1] INFO [3844e] {authentication} Authentication is turned off, authentication for unix sockets is turned on
 2020-12-11T08:11:41Z [1] ERROR [5a0b4] unable to create base application directory failed to create directory '/var/lib/arangodb3-apps/_db': Permission denied
 2020-12-11T08:11:41Z [1] ERROR [610c7] unable to initialize databases: system error
 2020-12-11T08:11:41Z [1] ERROR [4ec19] {startup} startup aborted: caught exception during start of feature 'Database': system error. shutting down
 2020-12-11T08:11:41Z [1] INFO [4bcb9] ArangoDB has been shut down
 2020-12-11T08:11:41Z [1] ERROR [5d508] arangod terminated because of an exception: startup aborted: caught exception during start of feature 'Database': system error

Looks like the permissions are not set properly while setting up the database. I already found this issue, but unfortunately we are not permitted to change the capabilities to DAC_OVERRIDE either.

^{Patrick Birkle [email protected], Daimler TSS GmbH, Imprint}

[ajanikow]

Hello!

We had same issue on OpenShift.

Can you try set this:

kube-arangodb/pkg/apis/deployment/v1/server_group_spec.go

Line 127 in 7c16bb2

FSGroup *int64 `json:"fsGroup,omitempty"`

Doc: https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

Best Regards,
Adam.

[pbirkle]

Unfortunately this does not work either. When looking into pod specs after it was spawned I recognized that fsGroup is set on the level of spec.securityContext (PodSecurityContext) and not on spec.container.securityContext and therefore still is fsGroup: 1. The value for the spec.container.securityContext was not set. Is this as intended?

Current settings ArangoDeployment:

spec:
...
  id:
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      runAsGroup: 1000
      fsGroup: 1000
...

Pod output (kubectl get pod arangodb-cluster-id-XXXXXX -o yaml):

...
spec:
  containers:
  - command:
    - /usr/sbin/arangod
    - --server.authentication=false
    - --server.endpoint=tcp://[::]:8529
    - --database.directory=/data
    - --log.output=+
    image: arangodb/arangodb:3.7.3
    imagePullPolicy: IfNotPresent
    name: server
    ports:
    - containerPort: 8529
      name: server
      protocol: TCP
    resources: {}
    securityContext:
      capabilities:
        drop:
        - ALL
      runAsGroup: 1000
      runAsNonRoot: true
      runAsUser: 1000
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /data
      name: arangod-data
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-8hk6x
      readOnly: true
  securityContext:
    fsGroup: 1
    supplementalGroups:
    - 1
...

We also tried using the arangodb/arangodb:3.7.5 image, which was released yesterday, but got the same result

[ajanikow]

Hello!

FSGroup is placed under pod because it is pod setting (PodSecurityContext) and not visible in container settings (SecurityContext).

FSGroup should be set in best case with same ID as your User (and your Storage needs to support mounting with permission override).

Otherwise, you can use initContainer to check permissions (for example ls -al /data) and current user (id) to ensure that all is set properly. If it is possible, you can use initContainer to adjust privileges if needed (if you can run initContainer as root, but application need to run as user)

Best,
Adam.

[pbirkle]

Hi Adam,

after further research we aren't really sure if the fsGroup will be set for the id pod. In our answer above you can see that we configured our id.securityContext.fsGroup with 1000, but the deployed pod had an fsGroup: 1. This could be set automatically by our cluster, as our podSecurityPolicy requires an fsGroup between 1 and 65535.

We tested the same configuration on our local environment (minikube) and the fsGroup for the id pod isn't set. The fsGroup for any other pods (like agents, servers, ...) are set as expected. Could you please verify if you can reproduce the same behavior with spec.id.securityContext.fsGroup?

We have seen that the ApplyPodSpec function in images.go returnes nil.

kube-arangodb/pkg/deployment/images.go

Line 402 in d232d3d

func (i *ImageUpdatePod) ApplyPodSpec(spec *core.PodSpec) error {

The pod_creator_arangod.go returns the podSecurityContext instead.

kube-arangodb/pkg/deployment/resources/pod_creator_arangod.go

Line 501 in d232d3d

func (m *MemberArangoDPod) ApplyPodSpec(p *core.PodSpec) error {

As the podSecurityContext defines the fsGroup we think this could be the reason why there is no fsGroup applied for the id pod. Unfortunately our Go knowledge is very limited, so this could be a wrong guess from our side.

Thanks,
Patrick

[ajanikow]

Hello!

For ID pod - it is not using any volumes, it only start for version discovery.

Did you solve this issue? If yes, can I close it?

[vbasem]

Hello,
I face a similar issue here as well. I am unable to change the database instance user to a different user and it fails my kubernetes cluster with a strict non-root-only policy.
It would help alot if the database itself had runAsUser setting and support similar to how the operator itself does it.

[pbirkle]

Hi Adam,
unfortunately we were never able to fix the issue. At the end we created our own ArangoDB Helm-Chart which we are using now.

[ognjen-it]

Hi @pbirkle ,

Could you please share with us the helm chart?

[ajanikow]

Fixed by feature Operator Ephemeral Volumes