Alpine based images create the data folder to the OS disk

[tuupola]

I have a StatefulSet With a PersistentVolume mounted to /var/lib/postgresql. Debian based Postgres images work as expected. Database is initialized and /var/lib/postgresql/data folder is stored in the PersistentVolume.

$ grep image statefulset.yaml | grep -v \#
          image: postgres:15.0

$ kubectl apply -k .
namespace/test created
statefulset.apps/postgres created
$ kubectl exec -it -n test statefulset/postgres -- bash

bash-5.1# df -h 
Filesystem                                              Size  Used Avail Use% Mounted on
overlay                                                 126G   34G   87G  28% /
tmpfs                                                    64M     0   64M   0% /dev
/dev/mapper/rl-root                                     126G   34G   87G  28% /etc/hosts
shm                                                      64M  1.1M   63M   2% /dev/shm
/dev/longhorn/pvc-7eab52f7-918e-4791-a924-74072998cd26  7.8G   28K  7.8G   1% /var/lib/postgresql
...

With Alpine based images the initialization process creates a separate mount for the /var/lib/postgresql/data folder to the OS disk. This will lead to data loss when the pod is recycled (data is not in the PersistenVolume).

$ grep image statefulset.yaml | grep -v \#
          image: postgres:15.0-alpine

$ kubectl apply -k .
namespace/test created
statefulset.apps/postgres created

$ kubectl exec -it -n test statefulset/postgres -- bash
bash-5.1# df -h
Filesystem                Size      Used Available Use% Mounted on
overlay                 125.4G     33.0G     86.7G  28% /
tmpfs                    64.0M         0     64.0M   0% /dev
/dev/mapper/rl-root     125.4G     33.0G     86.7G  28% /etc/hosts
/dev/mapper/rl-root     125.4G     33.0G     86.7G  28% /dev/termination-log
/dev/mapper/rl-root     125.4G     33.0G     86.7G  28% /etc/hostname
/dev/mapper/rl-root     125.4G     33.0G     86.7G  28% /etc/resolv.conf
shm                      64.0M      1.0M     63.0M   2% /dev/shm
/dev/longhorn/pvc-7a73320e-4498-403a-a5ec-33d3c4f8ae76
                          7.8G     28.0K      7.8G   0% /var/lib/postgresql
/dev/mapper/rl-root     125.4G     33.0G     86.7G  28% /var/lib/postgresql/data
...

Not sure if this is a configuration problem in the StatefulSet itself. Atleast it is suprising behaviour which is hard to detect at first. It took me couple of times losing the data before suspecting something is wrong. It is not Longhorn related, I have seen the problem happen with AWS based storage too.

To reproduce checkout the following repository.
https://github.com/tuupola/postgres-alpine/

For documentation purposes this is the StateFulset config.

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  replicas: 1
  serviceName: postgres
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
      app.kubernetes.io/component: database
      app.kubernetes.io/instance: test-postgres
  template:
    metadata:
      name: postgres
      # These are not needed by considered a good practice
      labels:
        app.kubernetes.io/name: postgres
        app.kubernetes.io/component: database
        app.kubernetes.io/instance: test-postgres
        app.kubernetes.io/version: "15.0"
    spec:
      containers:
        - name: postgres
          #image: postgres:16.0
          #image: postgres:16.0-alpine
          #image: postgres:15.0
          image: postgres:15.0-alpine
          #image: postgres:14.0
          #image: postgres:14.0-alpine
          env:
            # These are created automagically
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: example
            - name: POSTGRES_DB
              value: test
          resources:
            requests:
              cpu: 250m
              memory: 256Mi
            limits:
              cpu: 1000m
              memory: 1024Mi
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 8Gi
        storageClassName: longhorn

[tianon]

I think this:

          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql

Needs to be:

          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data

(The mount you're seeing at /var/lib/postgresql/data is likely an anonymous volume created by your underlying container runtime, which I'm guessing is most likely Docker, since I don't think any other runtime will auto-create anonymous volumes from the image config like that.)

[tuupola]

It cannot be /var/lib/postgresql/data since initdb will complain about lost+found folder when initialising and suggests using /var/lib/postgresql as the mountpoint.

initdb: error: directory "/var/lib/postgresql/data" exists but is not empty
initdb: detail: It contains a lost+found directory, perhaps due to it being a mount point.
initdb: hint: Using a mount point directly as the data directory is not recommended.
Create a subdirectory under the mount point

It is not Docker runtime related either. I tested and happens atleast with docker://19.3.1 and containerd://1.7.1-k3s1.

My point is, if you use Debian based image and mount the persistent volume to Postgres default /var/lib/postgresql everything is ok. Then again if using the same defaults with the Alpine image it will install and start up happily but you will end up losing your data when the pod is restarted or recycled. Ie if you do something like

$ kubectl rollout restart -n test statefulset/postgres

or

$ kubectl delete -n test pod/postgres-0     

your data is gone.

I was going to say if this is not considered a bug, atleast a warning would be nice. Reading the docs again there already is a note under PGDATA.

Important Note: when mounting a volume to /var/lib/postgresql, the /var/lib/postgresql/data path is a local volume from the container runtime, thus data is not persisted on the mounted volume.

It seems it all boils down to this issue: #404

[tuupola]

Did some more testing and it seems data is also lost with Debian based images. For some reason df does not show the /var/lib/postgresql/data mount, but you can see it with mount command.

# df -h | grep postgres
/dev/longhorn/pvc-b85dfe4e-5d64-486a-9d58-a06ca8ee1972  7.8G   28K  7.8G   1% /var/lib/postgresql

# mount | grep postgres
/dev/longhorn/pvc-b85dfe4e-5d64-486a-9d58-a06ca8ee1972 on /var/lib/postgresql type ext4 (rw,relatime,seclabel)
/dev/mapper/rl-root on /var/lib/postgresql/data type ext4 (rw,relatime,seclabel)

I will use some other folder for Postgres data.