clients report: Already Closed

[angelAtSequent]

What happened

immudb running with the backend on S3 reports "Already Closed" so clients can not connect to the database.
Tested from multiple clients, including the webinterface:

Some commands still works:

./immuadmin -a immudb-grpc database create helloissue
database 'helloissue' {replica: false} successfully created

Others, doesn't

./immuadmin -a immudb-grpc database list 
Error: rpc error: code = Unknown desc = already closed

What you expected to happen

No error message at all

How to reproduce it (as minimally and precisely as possible)

Not sure

Environment

immudb 1.9.3
Commit  : 5487dd300655083ff68c4a72c2edb38ca84dd1bb
Built at: Thu, 23 May 2024 10:07:40 UTC
Static  : true

Additional info (any other context about the problem)
As usual, it was running fine on S3, eventually it stopped working

[ostafen]

Hey, @angelAtSequent, I tried to execute the sequence of commands using minio, and the error didn't appear.
Could you provide more context, such as the configuration used? Did the error appear when upgrading to 1.9.3?

[angelAtSequent]

This is how the stateful set looks like:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: immudb
spec:
  persistentVolumeClaimRetentionPolicy:
    whenDeleted: Retain
    whenScaled: Retain
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: immudb
      app.kubernetes.io/name: immudb
  serviceName: immudb
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: immudb
        app.kubernetes.io/name: immudb
    spec:
      containers:
      - env:
        - name: IMMUDB_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              key: adminPassword
              name: immudb
        - name: IMMUDB_S3_STORAGE
          value: "true"
        - name: IMMUDB_S3_EXTERNAL_IDENTIFIER
          value: "true"
        - name: IMMUDB_S3_ACCESS_KEY_ID
          valueFrom:
            secretKeyRef:
              key: AWS_ACCESS_KEY_ID
              name: immudb-s3-backend
        - name: IMMUDB_S3_SECRET_KEY
          valueFrom:
            secretKeyRef:
              key: AWS_SECRET_ACCESS_KEY
              name: immudb-s3-backend
        - name: IMMUDB_S3_BUCKET_NAME
          valueFrom:
            secretKeyRef:
              key: BUCKET_NAME
              name: immudb-s3-backend
        - name: IMMUDB_S3_LOCATION
          valueFrom:
            secretKeyRef:
              key: AWS_REGION
              name: immudb-s3-backend
        - name: IMMUDB_S3_PATH_PREFIX
          value: immudb
        - name: IMMUDB_S3_ENDPOINT
          valueFrom:
            secretKeyRef:
              key: AWS_S3_ENDPOINT
              name: immudb-s3-backend
        image: codenotary/immudb:1.9.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 9
          httpGet:
            path: /readyz
            port: metrics
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: immudb
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 3322
          name: grpc
          protocol: TCP
        - containerPort: 9497
          name: metrics
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: metrics
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        securityContext:
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /var/lib/immudb
          name: immudb-storage
          subPath: immudb
        - mountPath: /mnt/secrets
          name: secrets-store-inline
          readOnly: true
        - mountPath: /mnt/s3-backend-secrets
          name: s3-backend-secrets-store-inline
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 3322
        fsGroupChangePolicy: OnRootMismatch
        runAsGroup: 3322
        runAsNonRoot: true
        runAsUser: 3322
      serviceAccount: immudb
      serviceAccountName: immudb
      terminationGracePeriodSeconds: 30
      volumes:
      - emptyDir:
          sizeLimit: 5Gi
        name: immudb-storage
      - csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: immudb
        name: secrets-store-inline
      - csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: immudb-s3-backend
        name: s3-backend-secrets-store-inline
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate

I would say yes, it has been updated from 1.9DOM2 but im not a 100% sure.

[angelAtSequent]

I tried downgrading from 1.9.3 to 1.9DOM2 and it "works" again.

The downside... the resource utilization is HUGE on 1.9DOM2.
With a few databases (mostly empty) consumes 12Gb Ram and 3 cores.
Is this resource consumption expected?

Does it make sense to... while running on 1.9DOM2 dump all the databases, then spin up a fresh 1.9.3 server and restore the databases?

Thanks

[angelAtSequent]

Does it make sense to... while running on 1.9DOM2 dump all the databases, then spin up a fresh 1.9.3 server and restore the databases?

Is this resource consumption expected?

WDYT @ostafen

[ostafen]

Yes, it makes sense

[angelAtSequent]

Now it is saying...

immudb 2024/07/04 05:14:48 ERROR: unable to open database: corrupted index: index size is too large
immudb 2024/07/04 05:14:48 ERROR: database 'systemdb' was not correctly initialized.
Use replication to recover from external source or start without data folder.
immudb 2024/07/04 05:14:48 ERROR: unable to load system database: corrupted index: index size is too large
Error: corrupted index: index size is too large

Don't know how to make it working to dump the databases and then restore into a fresh immudb installation

[angelAtSequent]

We do have the same issue on 1.9.3.
Now, is not about resource consumption, is about immudb unable to load the database:

immudb 2024/07/12 07:29:46 INFO: index '/var/lib/immudb/tenant90505c8a23a94cdfa26b4e19f6a097d5event0c37052a6ad9465e8bf50c18a82d2f57/index_0243544c2e' {ts=1, discarded_snapshots=0} successfully loaded
immudb 2024/07/12 07:29:46 INFO: successfully read snapshots at '/var/lib/immudb/tenant90505c8a23a94cdfa26b4e19f6a097d5event0c37052a6ad9465e8bf50c18a82d2f57/index_0243544c2e/commit'
immudb 2024/07/12 07:29:46 ERROR: unable to load sql-engine for database 'tenant90505c8a23a94cdfa26b4e19f6a097d5event0c37052a6ad9465e8bf50c18a82d2f57' {replica = false}. corrupted index: index size is too large
immudb 2024/07/12 07:29:46 INFO: database 'tenant90505c8a23a94cdfa26b4e19f6a097d5event0c37052a6ad9465e8bf50c18a82d2f57' could not be loaded. Reason: corrupted index: index size is too large: while opening database

Could it be an issue on the S3 implementation?

[ostafen]

Is this still working fine when downgrading to 1.9DOM2?

[angelAtSequent]

Is this still working fine when downgrading to 1.9DOM2?

Nop, is not.

immudb 2024/07/12 18:23:28 INFO: loadinig database 'tenant90505c8a23a94cdfa26b4e19f6a097d5event77367cec055c42bdab505bf47d4760cc'...
immudb 2024/07/12 18:23:28 INFO: opening database 'tenant90505c8a23a94cdfa26b4e19f6a097d5event77367cec055c42bdab505bf47d4760cc' {replica = false}...
immudb 2024/07/12 18:23:28 ERROR: unable to open database: buffer is full: while loading pre-committed transaction: 1002
immudb 2024/07/12 18:23:28 INFO: database 'tenant90505c8a23a94cdfa26b4e19f6a097d5event77367cec055c42bdab505bf47d4760cc' could not be loaded. Reason: buffer is full: while loading pre-committed transaction: 1002: while opening database