CA deployment 'forgets' Idemix keys when restarting

[arner]

This impacts anyone who uses the CA as Idemix credential issuer.

Refers to the Deployment of the CA: https://github.com/hyperledger-labs/fabric-operator/blob/main/definitions/ca/deployment.yaml

Issue
The FABRIC_CA_HOME directory (/data/ca) is not on a Persistent Volume (I've seen an implementation where it's explicitly set to the shared in Memory storage - is that the default?).

The normal and TLS keys of the CA are mounted from secrets, and fabric-server-config.yaml points to them, so that's all persistent. The CA, when starting, also checks if there are Idemix keys in the FABRIC_CA_HOME directory. It's about the following files:

• /data/ca/IssuerPublicKey
• /data/ca/IssuerRevocationPublicKey
• /data/ca/msp/keystore/IssuerRevocationPrivateKey
• /data/ca/msp/keystore/IssuerSecretKey

When the Pod is restarted, the previous keys are gone. The CA generates new ones.

Steps to reproduce
(don't do this if you rely on the Idemix keys, first back them up)

• Check the IssuerPublicKey in /api/v1/cainfo
• Restart the CA pod
• Verify that the IssuerPublicKey has changed in /api/v1/cainfo

Hotfix
I patched this issue on a live CA in the following way:

• kubectl cp all the keys to a local folder
• kubectl create secret from the keys
• patch the Deployment of the CA:
  • add a Volume for the secret
  • mount the volume in the InitContainer
  • change the command of the InitContainer to add mkdir -p /data/ca/msp/keystore && cp /tmp/IssuerPublicKey /data/ca/ && cp /tmp/IssuerRevocationPublicKey /data/ca/ && cp /tmp/IssuerRevocationPrivateKey /data/ca/msp/keystore && cp /tmp/IssuerSecretKey /data/ca/msp/keystore/

Now, when it restarts, it will copy the keys from a secret. It's maybe not as pretty as how the other keys are managed, but as far as I'm aware you can't configure the locations of the Idemix keys so they'd have to be in FABRIC_CA_HOME.

Potential solution
I don't know how the /crypto secrets are created, but we should probably do the same for the Idemix keys. And then do something similar as I did in my hotfix.