Adding postgresql-ha on obs cluster

[computate]

• created all the postgresql-ha config resources for the ai-telemetry project
• creation of external secrets
• creation of namespace

[larsks]

Elsewhere we have been using the Crunchy Data postgres operator (https://github.com/CrunchyData/postgres-operator) to manage HA postgres deployments. I'm curious if you looked at that first (or at any of the other postgres operators)? I'm a little uncomfortable with the big chunk of logic we have to provide in the postgresql-ha-postgresql-hooks-scripts configmap.

[computate]

Thanks for the feedback @larsks , My goal with postgresql-ha was to achieve multiple primary write pods, but I see it's not the case, at least with this configuration. The current configuration provides multiple standby read pods, which Crunchy Postgresql also provides, so there doesn't seem to be a reason to use postgresql-ha when we already have Crunchy Postgresql.

• For example, I can create a database, table, and insert a record on pod/postgresql-ha-postgresql-0.

$ oc --as system:admin -n postgresql-ha exec pod/postgresql-ha-postgresql-0 -- bash -c 'env PGPASSWORD=$POSTGRES_PASSWORD psql -U $POSTGRES_USER postgres -c "create database test0;"'
CREATE DATABASE

$ oc --as system:admin -n postgresql-ha exec pod/postgresql-ha-postgresql-0 -- bash -c 'env PGPASSWORD=$POSTGRES_PASSWORD psql -U $POSTGRES_USER test0 -c "create table test0(pk bigserial primary key, k text, v text);"'
CREATE TABLE

$ oc --as system:admin -n postgresql-ha exec pod/postgresql-ha-postgresql-0 -- bash -c 'env PGPASSWORD=$POSTGRES_PASSWORD psql -U $POSTGRES_USER test0 -c "insert into test0(k, v) values('"'"'key0'"'"', '"'"'value0'"'"');"'
INSERT 0 1

$ oc --as system:admin -n postgresql-ha exec pod/postgresql-ha-postgresql-0 -- bash -c 'env PGPASSWORD=$POSTGRES_PASSWORD psql -U $POSTGRES_USER test0 -c "select * from test0;"'
 pk |  k   |   v    
----+------+--------
  1 | key0 | value0

• But when I try to create a record on pod/postgresql-ha-postgresql-1 or pod/postgresql-ha-postgresql-2, the pod is read-only.

$ oc --as system:admin -n postgresql-ha exec pod/postgresql-ha-postgresql-1 -- bash -c 'env PGPASSWORD=$POSTGRES_PASSWORD psql -U $POSTGRES_USER test0 -c "insert into test0(k, v) values('"'"'key1'"'"', '"'"'value1'"'"');"'
ERROR:  cannot execute INSERT in a read-only transaction
command terminated with exit code 1
$ oc --as system:admin -n postgresql-ha exec pod/postgresql-ha-postgresql-2 -- bash -c 'env PGPASSWORD=$POSTGRES_PASSWORD psql -U $POSTGRES_USER test0 -c "insert into test0(k, v) values('"'"'key2'"'"', '"'"'value2'"'"');"'
ERROR:  cannot execute INSERT in a read-only transaction
command terminated with exit code 1

(1 row)

The only difference I see is that Crunchy Postgres forces you to have a large enough backup storage for your all your databases, so you have to have double the storage with Crunchy, whereas postgresql-ha does not force a database backup solution on you.
We can update this PR to use Crunchy Postgres instead.

[computate]

Closing this, we will use Crunchy Postgres instead.

[joachimweyl]

@computate do we have a PR for Crunchy Postgres?

[computate]

@joachimweyl not yet