Skip to content

Commit

Permalink
PG4K import for v1.next
Browse files Browse the repository at this point in the history
  • Loading branch information
josh-heyer committed Dec 4, 2024
1 parent 75c43fe commit 2b0751a
Show file tree
Hide file tree
Showing 48 changed files with 8,419 additions and 617 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,9 @@ algorithms via `barman-cloud-backup` (for backups) and
- snappy

The compression settings for backups and WALs are independent. See the
[DataBackupConfiguration](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#BarmanObjectStoreConfiguration) and
[DataBackupConfiguration](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#DataBackupConfiguration) and
[WALBackupConfiguration](https://pkg.go.dev/github.com/cloudnative-pg/barman-cloud/pkg/api#WalBackupConfiguration) sections in
the API reference.
the barman-cloud API reference.

It is important to note that archival time, restore time, and size change
between the algorithms, so the compression algorithm should be chosen according
Expand Down
162 changes: 114 additions & 48 deletions product_docs/docs/postgres_for_kubernetes/1/bootstrap.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ For more detailed information about this feature, please refer to the
EDB Postgres for Kubernetes requires both the `postgres` user and database to
always exists. Using the local Unix Domain Socket, it needs to connect
as `postgres` user to the `postgres` database via `peer` authentication in
order to perform administrative tasks on the cluster.
order to perform administrative tasks on the cluster.
**DO NOT DELETE** the `postgres` user or the `postgres` database!!!

!!! Info
Expand Down Expand Up @@ -212,36 +212,87 @@ The user that owns the database defaults to the database name instead.
The application user is not used internally by the operator, which instead
relies on the superuser to reconcile the cluster with the desired status.

### Passing options to `initdb`
### Passing Options to `initdb`

The actual PostgreSQL data directory is created via an invocation of the
`initdb` PostgreSQL command. If you need to add custom options to that command
(i.e., to change the `locale` used for the template databases or to add data
checksums), you can use the following parameters:
The PostgreSQL data directory is initialized using the
[`initdb` PostgreSQL command](https://www.postgresql.org/docs/current/app-initdb.html).

EDB Postgres for Kubernetes enables you to customize the behavior of `initdb` to modify
settings such as default locale configurations and data checksums.

!!! Warning
EDB Postgres for Kubernetes acts only as a direct proxy to `initdb` for locale-related
options, due to the ongoing and significant enhancements in PostgreSQL's locale
support. It is your responsibility to ensure that the correct options are
provided, following the PostgreSQL documentation, and to verify that the
bootstrap process completes successfully.

To include custom options in the `initdb` command, you can use the following
parameters:

builtinLocale
: When `builtinLocale` is set to a value, EDB Postgres for Kubernetes passes it to the
`--builtin-locale` option in `initdb`. This option controls the builtin locale, as
defined in ["Locale Support"](https://www.postgresql.org/docs/current/locale.html)
from the PostgreSQL documentation (default: empty). Note that this option requires
`localeProvider` to be set to `builtin`. Available from PostgreSQL 17.

dataChecksums
: When `dataChecksums` is set to `true`, CNP invokes the `-k` option in
: When `dataChecksums` is set to `true`, EDB Postgres for Kubernetes invokes the `-k` option in
`initdb` to enable checksums on data pages and help detect corruption by the
I/O system - that would otherwise be silent (default: `false`).

encoding
: When `encoding` set to a value, CNP passes it to the `--encoding` option in `initdb`,
which selects the encoding of the template database (default: `UTF8`).
: When `encoding` set to a value, EDB Postgres for Kubernetes passes it to the `--encoding`
option in `initdb`, which selects the encoding of the template database
(default: `UTF8`).

icuLocale
: When `icuLocale` is set to a value, EDB Postgres for Kubernetes passes it to the
`--icu-locale` option in `initdb`. This option controls the ICU locale, as
defined in ["Locale Support"](https://www.postgresql.org/docs/current/locale.html)
from the PostgreSQL documentation (default: empty).
Note that this option requires `localeProvider` to be set to `icu`.
Available from PostgreSQL 15.

icuRules
: When `icuRules` is set to a value, EDB Postgres for Kubernetes passes it to the
`--icu-rules` option in `initdb`. This option controls the ICU locale, as
defined in ["Locale
Support"](https://www.postgresql.org/docs/current/locale.html) from the
PostgreSQL documentation (default: empty). Note that this option requires
`localeProvider` to be set to `icu`. Available from PostgreSQL 16.

locale
: When `locale` is set to a value, EDB Postgres for Kubernetes passes it to the `--locale`
option in `initdb`. This option controls the locale, as defined in
["Locale Support"](https://www.postgresql.org/docs/current/locale.html) from
the PostgreSQL documentation. By default, the locale parameter is empty. In
this case, environment variables such as `LANG` are used to determine the
locale. Be aware that these variables can vary between container images,
potentially leading to inconsistent behavior.

localeCollate
: When `localeCollate` is set to a value, CNP passes it to the `--lc-collate`
: When `localeCollate` is set to a value, EDB Postgres for Kubernetes passes it to the `--lc-collate`
option in `initdb`. This option controls the collation order (`LC_COLLATE`
subcategory), as defined in ["Locale Support"](https://www.postgresql.org/docs/current/locale.html)
from the PostgreSQL documentation (default: `C`).

localeCType
: When `localeCType` is set to a value, CNP passes it to the `--lc-ctype` option in
: When `localeCType` is set to a value, EDB Postgres for Kubernetes passes it to the `--lc-ctype` option in
`initdb`. This option controls the collation order (`LC_CTYPE` subcategory), as
defined in ["Locale Support"](https://www.postgresql.org/docs/current/locale.html)
from the PostgreSQL documentation (default: `C`).

localeProvider
: When `localeProvider` is set to a value, EDB Postgres for Kubernetes passes it to the `--locale-provider`
option in `initdb`. This option controls the locale provider, as defined in
["Locale Support"](https://www.postgresql.org/docs/current/locale.html) from the
PostgreSQL documentation (default: empty, which means `libc` for PostgreSQL).
Available from PostgreSQL 15.

walSegmentSize
: When `walSegmentSize` is set to a value, CNP passes it to the `--wal-segsize`
: When `walSegmentSize` is set to a value, EDB Postgres for Kubernetes passes it to the `--wal-segsize`
option in `initdb` (default: not set - defined by PostgreSQL as 16 megabytes).

!!! Note
Expand Down Expand Up @@ -430,44 +481,59 @@ to the ["Recovery" section](recovery.md).

### Bootstrap from a live cluster (`pg_basebackup`)

The `pg_basebackup` bootstrap mode lets you create a new cluster (*target*) as
an exact physical copy of an existing and **binary compatible** PostgreSQL
instance (*source*), through a valid *streaming replication* connection.
The source instance can be either a primary or a standby PostgreSQL server.
The `pg_basebackup` bootstrap mode allows you to create a new cluster
(*target*) as an exact physical copy of an existing and **binary-compatible**
PostgreSQL instance (*source*) managed by EDB Postgres for Kubernetes, using a valid
*streaming replication* connection. The source instance can either be a primary
or a standby PostgreSQL server. It’s crucial to thoroughly review the
requirements section below, as the pros and cons of PostgreSQL physical
replication fully apply.

The primary use case for this method is represented by **migrations** to EDB Postgres for Kubernetes,
either from outside Kubernetes or within Kubernetes (e.g., from another operator).
The primary use cases for this method include:

!!! Warning
The current implementation creates a *snapshot* of the origin PostgreSQL
instance when the cloning process terminates and immediately starts
the created cluster. See ["Current limitations"](#current-limitations) below for details.
- Reporting and business intelligence clusters that need to be regenerated
periodically (daily, weekly)
- Test databases containing live data that require periodic regeneration
(daily, weekly, monthly) and anonymization
- Rapid spin-up of a standalone replica cluster
- Physical migrations of EDB Postgres for Kubernetes clusters to different namespaces or
Kubernetes clusters

Similar to the case of the `recovery` bootstrap method, once the clone operation
completes, the operator will take ownership of the target cluster, starting from
the first instance. This includes overriding some configuration parameters, as
required by EDB Postgres for Kubernetes, resetting the superuser password, creating
the `streaming_replica` user, managing the replicas, and so on. The resulting
cluster will be completely independent of the source instance.
!!! Important
Avoid using this method, based on physical replication, to migrate an
existing PostgreSQL cluster outside of Kubernetes into EDB Postgres for Kubernetes unless you
are completely certain that all requirements are met and the operation has been
thoroughly tested. The EDB Postgres for Kubernetes community does not endorse this approach
for such use cases and recommends using logical import instead. It is
exceedingly rare that all requirements for physical replication are met in a
way that seamlessly works with EDB Postgres for Kubernetes.

!!! Warning
In its current implementation, this method clones the source PostgreSQL
instance, thereby creating a *snapshot*. Once the cloning process has finished,
the new cluster is immediately started.
Refer to ["Current limitations"](#current-limitations) for more details.

Similar to the `recovery` bootstrap method, once the cloning operation is
complete, the operator takes full ownership of the target cluster, starting
from the first instance. This includes overriding certain configuration
parameters as required by EDB Postgres for Kubernetes, resetting the superuser password,
creating the `streaming_replica` user, managing replicas, and more. The
resulting cluster operates independently from the source instance.

!!! Important
Configuring the network between the target instance and the source instance
goes beyond the scope of EDB Postgres for Kubernetes documentation, as it depends
on the actual context and environment.
Configuring the network connection between the target and source instances
lies outside the scope of EDB Postgres for Kubernetes documentation, as it depends heavily on
the specific context and environment.

The streaming replication client on the target instance, which will be
transparently managed by `pg_basebackup`, can authenticate itself on the source
instance in any of the following ways:
The streaming replication client on the target instance, managed transparently
by `pg_basebackup`, can authenticate on the source instance using one of the
following methods:

1. via [username/password](#usernamepassword-authentication)
2. via [TLS client certificate](#tls-certificate-authentication)
1. [Username/password](#usernamepassword-authentication)
2. [TLS client certificate](#tls-certificate-authentication)

The latter is the recommended one if you connect to a source managed
by EDB Postgres for Kubernetes or configured for TLS authentication.
The first option is, however, the most common form of authentication to a
PostgreSQL server in general, and might be the easiest way if the source
instance is on a traditional environment outside Kubernetes.
Both cases are explained below.
Both authentication methods are detailed below.

#### Requirements

Expand Down Expand Up @@ -545,7 +611,7 @@ file on the source PostgreSQL instance:
host replication streaming_replica all md5
```

The following manifest creates a new PostgreSQL 17.0 cluster,
The following manifest creates a new PostgreSQL 17.2 cluster,
called `target-db`, using the `pg_basebackup` bootstrap method
to clone an external PostgreSQL cluster defined as `source-db`
(in the `externalClusters` array). As you can see, the `source-db`
Expand All @@ -560,7 +626,7 @@ metadata:
name: target-db
spec:
instances: 3
imageName: quay.io/enterprisedb/postgresql:17.0
imageName: quay.io/enterprisedb/postgresql:17.2
bootstrap:
pg_basebackup:
Expand All @@ -580,7 +646,7 @@ spec:
```

All the requirements must be met for the clone operation to work, including
the same PostgreSQL version (in our case 17.0).
the same PostgreSQL version (in our case 17.2).

#### TLS certificate authentication

Expand All @@ -595,7 +661,7 @@ in the same Kubernetes cluster.
This example can be easily adapted to cover an instance that resides
outside the Kubernetes cluster.

The manifest defines a new PostgreSQL 17.0 cluster called `cluster-clone-tls`,
The manifest defines a new PostgreSQL 17.2 cluster called `cluster-clone-tls`,
which is bootstrapped using the `pg_basebackup` method from the `cluster-example`
external cluster. The host is identified by the read/write service
in the same cluster, while the `streaming_replica` user is authenticated
Expand All @@ -610,7 +676,7 @@ metadata:
name: cluster-clone-tls
spec:
instances: 3
imageName: quay.io/enterprisedb/postgresql:17.0
imageName: quay.io/enterprisedb/postgresql:17.2
bootstrap:
pg_basebackup:
Expand Down Expand Up @@ -691,7 +757,7 @@ instance using a second connection (see the `--wal-method=stream` option for
Once the backup is completed, the new instance will be started on a new timeline
and diverge from the source.
For this reason, it is advised to stop all write operations to the source database
before migrating to the target database in Kubernetes.
before migrating to the target database.

!!! Important
Before you attempt a migration, you must test both the procedure
Expand Down
4 changes: 2 additions & 2 deletions product_docs/docs/postgres_for_kubernetes/1/certificates.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -132,14 +132,14 @@ Given the following files:

Create a secret containing the CA certificate:

```
```sh
kubectl create secret generic my-postgresql-server-ca \
--from-file=ca.crt=./server-ca.crt
```

Create a secret with the TLS certificate:

```
```sh
kubectl create secret tls my-postgresql-server \
--cert=./server.crt --key=./server.key
```
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -335,13 +335,17 @@ are the ones directly set by PgBouncer.

- [`application_name_add_host`](https://www.pgbouncer.org/config.html#application_name_add_host)
- [`autodb_idle_timeout`](https://www.pgbouncer.org/config.html#autodb_idle_timeout)
- [`cancel_wait_timeout`](https://www.pgbouncer.org/config.html#cancel_wait_timeout)
- [`client_idle_timeout`](https://www.pgbouncer.org/config.html#client_idle_timeout)
- [`client_login_timeout`](https://www.pgbouncer.org/config.html#client_login_timeout)
- [`default_pool_size`](https://www.pgbouncer.org/config.html#default_pool_size)
- [`disable_pqexec`](https://www.pgbouncer.org/config.html#disable_pqexec)
- [`dns_max_ttl`](https://www.pgbouncer.org/config.html#dns_max_ttl)
- [`dns_nxdomain_ttl`](https://www.pgbouncer.org/config.html#dns_nxdomain_ttl)
- [`idle_transaction_timeout`](https://www.pgbouncer.org/config.html#idle_transaction_timeout)
- [`ignore_startup_parameters`](https://www.pgbouncer.org/config.html#ignore_startup_parameters):
to be appended to `extra_float_digits,options` - required by CNP
to be appended to `extra_float_digits,options` - required by EDB Postgres for Kubernetes
- [`listen_backlog`](https://www.pgbouncer.org/config.html#listen_backlog)
- [`log_connections`](https://www.pgbouncer.org/config.html#log_connections)
- [`log_disconnections`](https://www.pgbouncer.org/config.html#log_disconnections)
- [`log_pooler_errors`](https://www.pgbouncer.org/config.html#log_pooler_errors)
Expand All @@ -350,13 +354,16 @@ are the ones directly set by PgBouncer.
export as described in the ["Monitoring"](#monitoring) section below
- [`max_client_conn`](https://www.pgbouncer.org/config.html#max_client_conn)
- [`max_db_connections`](https://www.pgbouncer.org/config.html#max_db_connections)
- [`max_packet_size`](https://www.pgbouncer.org/config.html#max_packet_size)
- [`max_prepared_statements`](https://www.pgbouncer.org/config.html#max_prepared_statements)
- [`max_user_connections`](https://www.pgbouncer.org/config.html#max_user_connections)
- [`min_pool_size`](https://www.pgbouncer.org/config.html#min_pool_size)
- [`pkt_buf`](https://www.pgbouncer.org/config.html#pkt_buf)
- [`query_timeout`](https://www.pgbouncer.org/config.html#query_timeout)
- [`query_wait_timeout`](https://www.pgbouncer.org/config.html#query_wait_timeout)
- [`reserve_pool_size`](https://www.pgbouncer.org/config.html#reserve_pool_size)
- [`reserve_pool_timeout`](https://www.pgbouncer.org/config.html#reserve_pool_timeout)
- [`sbuf_loopcnt`](https://www.pgbouncer.org/config.html#sbuf_loopcnt)
- [`server_check_delay`](https://www.pgbouncer.org/config.html#server_check_delay)
- [`server_check_query`](https://www.pgbouncer.org/config.html#server_check_query)
- [`server_connect_timeout`](https://www.pgbouncer.org/config.html#server_connect_timeout)
Expand All @@ -367,12 +374,18 @@ are the ones directly set by PgBouncer.
- [`server_reset_query`](https://www.pgbouncer.org/config.html#server_reset_query)
- [`server_reset_query_always`](https://www.pgbouncer.org/config.html#server_reset_query_always)
- [`server_round_robin`](https://www.pgbouncer.org/config.html#server_round_robin)
- [`server_tls_ciphers`](https://www.pgbouncer.org/config.html#server_tls_ciphers)
- [`server_tls_server_tls_protocols`](https://www.pgbouncer.org/config.html#server_tls_protocols)
- [`stats_period`](https://www.pgbouncer.org/config.html#stats_period)
- [`suspend_timeout`](https://www.pgbouncer.org/config.html#suspend_timeout)
- [`tcp_defer_accept`](https://www.pgbouncer.org/config.html#tcp_defer_accept)
- [`tcp_keepalive`](https://www.pgbouncer.org/config.html#tcp_keepalive)
- [`tcp_keepcnt`](https://www.pgbouncer.org/config.html#tcp_keepcnt)
- [`tcp_keepidle`](https://www.pgbouncer.org/config.html#tcp_keepidle)
- [`tcp_keepintvl`](https://www.pgbouncer.org/config.html#tcp_keepintvl)
- [`tcp_user_timeout`](https://www.pgbouncer.org/config.html#tcp_user_timeout)
- [`tcp_socket_buffer`](https://www.pgbouncer.org/config.html#tcp_socket_buffer)
- [`track_extra_parameters`](https://www.pgbouncer.org/config.html#track_extra_parameters)
- [`verbose`](https://www.pgbouncer.org/config.html#verbose)

Customizations of the PgBouncer configuration are written declaratively in the
Expand Down
Loading

0 comments on commit 2b0751a

Please sign in to comment.