From 031693d36c3736396f29894abd9cc29f3cf32195 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Wed, 21 Aug 2024 14:01:21 -0400 Subject: [PATCH 01/10] Edits to Migration Portal: added unsupported objects to known issues PR5943 --- .../02_mp_schema_assessment.mdx | 2 +- .../migration_portal/4/known_issues_notes.mdx | 18 +++++++++--------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx b/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx index 7ae13411f9f..a980b61f15e 100644 --- a/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx +++ b/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx @@ -12,7 +12,7 @@ legacyRedirectsGenerated: You can assess an Oracle database schema for compatibility with EDB Postgres Advanced Server. You can also generate a report about the assessment. !!!note - Review the [release notes](../01_mp_release_notes/) and [known issues](../known_issues_notes.mdx) section for an overview of unsupported schema objects. + Review the [release notes](../01_mp_release_notes/) and [known issues](../known_issues_notes.mdx) for an overview of unsupported schema objects. !!! ## Perform a schema assessment diff --git a/product_docs/docs/migration_portal/4/known_issues_notes.mdx b/product_docs/docs/migration_portal/4/known_issues_notes.mdx index ac66ce8072e..7fd4f6ab724 100644 --- a/product_docs/docs/migration_portal/4/known_issues_notes.mdx +++ b/product_docs/docs/migration_portal/4/known_issues_notes.mdx @@ -23,15 +23,15 @@ Some unsupported Oracle objects are removed when Migration Portal assesses your The following objects are removed during the schema assessment: -- Objects related to `MATERIALIZED VIEWS`. For example, backend `TABLE` or `INDEX` statements created to support `MVIEW`. -- Objects related to `Queues`. -- Objects related to `Nested Tables`. -- Objects related to `XMLType Tables`. -- Types that depend on `SYSTEM Schemas`. -- Indexes related to `PRIMARY KEY` and `UNIQUE` constraints. -- Unsupported system `GRANT` privileges: - - EDB only supports granting `CREATE DATABASE LINK`, `CREATE PUBLIC DATABASE LINK`, `DROP PUBLIC DATABASE LINK` and `EXEMPT ACCESS POLICY` system privileges. Other `GRANT` statements are not supported and removed from the DDL file. +- Objects related to `MATERIALIZED VIEWS`, for example, backend `TABLE` or `INDEX` statements created to support `MVIEW` +- Objects related to `Queues` +- Objects related to `Nested Tables` +- Objects related to `XMLType Tables` +- Types that depend on `SYSTEM Schemas` +- Indexes related to `PRIMARY KEY` and `UNIQUE` constraints +- Unsupported system `GRANT` privileges + + EDB supports granting only `CREATE DATABASE LINK`, `CREATE PUBLIC DATABASE LINK`, `DROP PUBLIC DATABASE LINK`, and `EXEMPT ACCESS POLICY` system privileges. Other `GRANT` statements aren't supported and are removed from the DDL file. ### File encoding From 4b213864a7015e1df043ce8cdf6dda56cbcc8997 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Wed, 21 Aug 2024 14:33:58 -0400 Subject: [PATCH 02/10] Removed screenshot of dialog box per doc standards --- .../4/04_mp_migrating_database/02_mp_schema_assessment.mdx | 2 -- 1 file changed, 2 deletions(-) diff --git a/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx b/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx index a980b61f15e..978380fb4e7 100644 --- a/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx +++ b/product_docs/docs/migration_portal/4/04_mp_migrating_database/02_mp_schema_assessment.mdx @@ -23,8 +23,6 @@ You can assess an Oracle database schema for compatibility with EDB Postgres Adv 1. Select **New** to create a project. - ![The Migration Portal New project dialog](../images/mp_schema_assessment_new_project_updated.png) - 1. In the dialog box, enter the project name in the **Project name** field. 1. Specify project details: From 2fb3bd69d8e0272e6fcbbff61dfa98be0348c66a Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 19 Aug 2024 11:25:05 +0100 Subject: [PATCH 03/10] First import 19 aug Signed-off-by: Dj Walker-Morgan --- product_docs/docs/tpa/23/INSTALL.mdx | 6 - .../tpa/23/architecture-BDR-Always-ON.mdx | 6 +- product_docs/docs/tpa/23/architecture-M1.mdx | 16 +-- .../tpa/23/architecture-PGD-Always-ON.mdx | 10 +- .../docs/tpa/23/configure-cluster.mdx | 8 +- .../docs/tpa/23/configure-instance.mdx | 8 +- .../23/reference/2q_and_edb_repositories.mdx | 129 ------------------ .../tpa/23/reference/edb_repositories.mdx | 31 +++-- product_docs/docs/tpa/23/reference/efm.mdx | 11 +- .../docs/tpa/23/reference/patroni.mdx | 2 +- product_docs/docs/tpa/23/reference/pem.mdx | 62 ++++++++- .../{pg_hba_conf.mdx => pg_hba.conf.mdx} | 3 +- .../{pg_ident_conf.mdx => pg_ident.conf.mdx} | 3 +- ...ostgresql_conf.mdx => postgresql.conf.mdx} | 2 - .../reference/reconciling-local-changes.mdx | 1 - .../tpa/23/reference/tpa_2q_repositories.mdx | 43 ------ .../reference/tpaexec-download-packages.mdx | 2 +- .../docs/tpa/23/reference/volumes.mdx | 53 ++++--- product_docs/docs/tpa/23/task-selection.mdx | 4 + .../docs/tpa/23/tpaexec-configure.mdx | 105 ++++++-------- product_docs/docs/tpa/23/tpaexec-deploy.mdx | 5 +- product_docs/docs/tpa/23/tpaexec-hooks.mdx | 8 ++ .../docs/tpa/23/tpaexec-provision.mdx | 87 ++++++------ 23 files changed, 225 insertions(+), 380 deletions(-) delete mode 100644 product_docs/docs/tpa/23/reference/2q_and_edb_repositories.mdx rename product_docs/docs/tpa/23/reference/{pg_hba_conf.mdx => pg_hba.conf.mdx} (98%) rename product_docs/docs/tpa/23/reference/{pg_ident_conf.mdx => pg_ident.conf.mdx} (94%) rename product_docs/docs/tpa/23/reference/{postgresql_conf.mdx => postgresql.conf.mdx} (99%) delete mode 100644 product_docs/docs/tpa/23/reference/tpa_2q_repositories.mdx diff --git a/product_docs/docs/tpa/23/INSTALL.mdx b/product_docs/docs/tpa/23/INSTALL.mdx index 64389690648..d40a5edc755 100644 --- a/product_docs/docs/tpa/23/INSTALL.mdx +++ b/product_docs/docs/tpa/23/INSTALL.mdx @@ -97,12 +97,6 @@ curl -1sLf 'https://downloads.enterprisedb.com///setup.de curl -1sLf 'https://downloads.enterprisedb.com///setup.rpm.sh' | sudo -E bash ``` -Alternatively, you may obtain TPA from the legacy 2ndQuadrant -repository. To do so, login to the EDB Customer Support Portal and -subscribe to the ["products/tpa/release" repository](https://techsupport.enterprisedb.com/software_subscriptions/add/products/tpa/) -by adding a subscription under Support/Software/Subscriptions, -and following the instructions to enable the repository on your system. - Once you have enabled one of these repositories, you may install TPA as follows: diff --git a/product_docs/docs/tpa/23/architecture-BDR-Always-ON.mdx b/product_docs/docs/tpa/23/architecture-BDR-Always-ON.mdx index c76e5af6342..09c83b93b77 100644 --- a/product_docs/docs/tpa/23/architecture-BDR-Always-ON.mdx +++ b/product_docs/docs/tpa/23/architecture-BDR-Always-ON.mdx @@ -8,10 +8,8 @@ originalFilePath: architecture-BDR-Always-ON.md EDB Postgres Distributed 3.7 or 4 in an Always-ON configuration, suitable for use in test and production. -This architecture requires a subscription to the legacy 2ndQuadrant -repositories, and some options require a subscription to EDB Repos 1.0. -See [How TPA uses 2ndQuadrant and EDB repositories](reference/2q_and_edb_repositories/) -for more detail on this topic. +This architecture requires an EDB subscription. +All software will be sourced from [EDB Repos 2.0](reference/edb_repositories/). The BDR-Always-ON architecture has four variants, which can be selected with the `--layout` configure option: diff --git a/product_docs/docs/tpa/23/architecture-M1.mdx b/product_docs/docs/tpa/23/architecture-M1.mdx index d4038e65426..64718467585 100644 --- a/product_docs/docs/tpa/23/architecture-M1.mdx +++ b/product_docs/docs/tpa/23/architecture-M1.mdx @@ -16,9 +16,8 @@ testing, demonstrating and learning due to its simplicity and ability to be configured with no proprietary components. If you select subscription-only EDB software with this architecture -it will be sourced from EDB Repos 2.0 and you will need to provide a token. -See [How TPA uses 2ndQuadrant and EDB repositories](reference/2q_and_edb_repositories/) -for more detail on this topic. +it will be sourced from EDB Repos 2.0 and you will need to +[provide a token](reference/edb_repositories/). ## Application and backup failover @@ -69,11 +68,11 @@ More detail on the options is provided in the following section. #### Mandatory Options -| Parameter | Description | -| ---------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | -| `--architecture` (`-a`) | Must be set to `M1`. | -| Postgres flavour and version (e.g. `--postgresql 15`) | A valid [flavour and version specifier](tpaexec-configure/#postgres-flavour-and-version). | -| One of: * `--failover-manager {efm, repmgr, patroni}` * `--enable-efm` * `--enable-repmgr` * `--enable-patroni` | Select the failover manager from [`efm`](reference/efm/), [`repmgr`](reference/repmgr/) and [`patroni`](reference/patroni/). | +| Parameter | Description | +| -------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- | +| `--architecture` (`-a`) | Must be set to `M1`. | +| Postgres flavour and version (e.g. `--postgresql 15`) | A valid [flavour and version specifier](tpaexec-configure/#postgres-flavour-and-version). | +| One of: - `--failover-manager {efm, repmgr, patroni}`- `--enable-efm` - `--enable-repmgr`- `--enable-patroni` | Select the failover manager from [`efm`](reference/efm/), [`repmgr`](reference/repmgr/) and [`patroni`](reference/patroni/). |

@@ -90,6 +89,7 @@ More detail on the options is provided in the following section. | `--enable-haproxy` | 2 additional nodes will be added as a load balancer layer.
Only supported with Patroni as the failover manager. | HAproxy nodes will not be added to the cluster. | | `--enable-pgbouncer` | PgBouncer will be configured in the Postgres nodes to pool connections for the primary. | PgBouncer will not be configured in the cluster. | | `--patroni-dcs` | Select the Distributed Configuration Store backend for patroni.
Only option is `etcd` at this time.
Only supported with Patroni as the failover manager. | Defaults to `etcd`. | +| `--efm-bind-by-hostname` | Enable efm to use hostnames instead of IP addresses to configure the cluster `bind.address`. | Defaults to use IP addresses |

diff --git a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx index 27427ac8f46..375317fa225 100644 --- a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx +++ b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx @@ -5,11 +5,16 @@ originalFilePath: architecture-PGD-Always-ON.md --- +!!!Note + +This architecture is for Postgres Distributed 5 only. +If you require PGD 4 or 3.7 please use [BDR-Always-ON](BDR-Always-ON/). + EDB Postgres Distributed 5 in an Always-ON configuration, suitable for use in test and production. -This architecture is valid for use with EDB Postgres Distributed 5 only -and requires a subscription to [EDB Repos 2.0](reference/2q_and_edb_repositories/). +This architecture requires an EDB subscription. +All software will be sourced from [EDB Repos 2.0](reference/edb_repositories/). ## Cluster configuration @@ -80,6 +85,7 @@ data centre that provides a level of redundancy, in whatever way this definition makes sense to your use case. For example, AWS regions, your own data centres, or any other designation to identify where your servers are hosted. +!!! !!! Note for AWS users diff --git a/product_docs/docs/tpa/23/configure-cluster.mdx b/product_docs/docs/tpa/23/configure-cluster.mdx index 8cc806bf07f..69bdece7b3a 100644 --- a/product_docs/docs/tpa/23/configure-cluster.mdx +++ b/product_docs/docs/tpa/23/configure-cluster.mdx @@ -63,7 +63,7 @@ All options translate to config.yml variables in some way. A single option may affect several variables (e.g., `--bdr-version` could set `postgres_version`, -`tpa_2q_repositories`, `edb_repositories`, `extra_postgres_extensions`, and so on), but +`edb_repositories`, `extra_postgres_extensions`, and so on), but you can always accomplish with an editor what you could by running the command. @@ -112,9 +112,9 @@ be translated directly into group variables in the Ansible inventory: ```yaml cluster_vars: postgres_version: 14 - tpa_2q_repositories: - - products/bdr3/release - - products/pglogical3/release + edb_repositories: + - enterprise + - postgres_distributed postgres_conf_settings: bdr.trace_replay: true ``` diff --git a/product_docs/docs/tpa/23/configure-instance.mdx b/product_docs/docs/tpa/23/configure-instance.mdx index 8ea2eee6ff3..d25c23b260f 100644 --- a/product_docs/docs/tpa/23/configure-instance.mdx +++ b/product_docs/docs/tpa/23/configure-instance.mdx @@ -38,7 +38,7 @@ to execute tasks before any package repositories are configured. - [Configure APT repositories](reference/apt_repositories/) (for Debian and Ubuntu) -- [Configure 2ndQuadrant and EDB repositories](reference/2q_and_edb_repositories/) +- [Configure EDB repositories](reference/edb_repositories/) (on any system) - [Configure a local package repository](reference/local-repo/) @@ -120,11 +120,11 @@ cluster configuration with a minimum of effort. - [Run initdb to create the PGDATA directory](reference/initdb/) -- [Configure pg_hba.conf](reference/pg_hba_conf/) +- [Configure pg_hba.conf](reference/pg_hba.conf/) -- [Configure pg_ident.conf](reference/pg_ident_conf/) +- [Configure pg_ident.conf](reference/pg_ident.conf/) -- [Configure postgresql.conf](reference/postgresql_conf/) +- [Configure postgresql.conf](reference/postgresql.conf/) You can use the [postgres-config hook](tpaexec-hooks/#postgres-config) diff --git a/product_docs/docs/tpa/23/reference/2q_and_edb_repositories.mdx b/product_docs/docs/tpa/23/reference/2q_and_edb_repositories.mdx deleted file mode 100644 index 7bb46c0da0c..00000000000 --- a/product_docs/docs/tpa/23/reference/2q_and_edb_repositories.mdx +++ /dev/null @@ -1,129 +0,0 @@ ---- -description: How TPA uses 2ndQuadrant and EDB repositories. -title: How TPA uses 2ndQuadrant and EDB repositories -originalFilePath: 2q_and_edb_repositories.md - ---- - -This page explains the package sources from which TPA can download EDB -(including 2ndQuadrant) software, how the source varies depending on the -selected software, and how to configure access to each source. - -Note that this page only describes the special configuration options and -logic for EDB and 2ndQuadrant sources. Arbitrary -[yum](yum_repositories/) or [apt](apt_repositories/) repositories -can be added independently of the logic described here. Likewise, -packages can be [downloaded in advance](tpaexec-download-packages/) -and added to a [local repository](local-repo/) if preferred. - -## Package sources used by TPA - -TPA downloads software from three package sources. Each of these -sources provides multiple repositories. In some cases, the same software -is available from more than one source. - -- [EDB Repos 2.0](https://www.enterprisedb.com/repos/) -- [EDB Repos 1.0](https://www.enterprisedb.com/repos/legacy) -- [2ndQuadrant Repos](https://techsupport.enterprisedb.com/customer_portal/sw/) - -By default, TPA will [select sources and repositories automatically](#how-sources-are-selected-by-default) -based on the architecture and other options you have specified, so it is -not generally necessary to change these. However, you will need to -ensure that you have a valid subscription for all the sources used and -that you have [exported the token](#authenticating-with-package-sources) -before running `tpaexec deploy` or the operation will fail. - -!!! Note - - EDB is in the process of publishing all software through Repos 2.0, - and will eventually remove the older repositories. - -## Authenticating with package sources - -To use [EDB Repos 2.0](https://www.enterprisedb.com/repos/) you must -`export EDB_SUBSCRIPTION_TOKEN=xxx` before you run tpaexec. You can get -your subscription token from [the web -interface](https://www.enterprisedb.com/repos-downloads). - -To use -[2ndQuadrant repositories](https://techsupport.enterprisedb.com/customer_portal/sw/), -you must `export TPA_2Q_SUBSCRIPTION_TOKEN=xxx` before you run -tpaexec. You can get your subscription token from the 2ndQuadrant -Portal, under "Company info" in the left menu, then "Company". Some -repositories are available only by prior arrangement. - -To use [EDB Repos 1.0](https://www.enterprisedb.com/repos/legacy) you -must create a text file that contains your access credentials in the -`username:password` format and run `export -EDB_REPO_CREDENTIALS_FILE=/path/to/credentials/file` before you run -tpaexec. - -If you do not have an account for any of the sites listed, you can -register for access at - - -## How sources are selected by default - -If the PGD-Always-ON architecture is selected, repositories will be -selected from EDB Repos 2.0 and all software will be sourced -from these repositories. - -If the M1 architecture is selected and no proprietary EDB software is -selected, all packages will be sourced from PGDG. If any proprietary EDB -software is selected, all packages will be sourced from EDB Repos 2.0. - -For the BDR-Always-ON architecture, the default source is -2ndQuadrant, and the necessary repositories will be added from this -source. In addition, the PGDG repositories will be used for community -packages such as PostgreSQL and etcd as required. -If EDB software not available in the 2ndQuadrant repos is required -(e.g. EDB Advanced Server), repositories will be selected from EDB Repos -1.0. - -## Specifying EDB 2.0 repositories - -To specify the complete list of repositories from EDB Repos 2.0 to -install on each instance, set `edb_repositories` to a list of EDB -repository names: - -```yaml -cluster_vars: - edb_repositories: - - enterprise - - postgres_distributed -``` - -This example will configure the `enterprise` and `postgres_distributed` -repositories, giving access to EPAS and PGD5 products. -On Debian or Ubuntu systems, it will use the APT repository, and on -RedHat or SLES systems, it will use the rpm repositories, through the yum -or zypper frontends, respectively. - -If any EDB repositories are specified, any 2ndQuadrant repositories -specified will be ignored and no EDB Repos 1.0 will be installed. - -## Specifying 2ndQuadrant repositories - -To specify the complete list of 2ndQuadrant repositories to install on -each instance in addition to the 2ndQuadrant public repository, set -`tpa_2q_repositories` to a list of 2ndQuadrant repository names: - -```yaml -cluster_vars: - tpa_2q_repositories: - - products/pglogical3/release - - products/bdr3/release -``` - -This example will install the pglogical3 and bdr3 release repositories. -On Debian and Ubuntu systems, it will use the APT repository, and on -RedHat systems, it will use the YUM repository. - -The `dl/default/release` repository is always installed by default, -unless you - -- explicitly set `tpa_2q_repositories: []`, or -- have at least one entry in `edb_repositories`. - -Either or the above will result in no 2ndQuadrant repositories being -installed. diff --git a/product_docs/docs/tpa/23/reference/edb_repositories.mdx b/product_docs/docs/tpa/23/reference/edb_repositories.mdx index c4226cc244a..e3e58e112ec 100644 --- a/product_docs/docs/tpa/23/reference/edb_repositories.mdx +++ b/product_docs/docs/tpa/23/reference/edb_repositories.mdx @@ -1,15 +1,24 @@ --- -description: How to configure EDB Repos 2.0 package repositories on any system. +description: How to configure EDB Repos 2.0 package repositories. title: Configuring EDB Repos 2.0 repositories originalFilePath: edb_repositories.md --- -This page explains how to configure EDB Repos 2.0 package repositories -on any system. +TPA sources EDB software from +[EDB Repos 2.0](https://www.enterprisedb.com/repos/). +To use EDB Repos 2.0 you must `export EDB_SUBSCRIPTION_TOKEN=xxx` +before you run tpaexec. +You can get your subscription token from +[the web interface](https://www.enterprisedb.com/repos-downloads). -For more details on the EDB and 2ndQuadrant package sources used by -TPA see [this page](2q_and_edb_repositories/). +!!!Note + +If you create your `config.yml` file using the `tpaexec configure` +command, the `edb_repositories` key will be automatically populated with +the necessary repositories for your selected configuration, so you +shouldn't need to edit it. +!!! To specify the complete list of repositories from EDB Repos 2.0 to install on each instance, set `edb_repositories` to a list of EDB @@ -22,16 +31,8 @@ cluster_vars: - postgres_distributed ``` -This example will install the enterprise subscription repository as well -as postgres_distributed giving access to EPAS and PGD5 products. +This example will install the 'enterprise' subscription repository as +well as 'postgres_distributed' giving access to EPAS and PGD products. On Debian or Ubuntu systems, it will use the APT repository and on RedHat or SLES systems, it will use the rpm repositories, through the yum or zypper frontends respectively. - -If any EDB repositories are specified, any 2ndQuadrant repositories -specified will be ignored and no EDB Repos 1.0 will be installed. - -To use [EDB Repos 2.0](https://www.enterprisedb.com/repos/) you must -`export EDB_SUBSCRIPTION_TOKEN=xxx` before you run tpaexec. You can get -your subscription token from [the web -interface](https://www.enterprisedb.com/repos-downloads). diff --git a/product_docs/docs/tpa/23/reference/efm.mdx b/product_docs/docs/tpa/23/reference/efm.mdx index 21ae5197df8..6f262752296 100644 --- a/product_docs/docs/tpa/23/reference/efm.mdx +++ b/product_docs/docs/tpa/23/reference/efm.mdx @@ -17,7 +17,9 @@ TPA will generate `efm.nodes` and `efm.properties` with the appropriate instance-specific settings, with remaining settings set to the respective default values. TPA will also place an `efm.notification.sh` script which basically contains nothing by default and leaves it up to the user to fill it -in however they want. +in however they want. TPA will override the default settings for +`auto.allow.hosts` and `stable.nodes.file` to simplify adding agents +to the cluster. See the [EFM documentation](https://www.enterprisedb.com/docs/efm/latest/) for more details on EFM configuration. @@ -31,11 +33,8 @@ would appear in `efm.properties`: ```yaml cluster_vars: efm_conf_settings: - standby.restart.delay: 1 - application.name: quarry - reconfigure.num.sync: true - reconfigure.num.sync.max: 1 - reconfigure.sync.primary: true + notification.level=WARNING + ping.server.ip= ``` If you make changes to values under `efm_conf_settings`, TPA will always diff --git a/product_docs/docs/tpa/23/reference/patroni.mdx b/product_docs/docs/tpa/23/reference/patroni.mdx index 40356a3bcd1..a5d9dacee21 100644 --- a/product_docs/docs/tpa/23/reference/patroni.mdx +++ b/product_docs/docs/tpa/23/reference/patroni.mdx @@ -81,7 +81,7 @@ DCS config to be sent to the API and stored in the bootstrap section of the config file: - TPA vars for `postgres` are loaded into the DCS settings, - see [postgresql.conf.md](postgresql_conf/). + see [postgresql.conf.md](postgresql.conf/). Some features are not supported, see notes below. - Patroni defaults for DCS settings - User supplied defaults in `patroni_dynamic_conf_settings`, if you want diff --git a/product_docs/docs/tpa/23/reference/pem.mdx b/product_docs/docs/tpa/23/reference/pem.mdx index d71ee16a67b..39e0dd41913 100644 --- a/product_docs/docs/tpa/23/reference/pem.mdx +++ b/product_docs/docs/tpa/23/reference/pem.mdx @@ -21,8 +21,8 @@ cluster configured for use as PEM backend. All configuration options available for a normal postgres instance are valid for PEM's backend postgres instance as well. See following for details: -- [Configure pg_hba.conf](pg_hba_conf/) -- [Configure postgresql.conf](postgresql_conf/) +- [Configure pg_hba.conf](pg_hba.conf/) +- [Configure postgresql.conf](postgresql.conf/) Note that PEM is only available via EDB's package repositories and therefore requires a valid subscription. @@ -52,6 +52,27 @@ backend database user, which is set to `postgres` for postgresql and password for the web interface by running `tpaexec show-password $clusterdir $user`. +## Passing additional options when registering PEM agents + +TPA registers each PEM agent in the cluster using the `pemworker` utility's +`--register agent` command. + +A list of additional registration options can be +passed by including `pemagent_registration_opts` in the cluster config. + +For example: + +```yml + pemagent_registration_opts: + - --enable-smtp true + - --enable-heartbeat-connection + - --allow-batch-probes true + - -l DEBUG1 +``` + +The [PEM documentation](https://www.enterprisedb.com/docs/pem/latest/registering_agent/#registering-a-pem-agent-using-the-pemworker-utility) +lists more information about registration options. + ## Useful extensions for the nodes with pem agent By default, TPA will add `sql_profiler`, `edb_wait_states` and @@ -62,6 +83,41 @@ setting `pemagent_extensions` in config.yml. If this list is empty, no extensions will be automatically included. +## Providing an external certificate for PEM server SSL authentication + +By default, the PEM server creates a self-signed certificate pair, +`server-pem.crt` and `server-pem.key` and configures the webserver to use them +for HTTPS access. + +To provide your own certificate pair, create a directory under the root of the +cluster directory named `ssl/pemserver` and place the certificate pair inside. + +``` +cluster directory +├── ssl +│ └── pemserver +│ ├── externally-provided.crt +│ └── externally-provided.key +``` + +Next, set the variables `pem_server_ssl_certificate` and `pem_server_ssl_key` +with the respective file names as values for the `vars:` under the pem server +instance or `cluster_vars` in the cluster config file. + +TPA will handle copying these files over to the pem server instance and +configure the webserver accordingly. + +```yml +- Name: pemserver + location: main + node: 4 + role: + - pem-server + vars: + pem_server_ssl_certificate: externally-provided.crt + pem_server_ssl_key: externally-provided.key +``` + ## Shared PEM server Some deployments may want to use a single PEM server for monitoring and @@ -117,7 +173,7 @@ would look something like this: $ ssh -F ssh_config pemserver ``` 4. Update postgresql config on pem server node so it allows connections - from the new pg-cluster. You can modify existing pg_hba_conf on pem + from the new pg-cluster. You can modify existing pg_hba.conf on pem server by adding new entries to `pem_postgres_extra_hba_settings` under `vars:` in pem-cluster's config.yml. For example: diff --git a/product_docs/docs/tpa/23/reference/pg_hba_conf.mdx b/product_docs/docs/tpa/23/reference/pg_hba.conf.mdx similarity index 98% rename from product_docs/docs/tpa/23/reference/pg_hba_conf.mdx rename to product_docs/docs/tpa/23/reference/pg_hba.conf.mdx index 6deafd5b865..f297727862e 100644 --- a/product_docs/docs/tpa/23/reference/pg_hba_conf.mdx +++ b/product_docs/docs/tpa/23/reference/pg_hba.conf.mdx @@ -2,8 +2,7 @@ description: Customizing the pg_hba.conf file for your Postgres cluster. title: pg_hba.conf originalFilePath: pg_hba.conf.md -redirects: - - pg_hba.conf + --- The Postgres documentation explains the various options available in diff --git a/product_docs/docs/tpa/23/reference/pg_ident_conf.mdx b/product_docs/docs/tpa/23/reference/pg_ident.conf.mdx similarity index 94% rename from product_docs/docs/tpa/23/reference/pg_ident_conf.mdx rename to product_docs/docs/tpa/23/reference/pg_ident.conf.mdx index 0cae0bff9fb..aebc4242e56 100644 --- a/product_docs/docs/tpa/23/reference/pg_ident_conf.mdx +++ b/product_docs/docs/tpa/23/reference/pg_ident.conf.mdx @@ -2,8 +2,7 @@ description: Working with pg_ident.conf. title: pg_ident.conf originalFilePath: pg_ident.conf.md -redirects: - - pg_ident.conf + --- You should not normally need to change `pg_ident.conf`, and by default, diff --git a/product_docs/docs/tpa/23/reference/postgresql_conf.mdx b/product_docs/docs/tpa/23/reference/postgresql.conf.mdx similarity index 99% rename from product_docs/docs/tpa/23/reference/postgresql_conf.mdx rename to product_docs/docs/tpa/23/reference/postgresql.conf.mdx index 616910e19ba..3c7092ff2ec 100644 --- a/product_docs/docs/tpa/23/reference/postgresql_conf.mdx +++ b/product_docs/docs/tpa/23/reference/postgresql.conf.mdx @@ -2,8 +2,6 @@ description: Modifying postgresql.conf on a TPA-managed Postgres cluster. title: postgresql.conf originalFilePath: postgresql.conf.md -redirects: - - postgresql.conf --- diff --git a/product_docs/docs/tpa/23/reference/reconciling-local-changes.mdx b/product_docs/docs/tpa/23/reference/reconciling-local-changes.mdx index 2c2dae0f94e..b073c0e3c94 100644 --- a/product_docs/docs/tpa/23/reference/reconciling-local-changes.mdx +++ b/product_docs/docs/tpa/23/reference/reconciling-local-changes.mdx @@ -215,7 +215,6 @@ cluster_vars: postgres_flavour: postgresql postgres_version: '15' preferred_python_version: python3 - tpa_2q_repositories: [] instance_defaults: image: tpa/debian:11 diff --git a/product_docs/docs/tpa/23/reference/tpa_2q_repositories.mdx b/product_docs/docs/tpa/23/reference/tpa_2q_repositories.mdx deleted file mode 100644 index f780a37cade..00000000000 --- a/product_docs/docs/tpa/23/reference/tpa_2q_repositories.mdx +++ /dev/null @@ -1,43 +0,0 @@ ---- -description: How to configure 2ndQuadrant package repositories on any system. -title: Configuring 2ndQuadrant repositories -originalFilePath: tpa_2q_repositories.md - ---- - -This page explains how to configure 2ndQuadrant package repositories on -any system. - -For more details on the EDB and 2ndQuadrant package sources used by -TPA see [this page](2q_and_edb_repositories/). - -To specify the complete list of 2ndQuadrant repositories to install on -each instance in addition to the 2ndQuadrant public repository, set -`tpa_2q_repositories` to a list of 2ndQuadrant repository names: - -```yaml -cluster_vars: - tpa_2q_repositories: - - products/pglogical3/release - - products/bdr3/release -``` - -This example will install the pglogical3 and bdr3 release repositories. -On Debian and Ubuntu systems, it will use the APT repository, and on -RedHat systems, it will use the YUM repository. The 2ndQuadrant -repositories are not available for SLES systems. - -To use -[2ndQuadrant repositories](https://techsupport.enterprisedb.com/customer_portal/sw/), -you must `export TPA_2Q_SUBSCRIPTION_TOKEN=xxx` before you run -tpaexec. You can get your subscription token from the 2ndQuadrant -Portal, under "Company info" in the left menu, then "Company". Some -repositories are available only by prior arrangement. - -The `dl/default/release` repository is always installed by default, -unless you - -- explicitly set `tpa_2q_repositories: []`, or -- have at least one entry in `edb_repositories`. - -Either or the above will result in no 2ndQuadrant repositories being installed. diff --git a/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx b/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx index 08c3fb4b241..b548d2c526e 100644 --- a/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx +++ b/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx @@ -24,7 +24,7 @@ are supported. container of the target operating system and uses that system's package manager to resolve dependencies and download all necessary packages. The required Docker setup for download-packages is the same as that for - [using Docker as a deployment platform](../platform-docker/). + [using Docker as a deployment platform](#platform-docker). ## Usage diff --git a/product_docs/docs/tpa/23/reference/volumes.mdx b/product_docs/docs/tpa/23/reference/volumes.mdx index 47cfecd5624..de1aa3fc2af 100644 --- a/product_docs/docs/tpa/23/reference/volumes.mdx +++ b/product_docs/docs/tpa/23/reference/volumes.mdx @@ -28,35 +28,37 @@ instances: - device_name: root volume_type: gp2 volume_size: 32 - - raid_device: /dev/md0 - device_name: /dev/xvdf + - device_name: /dev/xvdf volume_type: io2 volume_size: 64 - raid_units: 2 - raid_level: 1 iops: 5000 vars: volume_for: postgres_data encryption: luks - - raid_device: /dev/md1 - device_name: /dev/xvdh + - device_name: /dev/xvdb ephemeral: ephemeral0 - raid_units: all vars: mountpoint: /mnt/scratch ``` In this example, the EC2 instance will end up with a 32GB EBS root -volume, a 64GB RAID-1 volume comprising two provisioned-iops EBS volumes -mounted as /opt/postgres/data, and a /tmp/scratch filesystem comprising -all available instance-store (“ephemeral”) volumes, whose number and -size are determined by the instance type. +volume, a 64GB io2 volume (provisioned-iops EBS volumes) mounted as +/opt/postgres/data, and a /tmp/scratch filesystem provided by an +instance-store (“ephemeral”) volume, whose number and size are determined +by the instance type. The details are documented in the section on AWS below, but settings like `volume_type` and `volume_size` are used during provisioning, while settings under `vars` like `volume_for` or `mountpoint` are written to the inventory for use during deployment. +!!! NOTE ephemeral0 instance store + + nowadays most of the internal storage + is NVMe in which volumes are automatically enumerated and assigned a device + name by AWS, hence you might need to modify `device_name` in your config.yml + to whatever was given after the provision phase. + ## default_volumes Volumes are properties of an instance. You cannot set them in @@ -211,33 +213,26 @@ further actions during deployment. ## RAID arrays -On AWS EC2 instances, you can define RAID volumes: +On AWS EC2 instances, only RAID 0 is recommended by Amazon. You can +create RAID volumes with a similar command: + +```shell +sudo mdadm --create --verbose /dev/md0 --level=0 --name=MY_RAID --raid-devices=number_of_volumes device_name1 device_name2 +``` + +This example will attach the block device named `/dev/md0`. The handling of +`volume_for` or `mountpoint` during deployment happens as the same as with +any other volume. TPA will handle `mkfs` and `mount` for it. ```yaml -instances: - Name: one … volumes: - - raid_device: /dev/md0 - device_name: /dev/xvdf - raid_units: 2 - raid_level: 1 - volume_type: gp2 - volume_size: 100 + - device_name: /dev/md0 vars: volume_for: postgres_data ``` -This example will attach 4×100GB EBS gp2 volumes (`/dev/xvd[f-i]`) and -assemble them into a RAID-1 volume named `/dev/md0`. The handling of -`volume_for` or `mountpoint` during deployment happens as with any other -volume. - -TPA does not currently support the creation and assembly of RAID -arrays on other platforms, but you can use an existing array by adding -an entry to volumes with `device_name: /dev/md0` or `/dev/mapper/xyz`. -TPA will handle `mkfs` and `mount` as with any other block device. - ## LUKS encryption TPA can set up a LUKS-encrypted device: diff --git a/product_docs/docs/tpa/23/task-selection.mdx b/product_docs/docs/tpa/23/task-selection.mdx index 2373f379489..70bd94ed6f4 100644 --- a/product_docs/docs/tpa/23/task-selection.mdx +++ b/product_docs/docs/tpa/23/task-selection.mdx @@ -207,6 +207,10 @@ The following selectors are supported only for exclusion: Tasks which create config files. +- efm-pre-config + + The efm-pre-config hook, if one is defined. + - fs Tasks related to setting up additional [volumes](reference/volumes/) on diff --git a/product_docs/docs/tpa/23/tpaexec-configure.mdx b/product_docs/docs/tpa/23/tpaexec-configure.mdx index 4f7d6db7380..a7988cc550c 100644 --- a/product_docs/docs/tpa/23/tpaexec-configure.mdx +++ b/product_docs/docs/tpa/23/tpaexec-configure.mdx @@ -209,34 +209,25 @@ In general, you should be able to use "Debian", "RedHat", "Ubuntu", and This option is not meaningful for the "bare" platform, where TPA has no control over which distribution is installed. -### 2ndQuadrant and EDB repositories +### EDB repositories -TPA can enable any 2ndQuadrant or EDB software repository that you have -access to through a subscription. +TPA can enable any EDB software repository that you have +access to through a subscription. By default, TPA will install any +product repositories that the architecture requires. -By default, it will install the 2ndQuadrant public repository (which -does not need a subscription) and add on any product repositories that -the architecture may require (e.g., the PGD repository). +More detailed explanation of how TPA uses EDB +repositories is available [here](reference/edb_repositories/) and on the page +for each architecture. -More detailed explanation of how TPA uses 2ndQuadrant and EDB -repositories is available [here](reference/2q_and_edb_repositories/) - -Specify `--2Q-repositories source/name/maturity …` or -`--edb-repositories repository …` to specify the complete list of -2ndQuadrant or EDB repositories to install on each instance in addition -to the 2ndQuadrant public repository. - -If any EDB repositories are specified, any 2ndQuadrant ones will be -ignored. +Specify `--edb-repositories repository …` to specify the complete list +of EDB repositories to install on each instance. Use this option with care. TPA will configure the named repositories with no attempt to make sure the combination is appropriate. -To use these options, you must `export TPA_2Q_SUBSCRIPTION_TOKEN=xxx` -or `export EDB_SUBSCRIPTION_TOKEN=xxx` before you run tpaexec. -You can get a 2ndQuadrant token from the 2ndQuadrant Portal under -"Company info" in the left menu, then "Company". You can get an EDB -token from enterprisedb.com/repos. +To use this options, you must `export EDB_SUBSCRIPTION_TOKEN=xxx` +before you run TPA. +You can get an EDB token from enterprisedb.com/repos. ### Local repository support @@ -276,7 +267,7 @@ in `--redwood` or `--no-redwood` mode, i.e., whether to enable or disable its Oracle compatibility features. Installing EDB Postgres Extended or Postgres Advanced Server requires -a valid [EDB repository subscription](reference/2q_and_edb_repositories/). +a valid [EDB repository subscription](reference/edb_repositories/). #### Package versions @@ -332,15 +323,14 @@ to address this in a future release of TPA. If you specify `--install-from-source postgres`, Postgres will be built and installed from a git repository instead of installed from -packages. Use `2ndqpostgres` instead of `postgres` to build and -install 2ndQPostgres. By default, this will build the appropriate +packages. By default, this will build the appropriate `REL_nnn_STABLE` branch. -You may use `--install-from-source 2ndqpostgres pglogical3 bdr3` to -build and install all three components from source, or just use -`--install-from-source pglogical3 bdr3` to use packages for -2ndQPostgres, but build and install pglogical v3 and PGD v3 from source. -By default, this will build the `master` branch of pglogical and PGD. +You may use `--install-from-source postgres bdr5` to +build and install both components from source, or just use +`--install-from-source bdr5` to use packages for +Postgres, but build and install PGD v5 from source. +By default, this will build the `main` branch of PGD. To build a different branch, append `:branchname` to the corresponding argument. For example `--install-from-source 2ndqpostgres:dev/xxx`, or @@ -453,7 +443,7 @@ Let's see what happens when we run the following command: ```bash [tpa]$ tpaexec configure ~/clusters/speedy --architecture M1 \ - --num-cascaded-replicas 2 --distribution Debian \ + --distribution Debian \ --platform aws --region us-east-1 --network 10.33.0.0/16 \ --instance-type t2.medium --root-volume-size 32 \ --postgres-volume-size 64 --barman-volume-size 128 \ @@ -466,12 +456,11 @@ There is no output, so there were no errors. The cluster directory has been created and populated. ```bash -$ ls ~/clusters/speedy -total 8 -drwxr-xr-x 2 ams ams 4096 Aug 4 16:23 commands --rw-r--r-- 1 ams ams 1374 Aug 4 16:23 config.yml -lrwxrwxrwx 1 ams ams 51 Aug 4 16:23 deploy.yml -> - /home/ams/work/2ndq/TPA/architectures/M1/deploy.yml +$ ls -lh ~/clusters/speedy/ +total 8.0K +drwxrwxr-x 2 haroon haroon 4.0K Aug 17 02:33 commands +-rw-rw-r-- 1 haroon haroon 1.5K Aug 17 02:33 config.yml +lrwxrwxrwx 1 haroon haroon 53 Aug 17 02:33 deploy.yml -> /home/haroon/tpa/architectures/M1/deploy.yml ``` The cluster configuration is in config.yml, and its neighbours are links @@ -484,21 +473,20 @@ architecture: M1 cluster_name: speedy cluster_tags: {} +keyring_backend: system +vault_name: cfae3da3-ec00-46cd-ab05-e153f1c788db + cluster_rules: - cidr_ip: 0.0.0.0/0 from_port: 22 proto: tcp to_port: 22 -- cidr_ip: 10.33.76.176/28 - from_port: 0 - proto: tcp - to_port: 65535 -- cidr_ip: 10.33.148.240/28 +- cidr_ip: 10.33.120.80/28 from_port: 0 proto: tcp to_port: 65535 ec2_ami: - Name: debian-10-amd64-20210721-710 + Name: debian-11-amd64-20240104-1616 Owner: '136693071363' ec2_instance_reachability: public ec2_vpc: @@ -507,7 +495,7 @@ ec2_vpc: cidr: 10.33.0.0/16 cluster_vars: - enable_pg_backup_api: false + edb_repositories: [] failover_manager: repmgr postgres_flavour: postgresql postgres_version: '14' @@ -518,11 +506,7 @@ locations: - Name: main az: us-east-1a region: us-east-1 - subnet: 10.33.76.176/28 -- Name: dr - az: us-east-1b - region: us-east-1 - subnet: 10.33.148.240/28 + subnet: 10.33.120.80/28 instance_defaults: default_volumes: @@ -542,27 +526,26 @@ instance_defaults: ansible_user: admin instances: -- Name: upsets - backup: kayak +- Name: uproar + backup: kinsman location: main node: 1 role: - primary -- Name: zebra +- Name: unravel location: main node: 2 role: - replica - upstream: upsets -- Name: kayak + upstream: uproar +- Name: kinsman location: main node: 3 role: - barman - log-server - - monitoring-server - witness - upstream: upsets + upstream: uproar volumes: - device_name: /dev/sdf encrypted: true @@ -570,18 +553,6 @@ instances: volume_for: barman_data volume_size: 128 volume_type: gp2 -- Name: queen - location: dr - node: 4 - role: - - replica - upstream: zebra -- Name: knock - location: dr - node: 5 - role: - - replica - upstream: zebra ``` The next step is to run [`tpaexec provision`](tpaexec-provision/) diff --git a/product_docs/docs/tpa/23/tpaexec-deploy.mdx b/product_docs/docs/tpa/23/tpaexec-deploy.mdx index 1d748667348..f142dc53355 100644 --- a/product_docs/docs/tpa/23/tpaexec-deploy.mdx +++ b/product_docs/docs/tpa/23/tpaexec-deploy.mdx @@ -22,9 +22,8 @@ configuration and then provisioned the servers with [`tpaexec provision`](tpaexec-provision/). Before deployment, you must -`export TPA_2Q_SUBSCRIPTION_TOKEN=xxx` to enable any 2ndQuadrant -repositories that require subscription. You can use the subscription -token that you used to [install TPA](INSTALL/) itself. If you +`export EDB_SUBSCRIPTION_TOKEN=xxx` if you are using any +[EDB repositories](reference/edb_repositories/). If you forget to do this, an error message will soon remind you. ## Quickstart diff --git a/product_docs/docs/tpa/23/tpaexec-hooks.mdx b/product_docs/docs/tpa/23/tpaexec-hooks.mdx index 7be91213a3b..5fccd1f9597 100644 --- a/product_docs/docs/tpa/23/tpaexec-hooks.mdx +++ b/product_docs/docs/tpa/23/tpaexec-hooks.mdx @@ -92,6 +92,14 @@ setting up Barman users, but before generating any Barman configuration. You can use this hook, for example, to perform any tasks related with Barman certificate files or mount points. +### efm-pre-config + +TPA invokes `hooks/efm-pre-config.yml` after installing efm, creating +its configuration directory, and setting up the efm user, but before +generating any efm configuration. + +An example use of this hook is to install efm helper scripts. + ### harp-config TPA invokes `hooks/harp-config.yml` after generating HARP configuration diff --git a/product_docs/docs/tpa/23/tpaexec-provision.mdx b/product_docs/docs/tpa/23/tpaexec-provision.mdx index e53d2f6349f..4e99986df5a 100644 --- a/product_docs/docs/tpa/23/tpaexec-provision.mdx +++ b/product_docs/docs/tpa/23/tpaexec-provision.mdx @@ -37,23 +37,21 @@ PLAY [Provision cluster] ******************************************************* ... TASK [Set up EC2 instances] ******************************************************* -changed: [localhost] => (item=us-east-1:quirk) -changed: [localhost] => (item=us-east-1:keeper) -changed: [localhost] => (item=us-east-1:zealot) -changed: [localhost] => (item=us-east-1:quaver) -changed: [localhost] => (item=us-east-1:quavery) +changed: [localhost] => (item=us-east-1:uproar) +changed: [localhost] => (item=us-east-1:unravel) +changed: [localhost] => (item=us-east-1:kinsman) ... TASK [Generate ssh_config file for the cluster] *********************************** changed: [localhost] PLAY RECAP ************************************************************************ -localhost : ok=128 changed=20 unreachable=0 failed=0 +localhost : ok=163 changed=35 unreachable=0 failed=0 skipped=44 rescued=0 ignored=2 -real 2m19.386s -user 0m51.819s -sys 0m27.852s +real 4m42.726s +user 0m39.101s +sys 0m15.687s ``` This command will produce lots of output (append `-v`, `-vv`, etc. @@ -94,38 +92,23 @@ Host * Port 22 IdentitiesOnly yes IdentityFile "id_speedy" - UserKnownHostsFile "known_hosts tpa_known_hosts" + UserKnownHostsFile known_hosts tpa_known_hosts ServerAliveInterval 60 -Host quirk +Host uproar User admin - HostName 54.227.207.189 -Host keeper + HostName 3.88.255.205 +Host unravel User admin - HostName 34.229.111.196 -Host zealot + HostName 54.80.99.142 +Host kinsman User admin - HostName 18.207.108.211 -Host quaver - User admin - HostName 54.236.36.251 -Host quavery - User admin - HostName 34.200.214.150 -[tpa]$ ssh -F ssh_config quirk -Linux quirk 4.9.0-6-amd64 #1 SMP Debian 4.9.82-1+deb9u3 (2018-03-02) x86_64 - -The programs included with the Debian GNU/Linux system are free software; -the exact distribution terms for each program are described in the -individual files in /usr/share/doc/*/copyright. - -Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent -permitted by applicable law. -Last login: Sat Aug 4 12:31:28 2018 from 136.243.148.74 -admin@quirk:~$ sudo -i -root@quirk:~# + HostName 54.165.229.179 ``` +To login to a host, use the command `ssh -F ssh_config` followed by the +hostname. For example `ssh -F ssh_config uproar`. + You can run [`tpaexec deploy`](tpaexec-deploy/) immediately after provisioning. It will wait as long as required for the instances to come up. You do not need to wait for the instances to come up, or ssh in to @@ -166,28 +149,36 @@ as well as group and host variable definitions from config.yml. ```bash [tpa]$ cat inventory/00-speedy [tag_Cluster_speedy] -quirk ansible_host=54.227.207.189 node=1 platform=aws -keeper ansible_host=34.229.111.196 node=2 platform=aws -zealot ansible_host=18.207.108.211 node=3 platform=aws -quaver ansible_host=54.236.36.251 node=4 platform=aws -quavery ansible_host=34.200.214.150 node=5 platform=aws +uproar ansible_host=3.88.255.205 node=1 platform=aws +unravel ansible_host=54.80.99.142 node=2 platform=aws +kinsman ansible_host=54.165.229.179 node=3 platform=aws [tpa]$ cat inventory/group_vars/tag_Cluster_speedy/01-speedy.yml cluster_name: speedy cluster_tag: tag_Cluster_speedy -postgres_version: 15 -tpa_version: v23.10-22-g30c1d5ea -tpa_2q_repositories: [] -vpn_network: 192.168.33.0/24 - -[tpa]$ cat inventory/host_vars/zealot/02-topology.yml +edb_repositories: [] +failover_manager: repmgr +keyring_backend: system +postgres_flavour: postgresql +postgres_version: '14' +preferred_python_version: python3 +ssh_key_file: id_speedy +tpa_version: v23.33-24-g4c0909d1 +use_volatile_subscriptions: false + +[tpa]$ cat inventory/host_vars/kinsman/01-instance_vars.yml +ansible_user: admin +location: main +region: us-east-1 role: - barman - log-server -- openvpn-server -- monitoring-server - witness -upstream: quirk +upstream: uproar +volumes: +- device: /dev/xvda +- device: /dev/sdf + volume_for: barman_data ``` If you now change a variable in config.yml and rerun provision, these From 80701a0bd9f0f62a84ea458ebfff4a5528128d7a Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 19 Aug 2024 13:18:48 +0100 Subject: [PATCH 04/10] Path fixes for docs Signed-off-by: Dj Walker-Morgan --- product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx | 2 +- .../docs/tpa/23/reference/tpaexec-download-packages.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx index 375317fa225..4dd318ac627 100644 --- a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx +++ b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx @@ -8,7 +8,7 @@ originalFilePath: architecture-PGD-Always-ON.md !!!Note This architecture is for Postgres Distributed 5 only. -If you require PGD 4 or 3.7 please use [BDR-Always-ON](BDR-Always-ON/). +If you require PGD 4 or 3.7 please use [BDR-Always-ON](architecture-BDR-Always-ON/). EDB Postgres Distributed 5 in an Always-ON configuration, suitable for use in test and production. diff --git a/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx b/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx index b548d2c526e..801d59d2ece 100644 --- a/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx +++ b/product_docs/docs/tpa/23/reference/tpaexec-download-packages.mdx @@ -24,7 +24,7 @@ are supported. container of the target operating system and uses that system's package manager to resolve dependencies and download all necessary packages. The required Docker setup for download-packages is the same as that for - [using Docker as a deployment platform](#platform-docker). + [using Docker as a deployment platform](../platform-docker). ## Usage From f6c30f9c982a07e8046d5d0bce696c60e86b97b6 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 21 Aug 2024 18:29:05 +0100 Subject: [PATCH 05/10] Refix link and note Signed-off-by: Dj Walker-Morgan --- product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx index 4dd318ac627..a93aaf9ab80 100644 --- a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx +++ b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx @@ -8,7 +8,8 @@ originalFilePath: architecture-PGD-Always-ON.md !!!Note This architecture is for Postgres Distributed 5 only. -If you require PGD 4 or 3.7 please use [BDR-Always-ON](architecture-BDR-Always-ON/). +If you require PGD 4 or 3.7 please use [BDR-Always-ON](architecture BDR-Always-ON/). +!!! EDB Postgres Distributed 5 in an Always-ON configuration, suitable for use in test and production. From 49f4303a0d64ebe2af6b20c2f3c53c809d9e411e Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 21 Aug 2024 18:32:23 +0100 Subject: [PATCH 06/10] Add template for release notes and add to index Signed-off-by: Dj Walker-Morgan --- product_docs/docs/tpa/23/rel_notes/index.mdx | 2 ++ .../docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx | 12 ++++++++++++ 2 files changed, 14 insertions(+) create mode 100644 product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx diff --git a/product_docs/docs/tpa/23/rel_notes/index.mdx b/product_docs/docs/tpa/23/rel_notes/index.mdx index b9a05882a2f..086be0dc0a5 100644 --- a/product_docs/docs/tpa/23/rel_notes/index.mdx +++ b/product_docs/docs/tpa/23/rel_notes/index.mdx @@ -2,6 +2,7 @@ title: Trusted Postgres Architect release notes navTitle: "Release notes" navigation: + - tpa_23.34_rel_notes - tpa_23.33_rel_notes - tpa_23.32_rel_notes - tpa_23.31_rel_notes @@ -31,6 +32,7 @@ The Trusted Postgres Architect documentation describes the latest version of Tru | Version | Release date | | ---------------------------- | ------------ | +| [23.34](tpa_23.34_rel_notes) | 22 Aug 2024 | | [23.33](tpa_23.33_rel_notes) | 24 Jun 2024 | | [23.32](tpa_23.32_rel_notes) | 15 May 2024 | | [23.31](tpa_23.31_rel_notes) | 19 Mar 2024 | diff --git a/product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx b/product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx new file mode 100644 index 00000000000..841c7b83937 --- /dev/null +++ b/product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx @@ -0,0 +1,12 @@ +--- +title: Trusted Postgres Architect 23.34 release notes +navTitle: "Version 23.34" +--- + +Released: 22 August 2024 + +New features, enhancements, bug fixes, and other changes in Trusted Postgres Architect 23.34 include the following: + +| Type | Description | +|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Type-here | Description-here | From c2884a884973511069e1858efc1bb8ca9e4cf32b Mon Sep 17 00:00:00 2001 From: Simon Notley <43099400+sonotley@users.noreply.github.com> Date: Wed, 21 Aug 2024 19:40:03 +0100 Subject: [PATCH 07/10] Add TPA 23.34 release notes --- .../tpa/23/rel_notes/tpa_23.34_rel_notes.mdx | 21 ++++++++++++++++++- 1 file changed, 20 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx b/product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx index 841c7b83937..5433f77c464 100644 --- a/product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx +++ b/product_docs/docs/tpa/23/rel_notes/tpa_23.34_rel_notes.mdx @@ -9,4 +9,23 @@ New features, enhancements, bug fixes, and other changes in Trusted Postgres Arc | Type | Description | |---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| Type-here | Description-here | +| Enhancement | TPA now has an `efm-pre-config` hook which runs after efm has been installed and its configuration directory and user have been created, but before efm is configured. It can be used to install custom efm helper scripts. | +| Enhancement | TPA now has its own output plugin, which shows one line of information per task, omitting tasks for which even one line would be uninformative. The lines are indented to enable TPA's control flow to be visible, and include color-coded counts of successful, skipped, and ignored hosts. The fuller output can be turned on by setting TPA_USE_DEFAULT_OUTPUT=true in your environment, or by adding the -v switch to the command line. | +| Enhancement | TPA now allows you to specify additional options when registering PEM agents by listing them under `pemagent_registration_opts` in `cluster_vars`. | +| Enhancement | You can now provide your own web server certificates for use with the PEM server by including the names of the certificate and key pair for use on the PEM server in `config.yml` under the cluster_vars or pem-server instance vars `pem_server_ssl_certificate` and `pem_server_ssl_key`. TPA will copy them from the `ssl/pemserver` directory of the cluster directory to the PEM server and configure Apache/httpd accordingly. | +| Enhancement | TPA now runs the EFM `upgrade-conf` command on new cluster deployments to benefit from the comments and inline documentation that are added to both `.properties` and `.nodes` files. | +| Enhancement | TPA can now set up EFM clusters using hostname resolution instead of IP addresses for `bind.address` value. This can be invoked with tha `--efm-bind-by-hostname` option for the configure command or `efm_bind_by_hostname: true|false` in cluster_vars section of config.yml. | +| Enhancement | TPA now supports setting the EFM properties that added in EFM 4.9: `enable.stop.cluster: boolean, default true`, `priority.standbys: default ''`, `detach.on.agent.failure: boolean, default false`, `pid.dir: default ''`. | +| Change | TPA no longer supports RAID creation on AWS. | +| Change | Removed EFM dependency for resolving upstream_primary. Previously, EFM was queried for the current primary on a deploy after a switchover. If EFM is not running, this will fail. Now the cluster_facts collected through Postgres are used to determine the current primary after a switchover, removing the dependency on EFM. | +| Change | In EFM clusters, the `upstream_primary` is now correctly updated after switchover, resulting in the correct `auto.reconfigure` setting be set on replicas. Standbys now follow the new primary. | +| Bug Fix | Fixed an issue whereby TPA would incorrectly apply proxy settings when accessing the Patroni API. The Ansible default is to use a proxy, if defined. This does not work in the (rather common) case of an airgapped environment that needs a proxy to download packages from the internet, because the proxy also intercepts (and disrupts) calls to the Patroni API. | +| Bug Fix | Fixed an issue whereby TPA would set PEM agent parameters on all instances that were only appropriate for the `pemserver` instance. | +| Bug Fix | Added missing entries for pgd-proxy and pgdcli default package name when using SLES operating system as target for cluster nodes. | +| Bug Fix | Fix an issue whereby TPA would fail to reload/restart postgres on existing nodes to re-read configuration changes and the new node would therefore fail to connect to the cluster. | +| Bug Fix | Fixed an issue whereby when taking backups from a replica, barman could fail when taking its initial backup by timing out waiting for WAL files. This is fixed by waiting for barman to complete its base backup before forcing a WAL segment switch. | +| Bug Fix | Ensure that `repmgr witness register` command is used with the correct postgres_port value even when using non-default postgres port for the upstream_primary postgres. | +| Bug Fix | Fixed an issue whereby failover_manager override to `repmgr` would not work correctly when set at instance level for subscriber-only nodes and their replicas in PGD clusters. | +| Bug Fix | Fixed two cases of incorrect cgroup detection: on MacOSX, we no longer try to read `/proc/mounts`. On systems where `/sys/fs/cgroup` is `ro` but mounts under it are `rw`, TPA now correctly detects this. | +| Bug Fix | Ensure we can verify the actual config set on pgd-proxy nodes for the newly added `read_listen_port` option in pgd-proxy. | +| Bug Fix | Fixed an issue that would prevent deployment with PEM 9.7.0. PEM 9.7.0 no longer depends on Apache at a package level therefore to use Apache as the web server we install the packages explicitly. | \ No newline at end of file From 3a1ecfe141eb1e41f07b3e4116e951fd32829a6c Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Thu, 22 Aug 2024 08:57:14 +0100 Subject: [PATCH 08/10] Fix note in architecture-PGD-Always-ON.mdx --- product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx index a93aaf9ab80..f543850d667 100644 --- a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx +++ b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx @@ -5,8 +5,7 @@ originalFilePath: architecture-PGD-Always-ON.md --- -!!!Note - +!!! Note This architecture is for Postgres Distributed 5 only. If you require PGD 4 or 3.7 please use [BDR-Always-ON](architecture BDR-Always-ON/). !!! From 7b06669a217fccd09ac7a489119bd112f610780c Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 22 Aug 2024 09:07:05 +0100 Subject: [PATCH 09/10] Fix up notes and links on architecture page Signed-off-by: Dj Walker-Morgan --- product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx index f543850d667..948cf5d1aba 100644 --- a/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx +++ b/product_docs/docs/tpa/23/architecture-PGD-Always-ON.mdx @@ -7,7 +7,7 @@ originalFilePath: architecture-PGD-Always-ON.md !!! Note This architecture is for Postgres Distributed 5 only. -If you require PGD 4 or 3.7 please use [BDR-Always-ON](architecture BDR-Always-ON/). +If you require PGD 4 or 3.7 please use [BDR-Always-ON](architecture-BDR-Always-ON/). !!! EDB Postgres Distributed 5 in an Always-ON configuration, @@ -85,9 +85,9 @@ data centre that provides a level of redundancy, in whatever way this definition makes sense to your use case. For example, AWS regions, your own data centres, or any other designation to identify where your servers are hosted. -!!! -!!! Note for AWS users + +!!! Note Note for AWS users If you are using TPA to provision an AWS cluster, the locations will be mapped to separate availability zones within the `--region` you From aeb2f3abb8fc8546575c22c50368201d64d8b98f Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 22 Aug 2024 09:39:39 +0100 Subject: [PATCH 10/10] Add front page update Signed-off-by: Dj Walker-Morgan --- src/constants/updates.js | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/constants/updates.js b/src/constants/updates.js index ad517bb77c1..27c0a9416c2 100644 --- a/src/constants/updates.js +++ b/src/constants/updates.js @@ -1,6 +1,14 @@ import IconNames from "../components/icon/iconNames"; export const updates = [ + { + title: "Trusted Postgres Architect 23.34", + icon: IconNames.INSTANCES, + description: + "TPA 23.34 includes enhanced support for EFM, PEM and pgBouncer, and a new output plugin which improves readability of TPA progress.", + url: "/tpa/latest/", + moreUrl: "/tpa/latest/rel_notes/tpa_23.34_rel_notes/", + }, { title: "EDB Postgres Enterprise Manager 9.7", icon: IconNames.EDB_PEM,