From c320cb82553f51b5d3b67347c044a9ececa05495 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Thu, 30 Nov 2023 11:25:33 -0500 Subject: [PATCH 01/68] Edits to postgresql uninstall topic pr5055 --- .../postgresql/overview.mdx | 2 +- .../postgresql/uninstalling.mdx | 28 ++++--------------- 2 files changed, 6 insertions(+), 24 deletions(-) diff --git a/advocacy_docs/supported-open-source/postgresql/overview.mdx b/advocacy_docs/supported-open-source/postgresql/overview.mdx index 71f7dcac37c..af1b66ae9ce 100644 --- a/advocacy_docs/supported-open-source/postgresql/overview.mdx +++ b/advocacy_docs/supported-open-source/postgresql/overview.mdx @@ -11,7 +11,7 @@ EDB's PostgreSQL installers and installation packages simplify the process of in - The Stack Builder package manager for Windows and MacOS. Use this utility to download and install drivers, tools, and applications to complement your PostgreSQL installation. To obtain PostgreSQL installers and installation packages: -- EDB provides installers and installation packages for most common computing platforms. [EDB installation documention](installing) describes how to obtain installers and installation packages from an EDB repo. +- EDB provides installers and installation packages for most common computing platforms. [EDB installation documentation](installing) describes how to obtain installers and installation packages from an EDB repo. - The PostgreSQL community provides installers and installation packages for most computing platforms. For more information, see the [PostgreSQL downloads page](https://www.postgresql.org/download/). EDB also makes PostgreSQL Early Experience distributions available. For details, see [PostgreSQL Early Experience](https://www.enterprisedb.com/postgresql-early-experience). diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index 229cc4b60d7..aeb0bec9ff5 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -19,29 +19,19 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d ## Uninstalling PostgreSQL on Windows -1. On the Windows Control Panel, open the Uninstall or change a program dialog box. - - ![The Uninstall or change a program dialog box](/installing/images/change_program.png) +1. On the Windows Control Panel, open the Uninstall or Change a Program dialog box. 1. Right-click **PostgreSQL**, and select **Uninstall/Change** from the context menu. - ![Confirm that you want to uninstall PostgreSQL](/installing/images/uninstalling_postgreSQL_comfirmation.png) - 1. You can remove the entire application or individual components. - To remove the entire application, select **Entire application** and select **Next**. - To remove individual components, select **Individual components** and select **Next**. Then, from the list, select the components to uninstall, and select **Next**. - ![Select the components to uninstall](/installing/images/select_the_components_to_uninstall.png) - A progress bar keeps you informed as PostgreSQL is removed. - ![Uninstalling PostgreSQL](/installing/images/uninstalling_postgreSQL.png) - 1. When the uninstaller completes, select **OK**. - ![An Info dialog box confirms the uninstallation](/installing/images/uninstallationcomplete.png) - ## Uninstalling PostgreSQL on Mac These instructions apply if you used EDB's interactive Mac installer to install PostgreSQL. @@ -53,10 +43,10 @@ These instructions apply if you used EDB's interactive Mac installer to install To uninstall PostgreSQL on a Mac system: !!! Note Data is not deleted -This uninstaller will remove the application but, to avoid data loss, will -leave in place the Postgres data directory and the postgres user that was -created to manage the installation. The uninstaller will popup a message -confirming what it has not been removed at the end of its run. +This uninstaller removes the application but, to avoid data loss, +leaves in place the Postgres data directory and the postgres user that was +created to manage the installation. At the end of its run, the uninstaller +displays a message confirming what it didn't remove the data directory or user. !!! 1. Invoke the uninstaller: @@ -71,19 +61,11 @@ confirming what it has not been removed at the end of its run. The uninstaller opens and asks if you want to uninstall the entire application or individual components. - ![Remove the entire application or components](/installing/images/remove_the_entire_application_or_components.png) - 1. Respond to the installer: - To remove the entire application, select **Entire application**, and select **Next**. - To remove individual components, select **Individual components**. Select the components to uninstall, and select **Next** to start uninstalling components. - ![Selecting the components to uninstall](/installing/images/selecting_the_components_to_uninstall.png) - Progress bars appear as each component is removed. 1. When the uninstaller completes, select **OK**. - -![The Uninstallation is completed](/installing/images/mac_uninstallation_completed.png) - - From 76a1239b168661df466d8a50e8001eac227325a8 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:20:32 -0500 Subject: [PATCH 02/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index aeb0bec9ff5..a9d04e5a822 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -21,6 +21,8 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d 1. On the Windows Control Panel, open the Uninstall or Change a Program dialog box. +![The Uninstall or Change a Program dialog box](/installing/images/change_program.png) + 1. Right-click **PostgreSQL**, and select **Uninstall/Change** from the context menu. 1. You can remove the entire application or individual components. From fed4edf7724f05a1772079847ebc1de1b6e194a6 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:23:41 -0500 Subject: [PATCH 03/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index a9d04e5a822..fba15bf51d1 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -25,6 +25,8 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d 1. Right-click **PostgreSQL**, and select **Uninstall/Change** from the context menu. +![Confirm that you want to uninstall PostgreSQL](/installing/images/uninstalling_postgreSQL_comfirmation.png) + 1. You can remove the entire application or individual components. - To remove the entire application, select **Entire application** and select **Next**. From 462045582d76c6afc96728a84e32642879067c94 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:24:26 -0500 Subject: [PATCH 04/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index fba15bf51d1..803e50e45ef 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -31,6 +31,8 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d - To remove the entire application, select **Entire application** and select **Next**. - To remove individual components, select **Individual components** and select **Next**. Then, from the list, select the components to uninstall, and select **Next**. + + ![Select the components to uninstall](/installing/images/select_the_components_to_uninstall.png) A progress bar keeps you informed as PostgreSQL is removed. From 9a4f2949088c58d29ac95eb1e40b82862567a28e Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:25:40 -0500 Subject: [PATCH 05/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index 803e50e45ef..d0b9a623910 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -35,6 +35,8 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d ![Select the components to uninstall](/installing/images/select_the_components_to_uninstall.png) A progress bar keeps you informed as PostgreSQL is removed. + + ![Uninstalling PostgreSQL](/installing/images/uninstalling_postgreSQL.png) 1. When the uninstaller completes, select **OK**. From 4c5266eb483c9aefb6a59f86db9158a62b478fa7 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:26:31 -0500 Subject: [PATCH 06/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index d0b9a623910..79c00e14a8c 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -40,6 +40,8 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d 1. When the uninstaller completes, select **OK**. +![An Info dialog box confirms the uninstallation](/installing/images/uninstallationcomplete.png) + ## Uninstalling PostgreSQL on Mac These instructions apply if you used EDB's interactive Mac installer to install PostgreSQL. From 3c33d0bcbe5eab7eb57d20b28925b5756d5042c6 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:28:34 -0500 Subject: [PATCH 07/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx Co-authored-by: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index 79c00e14a8c..f5477457812 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -56,7 +56,7 @@ To uninstall PostgreSQL on a Mac system: This uninstaller removes the application but, to avoid data loss, leaves in place the Postgres data directory and the postgres user that was created to manage the installation. At the end of its run, the uninstaller -displays a message confirming what it didn't remove the data directory or user. +displays a message confirming that it didn't remove the data directory or user. !!! 1. Invoke the uninstaller: From d22e460c68275cd5f14c36397315c558cee83af8 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:35:12 -0500 Subject: [PATCH 08/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index f5477457812..b0dd9aa1d35 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -70,6 +70,8 @@ displays a message confirming that it didn't remove the data directory or user. 1. You're prompted for a password. If your user has admin rights on macOS, enter your password. Otherwise, enter the user name and password of a user with admin rights. The uninstaller opens and asks if you want to uninstall the entire application or individual components. + + ![Remove the entire application or components](/installing/images/remove_the_entire_application_or_components.png) 1. Respond to the installer: From c3aecf85ca213cee918fcc0fa5e304603702de76 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:39:44 -0500 Subject: [PATCH 09/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- advocacy_docs/supported-open-source/postgresql/uninstalling.mdx | 2 ++ 1 file changed, 2 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index b0dd9aa1d35..6936763aea0 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -77,6 +77,8 @@ displays a message confirming that it didn't remove the data directory or user. - To remove the entire application, select **Entire application**, and select **Next**. - To remove individual components, select **Individual components**. Select the components to uninstall, and select **Next** to start uninstalling components. + +![Selecting the components to uninstall](/installing/images/selecting_the_components_to_uninstall.png) Progress bars appear as each component is removed. From cf29a50eeb48680b27cb3685236d487d204613b3 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Tue, 5 Dec 2023 13:40:49 -0500 Subject: [PATCH 10/68] Update advocacy_docs/supported-open-source/postgresql/uninstalling.mdx --- .../supported-open-source/postgresql/uninstalling.mdx | 3 +++ 1 file changed, 3 insertions(+) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index 6936763aea0..ebec9d72288 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -83,3 +83,6 @@ displays a message confirming that it didn't remove the data directory or user. Progress bars appear as each component is removed. 1. When the uninstaller completes, select **OK**. + +![The Uninstallation is completed](/installing/images/mac_uninstallation_completed.png) + From 5c2ef988aee5632340936883cc592ba34546e1aa Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 5 Dec 2023 14:55:14 +0000 Subject: [PATCH 11/68] Added PGE 15.5 Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pge/15/release_notes/index.mdx | 8 +++++--- .../docs/pge/15/release_notes/rel_notes15.3.mdx | 2 +- .../docs/pge/15/release_notes/rel_notes15.4.mdx | 2 +- .../docs/pge/15/release_notes/rel_notes15.5.mdx | 16 ++++++++++++++++ 4 files changed, 23 insertions(+), 5 deletions(-) create mode 100644 product_docs/docs/pge/15/release_notes/rel_notes15.5.mdx diff --git a/product_docs/docs/pge/15/release_notes/index.mdx b/product_docs/docs/pge/15/release_notes/index.mdx index 82131a33b20..61bcc07609e 100644 --- a/product_docs/docs/pge/15/release_notes/index.mdx +++ b/product_docs/docs/pge/15/release_notes/index.mdx @@ -1,16 +1,18 @@ --- title: "Release notes" navigation: + - rel_notes15.5 - rel_notes15.4 - rel_notes15.3 - rel_notes15.2 --- -The EDB Postgres Extended Server documentation describes the latest version of EDB Postgres Extended Server 15, -including minor releases and patches. These release notes -cover what was new in each release. +The EDB Postgres Extended Server documentation describes the latest version of +EDB Postgres Extended Server 15, including minor releases and patches. These +release notes cover what was new in each release. | Version | Release date | | ------------------------ | ------------ | +| [15.5](rel_notes15.5) | 09 Nov 2023 | | [15.4](rel_notes15.4) | 21 Aug 2023 | | [15.3](rel_notes15.3) | 11 May 2023 | | [15.2](rel_notes15.2) | 14 Feb 2023 | diff --git a/product_docs/docs/pge/15/release_notes/rel_notes15.3.mdx b/product_docs/docs/pge/15/release_notes/rel_notes15.3.mdx index a91c59b27c9..602c21490cf 100644 --- a/product_docs/docs/pge/15/release_notes/rel_notes15.3.mdx +++ b/product_docs/docs/pge/15/release_notes/rel_notes15.3.mdx @@ -5,7 +5,7 @@ navTitle: Version 15.3 Released: 11 May 2023 -New features, enhancements, bug fixes, and other changes in EDB Postgres Extended Server 15.2 include: +New features, enhancements, bug fixes, and other changes in EDB Postgres Extended Server 15.3 include: | Type | Description | Category | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------| --------------------- | diff --git a/product_docs/docs/pge/15/release_notes/rel_notes15.4.mdx b/product_docs/docs/pge/15/release_notes/rel_notes15.4.mdx index c1a9b229e90..846de90f9e9 100644 --- a/product_docs/docs/pge/15/release_notes/rel_notes15.4.mdx +++ b/product_docs/docs/pge/15/release_notes/rel_notes15.4.mdx @@ -5,7 +5,7 @@ navTitle: Version 15.4 Released: 21 Aug 2023 -New features, enhancements, bug fixes, and other changes in EDB Postgres Extended Server 15.2 include: +New features, enhancements, bug fixes, and other changes in EDB Postgres Extended Server 15.4 include: | Type | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | diff --git a/product_docs/docs/pge/15/release_notes/rel_notes15.5.mdx b/product_docs/docs/pge/15/release_notes/rel_notes15.5.mdx new file mode 100644 index 00000000000..6215fc5b510 --- /dev/null +++ b/product_docs/docs/pge/15/release_notes/rel_notes15.5.mdx @@ -0,0 +1,16 @@ +--- +title: "EDB Postgres Extended Server 15.5 release notes" +navTitle: Version 15.5 +--- + +Released: 9 Nov 2023 + +New features, enhancements, bug fixes, and other changes in EDB Postgres Extended Server 15.2 include: + +| Type | Description | +| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Upstream merge | Merged with community PostgreSQL 15.5. See the [PostgreSQL 15 Release Notes](https://www.postgresql.org/docs/15/release-15-5.html) for more information. | + + + + From fdd4646e6c95e3437828b616b9b124b6f9654112 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Wed, 6 Dec 2023 18:44:00 +0000 Subject: [PATCH 12/68] Update postgres-configuration.mdx to address invalid ordering --- product_docs/docs/pgd/5/postgres-configuration.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/product_docs/docs/pgd/5/postgres-configuration.mdx b/product_docs/docs/pgd/5/postgres-configuration.mdx index 3ed58e9cf95..bcead543b7e 100644 --- a/product_docs/docs/pgd/5/postgres-configuration.mdx +++ b/product_docs/docs/pgd/5/postgres-configuration.mdx @@ -15,8 +15,10 @@ For PGD's own settings, see the [PGD settings reference](reference/pgd-settings) To run correctly, PGD requires these Postgres settings: - `wal_level` — Must be set to `logical`, since PGD relies on logical decoding. -- `shared_preload_libraries` — Must start with `bdr`, before other - entries. Don't include `pglogical` in this list +- `shared_preload_libraries` — Must include `bdr` to enable the extension. Most other + extensions can appear before of after the bdr entry in the comma-separated list. One exception + to that is `pgaudit` which must appear in the list before `bdr`. Also, do not include + `pglogical` in this list - `track_commit_timestamp` — Must be set to `on` for conflict resolution to retrieve the timestamp for each conflicting row. From 93570eafcfb50b0b721789fa87eae2c467e4e83a Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 6 Dec 2023 17:15:29 +0000 Subject: [PATCH 13/68] Added API compat table Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx b/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx index 4cdd374dc2e..f6301030ff2 100644 --- a/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx +++ b/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx @@ -7,6 +7,19 @@ title: Supported BDR upgrade paths Beginning with version 4, EDB Postgres Distributed adopted [semantic versioning](https://semver.org/). All changes within the same major version are backward compatible, lowering the risk when upgrading and allowing you to choose any later minor or patch release as the upgrade target. You can upgrade from any version 4.x release to a later 4.x release. +Due to interdependencies between EDB Postgres Distributed and EDB Postgres Advanced Server or EDB Postgres Extended, there are occasions when it is important to always upgrade both components at once. More detailed compatibility information can be found below. + +PGD version | EPAS version | PGE version +------------|--------------|------------------- +Up to 4.3.0 | Up to 14.7/ 15.2 | Up to 14.9/ 15.4 +4.3.1,4.3.2 | 14.8,14.9/ 15.3, 15.4 | Up to 14.9/ 15.4 +4.3.3 and higher | 14.10 / 15.5 and higher | 14.10 / 15.5 and higher + +This dependency is bidirectional dependency. + +* Upgrading your Postgres servers will require you to upgrade your PGD installation to the relevant version. +* Upgrading your PGD installation will require you to upgrade your Postgres servers to the relevant version. + ## Upgrading from version 3.7 to version 4 Generally, we recommend that you upgrade to the latest version 3.7 release before From 986e8f452fbf3f773440e8844471b257699a1c05 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 6 Dec 2023 17:28:06 +0000 Subject: [PATCH 14/68] Adjusted table Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx b/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx index f6301030ff2..fda5504a23a 100644 --- a/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx +++ b/product_docs/docs/pgd/4/upgrades/upgrade_paths.mdx @@ -9,10 +9,10 @@ Beginning with version 4, EDB Postgres Distributed adopted [semantic versioning] Due to interdependencies between EDB Postgres Distributed and EDB Postgres Advanced Server or EDB Postgres Extended, there are occasions when it is important to always upgrade both components at once. More detailed compatibility information can be found below. -PGD version | EPAS version | PGE version -------------|--------------|------------------- -Up to 4.3.0 | Up to 14.7/ 15.2 | Up to 14.9/ 15.4 -4.3.1,4.3.2 | 14.8,14.9/ 15.3, 15.4 | Up to 14.9/ 15.4 + PGD version | EPAS version | PGE version +-----------------|-------------------------|------------------ +Up to 4.3.0 | Up to 14.7 / 15.2 | Up to 14.9 / 15.4 +4.3.1, 4.3.2 | 14.8, 14.9 / 15.3, 15.4 | Up to 14.9/ 15.4 4.3.3 and higher | 14.10 / 15.5 and higher | 14.10 / 15.5 and higher This dependency is bidirectional dependency. From a1c296d7e5d7422b4d3f1da1914d14ec670ea2a7 Mon Sep 17 00:00:00 2001 From: Betsy Gitelman Date: Thu, 7 Dec 2023 09:57:03 -0500 Subject: [PATCH 15/68] Fixed step numbering issue --- .../supported-open-source/postgresql/uninstalling.mdx | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx index ebec9d72288..87457d12b3f 100644 --- a/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx +++ b/advocacy_docs/supported-open-source/postgresql/uninstalling.mdx @@ -21,11 +21,11 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d 1. On the Windows Control Panel, open the Uninstall or Change a Program dialog box. -![The Uninstall or Change a Program dialog box](/installing/images/change_program.png) + ![The Uninstall or Change a Program dialog box](/installing/images/change_program.png) 1. Right-click **PostgreSQL**, and select **Uninstall/Change** from the context menu. -![Confirm that you want to uninstall PostgreSQL](/installing/images/uninstalling_postgreSQL_comfirmation.png) + ![Confirm that you want to uninstall PostgreSQL](/installing/images/uninstalling_postgreSQL_comfirmation.png) 1. You can remove the entire application or individual components. @@ -40,7 +40,7 @@ The PostgreSQL installer creates an uninstaller in the PostgreSQL installation d 1. When the uninstaller completes, select **OK**. -![An Info dialog box confirms the uninstallation](/installing/images/uninstallationcomplete.png) + ![An Info dialog box confirms the uninstallation](/installing/images/uninstallationcomplete.png) ## Uninstalling PostgreSQL on Mac @@ -78,11 +78,11 @@ displays a message confirming that it didn't remove the data directory or user. - To remove the entire application, select **Entire application**, and select **Next**. - To remove individual components, select **Individual components**. Select the components to uninstall, and select **Next** to start uninstalling components. -![Selecting the components to uninstall](/installing/images/selecting_the_components_to_uninstall.png) + ![Selecting the components to uninstall](/installing/images/selecting_the_components_to_uninstall.png) Progress bars appear as each component is removed. 1. When the uninstaller completes, select **OK**. -![The Uninstallation is completed](/installing/images/mac_uninstallation_completed.png) + ![The Uninstallation is completed](/installing/images/mac_uninstallation_completed.png) From 109ebce4532f521b89c48653d5c8939f720719c9 Mon Sep 17 00:00:00 2001 From: Bobby Bissett <70302203+EFM-Bobby@users.noreply.github.com> Date: Thu, 7 Dec 2023 09:34:03 -0500 Subject: [PATCH 16/68] update old efm version reference An old reference to efm 4.2 was missed along the way. --- .../efm/4/04_configuring_efm/04_extending_efm_permissions.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx b/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx index 004880724b2..c974d7385ab 100644 --- a/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx +++ b/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx @@ -21,7 +21,7 @@ The `efm_db_functions` or `efm_root_functions` scripts perform management functi The `sudoers` file contains entries that allow the user efm to control the Failover Manager service for clusters owned by postgres or enterprisedb. You can modify a copy of the `sudoers` file to grant permission to efm to manage Postgres clusters owned by other users. -The `efm-42` file is located in `/etc/sudoers.d` and contains the following entries: +The `efm-48` file is located in `/etc/sudoers.d` and contains the following entries: ```text # Copyright EnterpriseDB Corporation, 2014-2021. All Rights Reserved. @@ -53,7 +53,7 @@ efm ALL=(ALL) NOPASSWD: /usr/edb/efm-4.8/bin/efm_pgpool_functions Defaults:efm !requiretty ``` -If you're using Failover Manager to monitor clusters that are owned by users other than postgres or enterprisedb, make a copy of the `efm-42` file and modify the content to allow the user to access the `efm_functions` script to manage their clusters. +If you're using Failover Manager to monitor clusters that are owned by users other than postgres or enterprisedb, make a copy of the `efm-48` file and modify the content to allow the user to access the `efm_functions` script to manage their clusters. If an agent can't start because of permission problems, make sure the default `/etc/sudoers` file contains the following line at the end of the file: From a5da9f8fa78bf0f8cfc5399a7cc1ca158f8edb7d Mon Sep 17 00:00:00 2001 From: Bobby Bissett <70302203+EFM-Bobby@users.noreply.github.com> Date: Thu, 7 Dec 2023 11:03:52 -0500 Subject: [PATCH 17/68] Update product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx Co-authored-by: Betsy Gitelman --- .../efm/4/04_configuring_efm/04_extending_efm_permissions.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx b/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx index c974d7385ab..c0906b88a0c 100644 --- a/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx +++ b/product_docs/docs/efm/4/04_configuring_efm/04_extending_efm_permissions.mdx @@ -53,7 +53,7 @@ efm ALL=(ALL) NOPASSWD: /usr/edb/efm-4.8/bin/efm_pgpool_functions Defaults:efm !requiretty ``` -If you're using Failover Manager to monitor clusters that are owned by users other than postgres or enterprisedb, make a copy of the `efm-48` file and modify the content to allow the user to access the `efm_functions` script to manage their clusters. +If you're using Failover Manager to monitor clusters that are owned by users other than postgres or enterprisedb, make a copy of the `efm-48` file. Then modify the content to allow the user to access the `efm_functions` script to manage their clusters. If an agent can't start because of permission problems, make sure the default `/etc/sudoers` file contains the following line at the end of the file: From 87cd821d881bb2e1c08bc05b8cf842e01ed2099a Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 3 Jul 2023 13:19:32 +0100 Subject: [PATCH 18/68] First draft pass on referencizing durability Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/camo.mdx | 225 ++++++------------ .../docs/pgd/5/durability/commit-scopes.mdx | 44 +--- .../docs/pgd/5/reference/functions.mdx | 145 +++++++++++ product_docs/docs/pgd/5/reference/index.json | 2 +- product_docs/docs/pgd/5/reference/index.mdx | 11 + 5 files changed, 241 insertions(+), 186 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/camo.mdx b/product_docs/docs/pgd/5/durability/camo.mdx index 2cfde7278d8..a786074477f 100644 --- a/product_docs/docs/pgd/5/durability/camo.mdx +++ b/product_docs/docs/pgd/5/durability/camo.mdx @@ -54,16 +54,15 @@ When the `DEGRADE ON ... TO ASYNC` clause is used in the commit scope, a node de This doesn't allow COMMIT status to be retrieved, but it does let you choose availability over consistency. This mode can tolerate a single-node failure. In case both nodes of a CAMO pair fail, they might choose incongruent commit decisions to maintain availability, leading to data inconsistencies. -For a CAMO partner to switch to ready, it needs to be connected, and the estimated catchup interval needs to drop below the `timeout` value of `TO ASYNC`. You can check the current readiness status of a CAMO partner with `bdr.is_camo_partner_ready`, while `bdr.node_replication_rates` provides the current estimate of the catchup time. +For a CAMO partner to switch to ready, it needs to be connected, and the estimated catchup interval needs to drop below the `timeout` value of `TO ASYNC`. You can check the current readiness status of a CAMO partner with [`bdr.is_camo_partner_ready()`](/pgd/latest/reference/functions#bdris_camo_partner_ready), while [`bdr.node_replication_rates`](/pgd/latest/reference/catalogs-visible#bdrnode_replication_rates) provides the current estimate of the catchup time. - The switch from CAMO protected to asynchronous mode is only ever triggered by an actual CAMO transaction either because the commit exceeds the `timeout` value of `TO ASYNC` or, in case the CAMO partner is already known, disconnected at the time of commit. This switch is independent of the estimated catchup interval. If the CAMO pair is configured to require the current node to be the write lead of a group as configured through `enable_proxy_routing` node group option. See [Commit scopes](commit-scopes) for syntax. This can prevent a split brain situation due to an isolated node from switching to asynchronous mode. If `enable_proxy_routing` isn't set for the CAMO group, the origin node switches to asynchronous mode immediately. The switch from asynchronous mode to CAMO mode depends on the CAMO partner node, which initiates the connection. The CAMO partner tries to reconnect at least every 30 seconds. After connectivity is reestablished, it might therefore take up to 30 seconds until the CAMO partner connects back to its origin node. Any lag that accumulated on the CAMO partner further delays the switch back to CAMO protected mode. Unlike during normal CAMO operation, in asynchronous mode there's no added commit overhead. This can be problematic, as it allows the node to continuously process more transactions than the CAMO pair can normally process. Even if the CAMO partner eventually reconnects and applies transactions, its lag only ever increases -in such a situation, preventing reestablishing the CAMO protection. To artificially throttle transactional throughput, PGD provides the `bdr.camo_local_mode_delay` setting, which allows you to delay a `COMMIT` in local mode by an arbitrary amount of time. We recommend measuring commit times in normal CAMO mode during expected workloads and configuring this delay accordingly. The default is 5 ms, which reflects a asynchronous network and a relatively quick CAMO partner response. +in such a situation, preventing reestablishing the CAMO protection. To artificially throttle transactional throughput, PGD provides the [`bdr.camo_local_mode_delay`](/pgd/latest/reference/pgd-settings#bdrcamo_local_mode_delay) setting, which allows you to delay a `COMMIT` in local mode by an arbitrary amount of time. We recommend measuring commit times in normal CAMO mode during expected workloads and configuring this delay accordingly. The default is 5 ms, which reflects a asynchronous network and a relatively quick CAMO partner response. Consider the choice of whether to allow asynchronous mode in view of the architecture and the availability requirements. The following examples provide some detail. @@ -109,157 +108,61 @@ CAMO relies on a retry loop and specific error handling on the client side. Ther The application must store the global transaction identifier only for the purpose of verifying the transaction status in case of disconnection during `COMMIT`. In particular, the application doesn't need another persistence layer. If the application fails, it needs only the information in the database to restart. -### CAMO partner connection status +To illustrate this, we have an example of a retry loop in a CAMO-aware client application, written in a C-like pseudo-code. It expects two DSNs `origin_dsn` and `partner_dsn` providing connection information. These usually are the same DSNs as used for the initial call to `bdr.create_node` and can be looked up in `bdr.node_summary`, column `interface_connstr`. -The function `bdr.is_camo_partner_connected` allows checking the connection status of a CAMO partner node configured in pair mode. There currently is no equivalent for CAMO used with Eager Replication. - -#### Synopsis - -```sql -bdr.is_camo_partner_connected() ``` - -#### Return value - -A Boolean value indicating whether the CAMO partner is currently connected to a WAL sender process on the local node and therefore can receive transactional data and send back confirmations. - -### CAMO partner readiness - -The function `bdr.is_camo_partner_ready` allows checking the readiness status of a CAMO partner node configured in pair mode. Underneath, this triggers the switch to and from local mode. - -#### Synopsis - -```sql -bdr.is_camo_partner_ready() +PGconn *conn = PQconnectdb(origin_dsn); ``` -#### Return value - -A Boolean value indicating whether the CAMO partner can reasonably be expected to confirm transactions originating from the local node in a timely manner, that is, before `timeout` for `TO ASYNC` expires. - -!!! Note - This function queries the past or current state. A positive return value doesn't indicate whether the CAMO partner can confirm future transactions. - -### Fetch the CAMO partner +The process starts connecting to the origin node. We now enter our loop: -This function shows the local node's CAMO partner (configured by pair mode). - -```sql -bdr.get_configured_camo_partner() ``` - -### Wait for consumption of the apply queue from the CAMO partner - -The function `bdr.wait_for_camo_partner_queue` is a wrapper of `bdr.wait_for_apply_queue` defaulting to query the CAMO partner node. It returns an error if the local node isn't part of a CAMO pair. - -#### Synopsis - -```sql -bdr.wait_for_camo_partner_queue() +loop { + PQexec(conn, "BEGIN"); ``` -### Transaction status between CAMO nodes +We start our transaction and begin populating it with changes: -This function enables a wait for CAMO transactions to be fully resolved. - -```sql -bdr.camo_transactions_resolved() ``` - -### Transaction status query function - -To check the status of a transaction that was being committed when the node failed, the application must use this function: - -```sql -bdr.logical_transaction_status(node_id OID, xid OID, - require_camo_partner boolean) + PQexec(conn, "INSERT INTO ..."); + ... ``` -With CAMO used in pair mode, use this function only on a node that's part of a CAMO pair. Along with Eager -Replication, you can use it on all nodes. - -In both cases, you must call the function within 15 minutes after the commit was issued. The CAMO partner must regularly purge such meta-information and therefore can't provide correct answers for older transactions. - -Before querying the status of a transaction, this function waits for the receive queue to be consumed and fully applied. This mechanism prevents early negative answers for transactions that were received but not yet applied. - -Despite its name, it's not always a read-only operation. If the status is unknown, the CAMO partner decides whether to commit or abort the transaction, storing that decision locally to ensure consistency going forward. - -The client must not call this function before attempting to commit on the origin. Otherwise the transaction might be -forced to roll back. +Once we are done, we need to make a record of the local node id and the transaction id. Both are available as parameters. -#### Synopsis - -```sql -bdr.logical_transaction_status(node_id OID, xid OID, - require_camo_partner boolean DEFAULT true) ``` - -#### Parameters - -- `node_id` — The node id of the PGD node the transaction originates from, usually retrieved by the client before `COMMIT` from the PQ parameter `bdr.local_node_id`. -- `xid` — The transaction id on the origin node, usually retrieved by the client before `COMMIT` from the PQ parameter `transaction_id`. -- `require_camo_partner` — Defaults to true and enables configuration checks. Set to false to disable these checks and query the status of a transaction that was not a CAMO transaction. - -#### Return value - -The function returns one of these results: - -- `'committed'::TEXT` — The transaction was committed, is visible on both nodes of the CAMO pair, and is eventually replicated to all other PGD nodes. No need for the client to retry it. - -- `'aborted'::TEXT` — The transaction was aborted and isn't replicated to any other PGD node. The client needs to either retry it or escalate the failure to commit the transaction. - -- `'in progress'::TEXT` — The transaction is still in progress on this local node and wasn't committed or aborted yet. The transaction might be in the COMMIT phase, waiting for the CAMO partner to confirm or deny the commit. The recommended client reaction is to disconnect from the origin node and reconnect to the CAMO partner to query that instead. With a load balancer or proxy in between, where the client lacks control over which node gets queried, the client can only poll repeatedly until the status switches to either `'committed'` or `'aborted'`. - - For Eager All-Node Replication, peer nodes yield this result for transactions that aren't yet committed or aborted. Even transactions not yet replicated (or not even started on the origin node) might yield an `in progress` result on a peer PGD node in this case. However, the client must not query the transaction - status prior to attempting to commit on the origin. - -- `'unknown'::TEXT` — The transaction specified is unknown, because either it's in the future, not replicated to that specific node yet, or too far in the past. The status of such a transaction isn't yet or is no longer known. This return value is a sign of improper use by the client. - -The client must be prepared to retry the function call on error. - -### Connection pools and proxies - -Consider the effect of connection pools and proxies when designing a CAMO cluster. A proxy might freely distribute transactions to all nodes in the commit group, that is, to both nodes of a CAMO pair or to all PGD nodes in case of Eager All-Node Replication). - -Take care to ensure that the application fetches the proper node id. When using session pooling, the client remains connected to the same node, so the node id remains constant for the lifetime of the client session. However, with finer-grained transaction pooling, the client needs to fetch the node id for every transaction (as in the example that follows. - -A client that isn't directly connected to the PGD nodes might not even notice a failover or switchover. But it can always use the `bdr.local_node_id` parameter to determine which node it's currently connected to. In the crucial situation of a disconnect during COMMIT, the proxy must properly forward that disconnect as an error to the client applying the CAMO protocol. - -For CAMO in `received` mode, a proxy that potentially switches between the CAMO pairs must use the `bdr.wait_for_camo_partner_queue` function to prevent stale reads. - -### Example - -This example shows what a retry loop of a CAMO-aware client application looks like in C-like pseudo-code. It expects two DSNs `origin_dsn` and `partner_dsn` providing connection information. These usually are the same DSNs as used for the initial call to `bdr.create_node` and can be looked up in `bdr.node_summary`, column -`interface_connstr`. - -```shell -PGconn *conn = PQconnectdb(origin_dsn); - -loop { - // start a transaction - PQexec(conn, "BEGIN"); - - // apply transactional changes - PQexec(conn, "INSERT INTO ..."); - ... - - // store a globally unique transaction identifier node_id = PQparameterStatus(conn, "bdr.local_node_id"); xid = PQparameterStatus(conn, "transaction_id"); +``` - // attempt to commit +We are now ready to try and commit. + +``` PQexec(conn, "COMMIT"); if (PQresultStatus(res) == PGRES_COMMAND_OK) return SUCCESS; +``` + +If the result is `PGRES_COMMAND_OK`, we are good and ready to move on. But if it isn't, we need to use CAMO to track the transaction to completion. The first question we need to ask is was the connection bad? + +``` else if (PQstatus(res) == CONNECTION_BAD) { - // Re-connect to the partner +``` + +If it was a bad connection, then we can check on the CAMO partner node to see if the transaction made it there. + +``` conn = PQconnectdb(partner_dsn); - // Check if successfully reconnected if (!connectionEstablished()) panic(); +``` + +If we can't connect to the partner node, there's not a lot we can do. So we should panic, or take similar actions. - // Check the attempted transaction's status +But if we can connect, we can use [`bdr.logical_transaction_status()`](/pgd/latest/reference/dc-functions#bdrlogical_transaction_status) to find out how the transaction did. We recorded the required values, node_id and xid - the transaction id, just before we committed the transaction. + +``` sql = "SELECT bdr.logical_transaction_status($node_id, $xid)"; txn_status = PQexec(conn, sql); if (txn_status == "committed") @@ -267,18 +170,17 @@ loop { else continue; // to retry the transaction on the partner } +``` + +If the transaction reports it has been committed, then we can call this transaction a success. No more action is required. If, on the other hand, it does not report it has been committed, we should continue in the loop so the transaction can be retried on the partner node. + +``` else { - // The connection is intact, but the transaction failed for some - // other reason. Differentiate between permanent and temporary - // errors. if (isPermanentError()) return FAILURE; else { - // Determine an appropriate delay to back-off to account for - // temporary failures due to congestion, so as to decrease - // the overall load put on the servers. sleep(increasing_retry_delay); continue; @@ -287,30 +189,51 @@ loop { } ``` -This example needs to be extended with proper logic for connecting, including retries and error handling. If using a load balancer like PgBouncer, you can implement reconnecting by using `PQreset`. Ensure that the load balancer only ever redirects a client to a CAMO partner and not any other PGD node. +If status of the transaction wasn't success or connection was bad, we check if the problem was a permanent error; if so, we should report a failure of the transaction. If not, it is still able to be retried and the code should sleep for a period of time that increases with each retry, and then retry the transaction. + + +### Working with the CAMO partner + +The function [`bdr.is_camo_partner_connected()`](/pgd/latest/reference/functions#bdris_camo_partner_connected) allows checking the connection status of a CAMO partner node configured in pair mode. There currently is no equivalent for CAMO used with eager replication. -In practice, we recommend an upper limit of retries. Depending on the actions performed in the transaction, other temporary errors might be possible. They need to be handled by retrying the transaction depending on the error code, similarly to the best practices on deadlocks or on serialization failures while in `SERIALIZABLE` isolation mode. +To check that the CAMO partner is ready, use the function [`bdr.is_camo_partner_ready`](/pgd/latest/reference/functions#bdris_camo_partner_ready). Underneath, this triggers the switch to and from local mode. -## Interaction with DDL and global locks +To find out more about the configured CAMO partner, use [`bdr.get_configured_camo_partner()`](/pgd/latest/reference/functions#bdrget_configured_camo_partner). This will returns the local node's CAMO partner. -Transactions protected by CAMO can contain DDL operations. However, DDL uses global locks, which already provide some synchronization among nodes. See [DDL locking details](../ddl#ddl-locking-details) for more information. +You can wait on the camo partner to process the queue with the function +[`bdr.wait_for_camo_partner_queue()`](/pgd/latest/reference/functions#bdrwait_for_camo_partner_queue). +This function is a wrapper of +[`bdr.wait_for_apply_queue`](/pgd/latest/reference/functions#bdrwait_for_apply_queue), +the difference being that +[`bdr.wait_for_camo_partner_queue()`](/pgd/latest/reference/functions#bdrwait_for_camo_partner_queue) +defaults to querying the CAMO partner node. It returns an error if the local +node isn't part of a CAMO pair. -Combining CAMO with DDL imposes a higher latency and also increases the chance of global deadlocks. We therefore recommend using a relatively low `bdr.global_lock_timeout`, which aborts the DDL and therefore resolves a deadlock in a reasonable amount of time. +To check the status of a transaction that was being committed when the node +failed, the application must use the function +[`bdr.logical_transaction_status`](/pgd/latest/reference/functions#bdrlogical_transaction_status). -### Nontransactional DDL +You pass this function the the node_id and transaction_id of the transaction you want to check +on. With CAMO used in pair mode, you can only use this function on a node that's part of a CAMO pair. Along with Eager Replication, you can use it on all nodes. -The following DDL operations aren't allowed in a transaction block and therefore can't benefit from CAMO protection. For these, CAMO is automatically disabled internally: +In all cases, you must call the function within 15 minutes after the commit was issued. The CAMO partner must regularly purge such meta-information and therefore can't provide correct answers for older transactions. -* All concurrent index operations (`CREATE`, `DROP`, and `REINDEX`) -* `REINDEX DATABASE`, `REINDEX SCHEMA`, and `REINDEX SYSTEM` -* `VACUUM` -* `CLUSTER` without any parameter -* `ALTER TABLE DETACH PARTITION CONCURRENTLY` -* `ALTER TYPE [enum] ADD VALUE` -* `ALTER SYSTEM` -* `CREATE` and `DROP DATABASE` -* `CREATE` and `DROP TABLESPACE` -* `ALTER DATABASE [db] TABLESPACE` +Before querying the status of a transaction, this function waits for the receive queue to be consumed and fully applied. This mechanism prevents early negative answers for transactions that were received but not yet applied. + +Despite its name, it's not always a read-only operation. If the status is unknown, the CAMO partner decides whether to commit or abort the transaction, storing that decision locally to ensure consistency going forward. + +The client must not call this function before attempting to commit on the origin. Otherwise the transaction might be +forced to roll back. + +### Connection pools and proxies + +Consider the effect of connection pools and proxies when designing a CAMO cluster. A proxy might freely distribute transactions to all nodes in the commit group, that is, to both nodes of a CAMO pair or to all PGD nodes in case of Eager All-Node Replication). + +Take care to ensure that the application fetches the proper node id. When using session pooling, the client remains connected to the same node, so the node id remains constant for the lifetime of the client session. However, with finer-grained transaction pooling, the client needs to fetch the node id for every transaction (as in the example that follows. + +A client that isn't directly connected to the PGD nodes might not even notice a failover or switchover. But it can always use the `bdr.local_node_id` parameter to determine which node it's currently connected to. In the crucial situation of a disconnect during COMMIT, the proxy must properly forward that disconnect as an error to the client applying the CAMO protocol. + +For CAMO in `received` mode, a proxy that potentially switches between the CAMO pairs must use the `bdr.wait_for_camo_partner_queue` function to prevent stale reads. ## CAMO limitations diff --git a/product_docs/docs/pgd/5/durability/commit-scopes.mdx b/product_docs/docs/pgd/5/durability/commit-scopes.mdx index ed3b5f4fab8..5a515e49406 100644 --- a/product_docs/docs/pgd/5/durability/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scopes.mdx @@ -110,7 +110,7 @@ The grammar for commit scopes is composed as follows: ``` commit_scope: - confirmation [AND ...] + commit_scope_operation [AND ...] commit_scope_operation: commit_scope_group [ ON { received|replicated|durable|visible } ] commit_scope_kind @@ -144,51 +144,27 @@ commit_scope_kind: * `max_commit_delay` (interval) — Maximum delay that can be injected to commit to try to keep within the lag limits. !!! Note - `CAMO` commit scope kind is mostly syntactic sugar for `GROUP COMMIT (transaction_tracking = true, commit_decision = partner)` with additional `DEGRADE ON` clause. It's expected that `GROUP COMMIT` will eventually gain the `DEGRADE ON` clause as well, making `CAMO` syntax deprecated. +`CAMO` commit scope kind is mostly syntactic sugar for `GROUP COMMIT (transaction_tracking = true, commit_decision = partner)` with additional `DEGRADE ON` clause. It's expected that `GROUP COMMIT` will eventually gain the `DEGRADE ON` clause as well, making `CAMO` syntax deprecated. +!!! !!! Note - While the grammar for `synchronous_standby_names` and commit scopes can look similar, the former doesn't account for the origin node, but the latter does. Therefore, for example `synchronous_standby_names = 'ANY 1 (..)'` is equivalent to a commit scope of `ANY 2 (...)`. This choice makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction. +While the grammar for `synchronous_standby_names` and commit scopes can look similar, the former doesn't account for the origin node, but the latter does. Therefore, for example `synchronous_standby_names = 'ANY 1 (..)'` is equivalent to a commit scope of `ANY 2 (...)`. This choice makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction. +!!! ### Adding a commit scope rule -The function `bdr.add_commit_scope` creates a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the EDB Postgres Distributed cluster, invoking this function once for the top-level node group is enough to fully define the commit scope. +To add a commit scope rule, use the function [`bdr.add_commit_scope`](/pgd/latest/reference/functions#bdradd_commit_scope). This function will create a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the EDB Postgres Distributed cluster, invoking this function once for the top-level node group is enough to fully define the commit scope. Alternatively, you can invoke it multiple times with the same `commit_scope_name` but different origin node groups and rules for commit scopes that vary depending on the origin of the transaction. -#### Synopsis - -```sql -bdr.add_commit_scope( - commit_scope_name NAME, - origin_node_group NAME, - rule TEXT - wait_for_ready boolean DEFAULT true) -``` - ### Changing a commit scope rule -To change a specific rule for a single origin node group in a commit scope, you can use the function `bdr.alter_commit_scope`. - -#### Synopsis - -```sql -bdr.alter_commit_scope( - commit_scope_name NAME, - origin_node_group NAME, - rule TEXT) -``` +To change a specific rule for a single origin node group in a commit scope, you can use the function [`bdr.alter_commit_scope`](/pgd/latest/reference/functions#bdralter_commit_scope). ### Removing a commit scope rule -You can use `bdr.remove_commit_scope` to drop a single rule in a commit scope. If you define multiple rules for the commit scope, you must invoke this function once per rule to fully remove the entire commit scope. - -#### Synopsis - -```sql -bdr.remove_commit_scope( - commit_scope_name NAME, - origin_node_group NAME) -``` +To remove a commit scope rule, use [`bdr.remove_commit_scope`](/pgd/latest/reference/functions#bdrremove_commit_scope). This can drop a single rule in a commit scope. If you have defined multiple rules for the commit scope, you will have to invoke this function for each defined rule to remove the entire commit scope. !!! Note - Removing a commit scope that's still used as default by a node group isn't allowed. +Removing a commit scope that's still used as default by a node group isn't allowed. +!!! diff --git a/product_docs/docs/pgd/5/reference/functions.mdx b/product_docs/docs/pgd/5/reference/functions.mdx index 82d23c4cb08..cc7119c8423 100644 --- a/product_docs/docs/pgd/5/reference/functions.mdx +++ b/product_docs/docs/pgd/5/reference/functions.mdx @@ -973,3 +973,148 @@ bdr.lag_control() - `sample_interval` — Configured minimum time between lag samples and possible commit delay adjustments, in milliseconds. +## CAMO functions + +CAMO requires that a client actively participates in the committing of a transaction by following the transactions progress. The functions listed here are used for that purpose and explained in the [CAMO](../durability/camo) section. + +### `bdr.is_camo_partner_connected` + +Allows checking of the connection status of a CAMO partner node configured in pair mode. There currently is no equivalent for CAMO used with eager replication. + +#### Synopsis + +```sql +bdr.is_camo_partner_connected() +``` + +#### Return value + +A Boolean value indicating whether the CAMO partner is currently connected to a WAL sender process on the local node and therefore can receive transactional data and send back confirmations. + +### `bdr.is_camo_partner_ready` + +Allows checking of the readiness status of a CAMO partner node configured in pair mode. Underneath, this triggers the switch to and from local mode. + +#### Synopsis + +```sql +bdr.is_camo_partner_ready() +``` + +#### Return value + +A Boolean value indicating whether the CAMO partner can reasonably be expected to confirm transactions originating from the local node in a timely manner, that is, before `timeout` for `TO ASYNC` expires. + +!!! Note + This function queries the past or current state. A positive return value doesn't indicate whether the CAMO partner can confirm future transactions. + +### `bdr.get_configured_camo_partner` + +This function shows the local node's CAMO partner (configured by pair mode). + +#### Synopsis + +```sql +bdr.get_configured_camo_partner() +``` + +### `bdr.wait_for_camo_partner_queue` + +The function is a wrapper around `bdr.wait_for_apply_queue` defaulting to query the CAMO partner node. It returns an error if the local node isn't part of a CAMO pair. + +#### Synopsis + +```sql +bdr.wait_for_camo_partner_queue() +``` + +### `bdr.camo_transactions_resolved` + +This function begins a wait for CAMO transactions to be fully resolved. + +#### Synopsis + +```sql +bdr.camo_transactions_resolved() +``` + +### bdr.logical_transaction_status + +To check the status of a transaction that was being committed when the node failed, the application must use this function, passing as parameters the node id of the node the transaction originated from and the transaction id on the origin node. These values can be + + +#### Synopsis + +```sql +bdr.logical_transaction_status(node_id OID, xid OID, + require_camo_partner boolean DEFAULT true) +``` + +#### Parameters + +- `node_id` — The node id of the PGD node the transaction originates from, usually retrieved by the client before `COMMIT` from the [PQ parameter](https://www.postgresql.org/docs/current/libpq-status.html#LIBPQ-PQPARAMETERSTATUS) `bdr.local_node_id`. +- `xid` — The transaction id on the origin node, usually retrieved by the client before `COMMIT` from the [PQ parameter](https://www.postgresql.org/docs/current/libpq-status.html#LIBPQ-PQPARAMETERSTATUS) `transaction_id`. +- `require_camo_partner` — Defaults to true and enables configuration checks. Set to false to disable these checks and query the status of a transaction that was not a CAMO transaction. + +#### Return value + +The function returns one of these results: + +- `'committed'::TEXT` — The transaction was committed, is visible on both nodes of the CAMO pair, and is eventually replicated to all other PGD nodes. No need for the client to retry it. + +- `'aborted'::TEXT` — The transaction was aborted and isn't replicated to any other PGD node. The client needs to either retry it or escalate the failure to commit the transaction. + +- `'in progress'::TEXT` — The transaction is still in progress on this local node and wasn't committed or aborted yet. The transaction might be in the COMMIT phase, waiting for the CAMO partner to confirm or deny the commit. The recommended client reaction is to disconnect from the origin node and reconnect to the CAMO partner to query that instead. With a load balancer or proxy in between, where the client lacks control over which node gets queried, the client can only poll repeatedly until the status switches to either `'committed'` or `'aborted'`. + + For eager all-node replication, peer nodes yield this result for transactions that aren't yet committed or aborted. Even transactions not yet replicated (or not even started on the origin node) might yield an `in progress` result on a peer PGD node in this case. However, the client must not query the transaction + status prior to attempting to commit on the origin. + +- `'unknown'::TEXT` — The transaction specified is unknown, because either it's in the future, not replicated to that specific node yet, or too far in the past. The status of such a transaction isn't yet or is no longer known. This return value is a sign of improper use by the client. + +The client must be prepared to retry the function call on error. + +## Commit Scope functions + +### `bdr.add_commit_scope` + +`bdr.add_commit_scope` creates a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the EDB Postgres Distributed cluster, invoking this function once for the top-level node group is enough to fully define the commit scope. + +Alternatively, you can invoke it multiple times with the same `commit_scope_name` but different origin node groups and rules for commit scopes that vary depending on the origin of the transaction. + +#### Synopsis + +```sql +bdr.add_commit_scope( + commit_scope_name NAME, + origin_node_group NAME, + rule TEXT + wait_for_ready boolean DEFAULT true) +``` + +### `bdr.alter_commit_scope` + +`bdr.alter_commit_scope` allows you to change a specific rule for a single origin node group in a commit scope. + +#### Synopsis + +```sql +bdr.alter_commit_scope( + commit_scope_name NAME, + origin_node_group NAME, + rule TEXT) +``` + +### `bdr.remove_commit_scope` + +Drops a single rule in a commit scope. If you define multiple rules for the commit scope, you must invoke this function once per rule to fully remove the entire commit scope. + +#### Synopsis + +```sql +bdr.remove_commit_scope( + commit_scope_name NAME, + origin_node_group NAME) +``` + +!!! Note + Removing a commit scope that's still used as default by a node group isn't allowed. diff --git a/product_docs/docs/pgd/5/reference/index.json b/product_docs/docs/pgd/5/reference/index.json index b7c8120a5fa..699580d0243 100644 --- a/product_docs/docs/pgd/5/reference/index.json +++ b/product_docs/docs/pgd/5/reference/index.json @@ -280,4 +280,4 @@ "bdrshow_workers": "/pgd/latest/reference/functions-internal#bdrshow_workers", "bdrshow_writers": "/pgd/latest/reference/functions-internal#bdrshow_writers", "bdrnode_kind_name": "/pgd/latest/reference/functions-internal#bdrnode_kind_name" -} \ No newline at end of file +} diff --git a/product_docs/docs/pgd/5/reference/index.mdx b/product_docs/docs/pgd/5/reference/index.mdx index 1b3ceed0870..151ce52a544 100644 --- a/product_docs/docs/pgd/5/reference/index.mdx +++ b/product_docs/docs/pgd/5/reference/index.mdx @@ -134,6 +134,17 @@ The reference section is a definitive listing of all functions, views and comman * [`bdr.wal_sender_stats`](functions#bdrwal_sender_stats) * [`bdr.get_decoding_worker_stat`](functions#bdrget_decoding_worker_stat) * [`bdr.lag_control`](functions#bdrlag_control) +### [CAMO functions](functions#camo-functions) + * [`bdr.is_camo_partner_connected`](functions#bdris_camo_partner_connected) + * [`bdr.is_camo_partner_ready`](functions#bdris_camo_partner_ready) + * [`bdr.get_configured_camo_partner`](functions#bdrget_configured_camo_partner) + * [`bdr.wait_for_camo_partner_queue`](functions#bdrwait_for_camo_partner_queue) + * [`bdr.camo_transactions_resolved`](functions#bdrcamo_transactions_resolved) + * [bdr.logical_transaction_status](functions#bdrlogical_transaction_status) +### [Commit Scope functions](functions#commit-scope-functions) + * [`bdr.add_commit_scope`](functions#bdradd_commit_scope) + * [`bdr.alter_commit_scope`](functions#bdralter_commit_scope) + * [`bdr.remove_commit_scope`](functions#bdrremove_commit_scope) ## [PGD settings](pgd-settings) From 29e1090048870595f10ad47c7046b685b77d8880 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 3 Jul 2023 13:32:32 +0100 Subject: [PATCH 19/68] Text tweaks Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/index.mdx | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 54a038b5771..13a06de00c9 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -228,7 +228,6 @@ The `default_commit_scope` is checked in the group tree that the given origin node belongs to from bottom to top. The `default_commit_scope` can't be set to the special value `local`, which means there's no way for the commit scope to use the `bdr.commit_scope` configuration parameter. - For full details of the commit scope language with all the options described, see [Commit scopes](commit-scopes). @@ -291,11 +290,13 @@ instead of configuring `synchronous_standby_names` on each node individually, Group Commit uses globally synchronized commit scopes. !!! Note - While the grammar for `synchronous_standby_names` and commit - scopes looks similar, the former - doesn't account for the origin node, but the latter does. - Therefore, for example, `synchronous_standby_names = 'ANY 1 (..)'` - is equivalent to a commit scope of `ANY 2 (...)`. This choice - makes reasoning about majority easier and reflects that the origin - node also contributes to the durability and visibility of the - transaction. +While the grammar for `synchronous_standby_names` and commit +scopes looks similar, the former +doesn't account for the origin node, but the latter does. +Therefore, for example, `synchronous_standby_names = 'ANY 1 (..)'` +is equivalent to a commit scope of `ANY 2 (...)`. This choice +makes reasoning about majority easier and reflects that the origin +node also contributes to the durability and visibility of the +transaction. +!!! + From af192709325d43556ab4f09925b34ff23830b70c Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 4 Jul 2023 16:16:17 +0100 Subject: [PATCH 20/68] Fixed wrong link Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/camo.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/camo.mdx b/product_docs/docs/pgd/5/durability/camo.mdx index a786074477f..c2fe138f4c2 100644 --- a/product_docs/docs/pgd/5/durability/camo.mdx +++ b/product_docs/docs/pgd/5/durability/camo.mdx @@ -160,7 +160,7 @@ If it was a bad connection, then we can check on the CAMO partner node to see if If we can't connect to the partner node, there's not a lot we can do. So we should panic, or take similar actions. -But if we can connect, we can use [`bdr.logical_transaction_status()`](/pgd/latest/reference/dc-functions#bdrlogical_transaction_status) to find out how the transaction did. We recorded the required values, node_id and xid - the transaction id, just before we committed the transaction. +But if we can connect, we can use [`bdr.logical_transaction_status()`](/pgd/latest/reference/functions#bdrlogical_transaction_status) to find out how the transaction did. We recorded the required values, node_id and xid - the transaction id, just before we committed the transaction. ``` sql = "SELECT bdr.logical_transaction_status($node_id, $xid)"; From 659fce0f62546dbbf139bfdd9933a07fd3f2237e Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 4 Jul 2023 16:58:28 +0100 Subject: [PATCH 21/68] Case fixes Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/camo.mdx | 2 +- product_docs/docs/pgd/5/durability/group-commit.mdx | 2 +- product_docs/docs/pgd/5/durability/index.mdx | 3 ++- 3 files changed, 4 insertions(+), 3 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/camo.mdx b/product_docs/docs/pgd/5/durability/camo.mdx index c2fe138f4c2..c7caad6c071 100644 --- a/product_docs/docs/pgd/5/durability/camo.mdx +++ b/product_docs/docs/pgd/5/durability/camo.mdx @@ -194,7 +194,7 @@ If status of the transaction wasn't success or connection was bad, we check if t ### Working with the CAMO partner -The function [`bdr.is_camo_partner_connected()`](/pgd/latest/reference/functions#bdris_camo_partner_connected) allows checking the connection status of a CAMO partner node configured in pair mode. There currently is no equivalent for CAMO used with eager replication. +The function [`bdr.is_camo_partner_connected()`](/pgd/latest/reference/functions#bdris_camo_partner_connected) allows checking the connection status of a CAMO partner node configured in pair mode. There currently is no equivalent for CAMO used with Eager Replication. To check that the CAMO partner is ready, use the function [`bdr.is_camo_partner_ready`](/pgd/latest/reference/functions#bdris_camo_partner_ready). Underneath, this triggers the switch to and from local mode. diff --git a/product_docs/docs/pgd/5/durability/group-commit.mdx b/product_docs/docs/pgd/5/durability/group-commit.mdx index 95b06a58253..56a3683657b 100644 --- a/product_docs/docs/pgd/5/durability/group-commit.mdx +++ b/product_docs/docs/pgd/5/durability/group-commit.mdx @@ -85,7 +85,7 @@ You can combine only CAMO transactions with the `DEGRADE TO` clause for switchin to asynchronous operation in case of lowered availability. Eager and CAMO transactions aren't currently supported in combination -with the decoding worker feature or with transaction streaming. +with the Decoding Worker feature or with transaction streaming. Installations using Eager must keep `enable_wal_decoder` and `streaming_mode` disabled for the PGD node group. diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 13a06de00c9..c6813550121 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -261,7 +261,8 @@ required synchronization level and prevents loss of data. ## Legacy synchronous replication using PGD !!! Note - Consider using [Group Commit](group-commit) instead. +Consider using [Group Commit](group-commit) instead. +!!! ### Usage From 0af40649e78dd3d16946b2f028a0b10512442e46 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 4 Jul 2023 18:21:36 +0100 Subject: [PATCH 22/68] Extracted Legacy Sync options into their own page Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/index.mdx | 76 ++---------------- .../docs/pgd/5/durability/legacy-sync.mdx | 80 +++++++++++++++++++ 2 files changed, 88 insertions(+), 68 deletions(-) create mode 100644 product_docs/docs/pgd/5/durability/legacy-sync.mdx diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index c6813550121..7fc37bf1048 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -20,7 +20,9 @@ performance needs using *commit scopes*. In its basic configuration, EDB Postgres Distributed uses asynchronous replication. However, commit scopes can change both the default and the -per-transaction behavior. It's also possible to configure the legacy Postgres +per-transaction behavior. + +It's also possible to configure the legacy Postgres synchronous replication using standard `synchronous_standby_names` in the same way as the built-in physical or logical replication. However, commit scopes provide much more flexibility and control over the replication behavior. @@ -51,12 +53,11 @@ Commit scopes allow two ways of controlling durability of the transaction: - [Lag Control](lag-control). This option controls how far behind nodes can be in terms of replication before allowing commit to proceed. -Postgres provides [physical streaming replication](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION) -(PSR), which is unidirectional but offers a [synchronous variant](https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION). + For backward compatibility, PGD still supports configuring synchronous replication with `synchronous_commit` and `synchronous_standby_names`. See -[Legacy synchronous replication](#legacy-synchronous-replication-using-pgd), -but consider using [Group Commit](group-commit) instead. +[Legacy synchronous replication](legacy-sync) for more on this option, +but consider using [Group Commit](group-commit) instead. ## Terms and definitions @@ -85,24 +86,17 @@ outages. The following table summarizes what a client can expect from a peer node replicated to after receiving a COMMIT confirmation from -the origin node the transaction was issued to. The Mode column takes -on different meaning depending on the variant. For PSR and legacy -synchronous replication with PGD, it refers to the -`synchronous_commit` setting. For commit scopes, it refers to the -confirmation requirements of the +the origin node the transaction was issued to. For commit scopes the Mode +column refers to the confirmation requirements of the [commit scope configuration](commit-scopes#configuration). | Variant | Mode | Received | Visible | Durable | |---------------------|-----------------------|----------|---------|---------| -| PSR Async | off (default) | no | no | no | | PGD Async | off (default) | no | no | no | | PGD Lag Control | 'ON received' nodes | no | no | no | | PGD Lag Control | 'ON replicated' nodes | no | no | no | | PGD Lag Control | 'ON durable' nodes | no | no | no | | PGD Lag Control | 'ON visible' nodes | no | no | no | -| PSR Sync | remote_write (2) | yes | no | no (1) | -| PSR Sync | on (2) | yes | no | yes | -| PSR Sync | remote_apply (2) | yes | yes | yes | | PGD Group Commit | 'ON received' nodes | yes | no | no | | PGD Group Commit | 'ON replicated' nodes | yes | no | no | | PGD Group Commit | 'ON durable' nodes | yes | no | yes | @@ -111,18 +105,6 @@ confirmation requirements of the | PGD CAMO | 'ON replicated' nodes | yes | no | no | | PGD CAMO | 'ON durable' nodes | yes | no | yes | | PGD CAMO | 'ON visible' nodes | yes | yes | yes | -| PGD Legacy Sync (3) | remote_write (2) | yes | no | no | -| PGD Legacy Sync (3) | on (2) | yes | yes | yes | -| PGD Legacy Sync (3) | remote_apply (2) | yes | yes | yes | - -*(1) Written to the OS, durable if the OS remains running and only -Postgres crashes.* - -*(2) Unless switched to local mode (if allowed) by setting -`synchronous_replication_availability` to `async'`, otherwise the -values for the asynchronous PGD default apply.* - -*(3) Consider using Group Commit instead.* Reception ensures the peer operating normally can eventually apply the transaction without requiring any further @@ -258,46 +240,4 @@ To ensure the apply queue gets flushed to disk, use either `smart` or `fast` shutdown for maintenance tasks. This approach maintains the required synchronization level and prevents loss of data. -## Legacy synchronous replication using PGD - -!!! Note -Consider using [Group Commit](group-commit) instead. -!!! - -### Usage - -To enable synchronous replication using PGD, you need to add the application -name of the relevant PGD peer nodes to -`synchronous_standby_names`. The use of `FIRST x` or `ANY x` offers -some flexibility if this doesn't conflict with the requirements of -non-PGD standby nodes. - -Once you've added it, you can configure the level of synchronization per -transaction using `synchronous_commit`, which defaults to `on`. This setting means that -adding the application name to to `synchronous_standby_names` already enables synchronous -replication. Setting `synchronous_commit` to `local` or `off` turns -off synchronous replication. - -Due to PGD applying the transaction before persisting it, the -values `on` and `remote_apply` are equivalent for logical -replication. - -### Migration to commit scopes - -You configure the Group Commit feature of PGD independent of -`synchronous_commit` and `synchronous_standby_names`. Instead, the -`bdr.commit_scope` GUC allows you to select the scope per transaction. And -instead of configuring `synchronous_standby_names` on each node -individually, Group Commit uses globally synchronized commit scopes. - -!!! Note -While the grammar for `synchronous_standby_names` and commit -scopes looks similar, the former -doesn't account for the origin node, but the latter does. -Therefore, for example, `synchronous_standby_names = 'ANY 1 (..)'` -is equivalent to a commit scope of `ANY 2 (...)`. This choice -makes reasoning about majority easier and reflects that the origin -node also contributes to the durability and visibility of the -transaction. -!!! diff --git a/product_docs/docs/pgd/5/durability/legacy-sync.mdx b/product_docs/docs/pgd/5/durability/legacy-sync.mdx new file mode 100644 index 00000000000..f4eb2911012 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/legacy-sync.mdx @@ -0,0 +1,80 @@ +--- +title: Legacy synchronous replication using PGD +navTitle: Legacy synchronous replication +--- + +!!! Warning Important +Consider using [Group Commit](group-commit) instead. +!!! + +Postgres provides [physical streaming +replication](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION) +(PSR), which is unidirectional but offers a [synchronous +variant](https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION). + +For backward compatibility, PGD still supports configuring synchronous +replication with `synchronous_commit` and `synchronous_standby_names`. See +[Legacy synchronous replication](#legacy-synchronous-replication-using-pgd), but +consider using [Group Commit](group-commit) instead. + +### Usage + +To enable synchronous replication using PGD, you need to add the application +name of the relevant PGD peer nodes to `synchronous_standby_names`. The use of +`FIRST x` or `ANY x` offers some flexibility if this doesn't conflict with the +requirements of non-PGD standby nodes. + +Once you've added it, you can configure the level of synchronization per +transaction using `synchronous_commit`, which defaults to `on`. This setting +means that adding the application name to to `synchronous_standby_names` already +enables synchronous replication. Setting `synchronous_commit` to `local` or +`off` turns off synchronous replication. + +Due to PGD applying the transaction before persisting it, the values `on` and +`remote_apply` are equivalent for logical replication. + +### Comparison + +The following table summarizes what a client can expect from a peer node +replicated to after receiving a COMMIT confirmation from the origin node the +transaction was issued to. The Mode column takes on different meaning depending +on the variant. For PSR and legacy synchronous replication with PGD, it refers +to the `synchronous_commit` setting. + +| Variant | Mode | Received | Visible | Durable | +|---------------------|-----------------------|----------|---------|---------| +| PSR Async | off (default) | no | no | no | +| PSR Sync | remote_write (2) | yes | no | no (3) | +| PSR Sync | on (2) | yes | no | yes | +| PSR Sync | remote_apply (2) | yes | yes | yes | +| PGD Legacy Sync (1) | remote_write (2) | yes | no | no | +| PGD Legacy Sync (1) | on (2) | yes | yes | yes | +| PGD Legacy Sync (1) | remote_apply (2) | yes | yes | yes | + +*(1) Consider using Group Commit instead.* + +*(2) Unless switched to local mode (if allowed) by setting +`synchronous_replication_availability` to `async'`, otherwise the +values for the asynchronous PGD default apply.* + +*(3) Written to the OS, durable if the OS remains running and only +Postgres crashes.* + +### Migration to commit scopes + +You configure the Group Commit feature of PGD independent of +`synchronous_commit` and `synchronous_standby_names`. Instead, the +`bdr.commit_scope` GUC allows you to select the scope per transaction. And +instead of configuring `synchronous_standby_names` on each node +individually, Group Commit uses globally synchronized commit scopes. + +!!! Note +While the grammar for `synchronous_standby_names` and commit +scopes looks similar, the former +doesn't account for the origin node, but the latter does. +Therefore, for example, `synchronous_standby_names = 'ANY 1 (..)'` +is equivalent to a commit scope of `ANY 2 (...)`. This choice +makes reasoning about majority easier and reflects that the origin +node also contributes to the durability and visibility of the +transaction. +!!! From 94cfff024fdddb1b0e70d8b4936a610b625a1b6e Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 5 Jul 2023 13:42:25 +0100 Subject: [PATCH 23/68] Resolving comments Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/lag-control.mdx | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/lag-control.mdx b/product_docs/docs/pgd/5/durability/lag-control.mdx index 73301d424d8..db4242e68ff 100644 --- a/product_docs/docs/pgd/5/durability/lag-control.mdx +++ b/product_docs/docs/pgd/5/durability/lag-control.mdx @@ -11,10 +11,10 @@ time or chronically in high availability applications, then organizational objectives related to disaster recovery or business continuity plans might not be satisfied. -The replication Lag Control (RLC) feature is designed to regulate this -imbalance using a dynamic rate-limiting device so that data flow between -PGD group nodes complies with these organizational objectives. It does so -by controlling the extent of replication lag between PGD nodes. +The Lag Control feature is designed to regulate this imbalance using a dynamic +rate-limiting device so that data flow between PGD group nodes complies with +these organizational objectives. It does so by controlling the extent of +replication lag between PGD nodes. Some of these objectives include the following: From 8a6dbb20f515f881df61bfc8fc52ebe6ce6d1270 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 10 Jul 2023 09:59:33 +0100 Subject: [PATCH 24/68] Typos and formatting Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/lag-control.mdx | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/lag-control.mdx b/product_docs/docs/pgd/5/durability/lag-control.mdx index db4242e68ff..06648116b6f 100644 --- a/product_docs/docs/pgd/5/durability/lag-control.mdx +++ b/product_docs/docs/pgd/5/durability/lag-control.mdx @@ -104,9 +104,9 @@ PGD nodes can replicate changes fast enough to remain below configured lag measure thresholds, then the PGD runtime commit delay stays fixed at 0 milliseconds. -If this isn't the case, minimally -adjust the PGD runtime commit delay as high as needed, but no higher, until the number of -conforming nodes returns to the minimum threshold. +If this isn't the case, minimally adjust the PGD runtime commit delay as high as +needed, but no higher, until the number of conforming nodes returns to the +minimum threshold. Even after the minimum node threshold is reached, Lag Control continues to attempt to drive the PGD runtime commit delay back to zero. The PGD commit delay @@ -131,10 +131,10 @@ OS schedulers, clock interrupts, and variation due to system load. It also allows the PGD runtime commit delay to settle within microseconds of the lowest duration possible to maintain a lag measure threshold. -!!! Note -Don't conflate the PGD commit delay with the Postgres -commit delay. They are unrelated and perform different functions. Don't -substitute one for the other. +!!! Note +Don't conflate the PGD commit delay with the Postgres commit delay. +They are unrelated and perform different functions. Don't substitute one for the +other. !!! ## Transaction application From f6b787564c6b048467dc15fa86907561a0059500 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 10 Jul 2023 10:31:13 +0100 Subject: [PATCH 25/68] More frontmatter dupes Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/camo.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/camo.mdx b/product_docs/docs/pgd/5/durability/camo.mdx index c7caad6c071..b48d0055fff 100644 --- a/product_docs/docs/pgd/5/durability/camo.mdx +++ b/product_docs/docs/pgd/5/durability/camo.mdx @@ -1,6 +1,6 @@ --- -navTitle: Commit At Most Once title: Commit At Most Once +navTitle: Commit At Most Once redirects: - /pgd/latest/bdr/camo/ --- From 604d9d89125009cfd8dbaa0be2a89a6b494a821a Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 11 Jul 2023 11:11:00 +0100 Subject: [PATCH 26/68] Move tables into legacy sync Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/index.mdx | 16 ++-------------- .../docs/pgd/5/durability/legacy-sync.mdx | 16 ++++++++++++++++ 2 files changed, 18 insertions(+), 14 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 7fc37bf1048..da69e0dbdfc 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -54,10 +54,12 @@ Commit scopes allow two ways of controlling durability of the transaction: be in terms of replication before allowing commit to proceed. +!!! Note Legacy synchronization availability For backward compatibility, PGD still supports configuring synchronous replication with `synchronous_commit` and `synchronous_standby_names`. See [Legacy synchronous replication](legacy-sync) for more on this option, but consider using [Group Commit](group-commit) instead. +!!! ## Terms and definitions @@ -163,7 +165,6 @@ The following table summarizes the differences. | Variant | Order of apply vs persist | Replication before or after commit | |:-----------------|:-------------------------:|:----------------------------------:| -| PSR | persist first | after WAL flush of commit record | | PGD Async | apply first | after WAL flush of commit record | | PGD Lag Control | apply first | after WAL flush of commit record | | PGD Group Commit | apply first | before COMMIT on origin | @@ -214,19 +215,6 @@ the `bdr.commit_scope` configuration parameter. For full details of the commit scope language with all the options described, see [Commit scopes](commit-scopes). -### Postgres configuration parameters - -The following table provides an overview of the configuration -settings that you must set to a non-default value (req) and those that are -optional (opt) but affecting a specific variant. - -| Setting (GUC) | Group Commit | Lag Control | PSR (1) | Legacy Sync | -|--------------------------------------|:------------:|:-----------:|:-------:|:-----------:| -| synchronous_standby_names | n/a | n/a | req | req | -| synchronous_commit | n/a | n/a | opt | opt | -| synchronous_replication_availability | n/a | n/a | opt | opt | -| bdr.commit_scope | opt | opt | n/a | n/a | - ## Planned shutdown and restarts When using Group Commit with receive confirmations, take care diff --git a/product_docs/docs/pgd/5/durability/legacy-sync.mdx b/product_docs/docs/pgd/5/durability/legacy-sync.mdx index f4eb2911012..0f3e707ddcf 100644 --- a/product_docs/docs/pgd/5/durability/legacy-sync.mdx +++ b/product_docs/docs/pgd/5/durability/legacy-sync.mdx @@ -17,6 +17,10 @@ replication with `synchronous_commit` and `synchronous_standby_names`. See [Legacy synchronous replication](#legacy-synchronous-replication-using-pgd), but consider using [Group Commit](group-commit) instead. +Unlike PGD replication options, PSR sync will persist first, replicating after the WAL flush of commit record. + + + ### Usage To enable synchronous replication using PGD, you need to add the application @@ -60,6 +64,18 @@ values for the asynchronous PGD default apply.* *(3) Written to the OS, durable if the OS remains running and only Postgres crashes.* +### Postgres configuration parameters + +The following table provides an overview of the configuration +settings that you must set to a non-default value (req) and those that are +optional (opt) but affecting a specific variant. + +| Setting (GUC) | Group Commit | Lag Control | PSR | Legacy Sync | +|--------------------------------------|:------------:|:-----------:|:-------:|:-----------:| +| synchronous_standby_names | n/a | n/a | req | req | +| synchronous_commit | n/a | n/a | opt | opt | +| synchronous_replication_availability | n/a | n/a | opt | opt | + ### Migration to commit scopes You configure the Group Commit feature of PGD independent of From e62a95e4c4b4146cde6a54025792b5b80f57bdbf Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 23 Aug 2023 14:45:18 +0100 Subject: [PATCH 27/68] First Split Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/administering.mdx | 22 ++ .../docs/pgd/5/durability/comparing.mdx | 54 +++++ .../docs/pgd/5/durability/configuring.mdx | 49 ++++ .../docs/pgd/5/durability/group-commit.mdx | 1 + product_docs/docs/pgd/5/durability/index.mdx | 216 ++++-------------- .../docs/pgd/5/durability/overview.mdx | 71 ++++++ 6 files changed, 239 insertions(+), 174 deletions(-) create mode 100644 product_docs/docs/pgd/5/durability/administering.mdx create mode 100644 product_docs/docs/pgd/5/durability/comparing.mdx create mode 100644 product_docs/docs/pgd/5/durability/configuring.mdx create mode 100644 product_docs/docs/pgd/5/durability/overview.mdx diff --git a/product_docs/docs/pgd/5/durability/administering.mdx b/product_docs/docs/pgd/5/durability/administering.mdx new file mode 100644 index 00000000000..a3cf0a4fc58 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/administering.mdx @@ -0,0 +1,22 @@ +--- +title: Administering +navTitle: Administering +--- + +When running a PGD cluster with Group Commit, there are some things you need to +be aware of when administering the system, such as how to safely shut down and +restart nodes. + +## Planned shutdown and restarts + +When using Group Commit with receive confirmations, take care +with planned shutdown or restart. By default, the apply queue is consumed +prior to shutting down. However, in the `immediate` shutdown mode, the queue +is discarded at shutdown, leading to the stopped node "forgetting" +transactions in the queue. A concurrent failure of the origin node can +lead to loss of data, as if both nodes failed. + +To ensure the apply queue gets flushed to disk, use either +`smart` or `fast` shutdown for maintenance tasks. This approach maintains the +required synchronization level and prevents loss of data. + diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx new file mode 100644 index 00000000000..d47ef831085 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -0,0 +1,54 @@ +--- +title: Comparing Durability Options +navTitle: Comparing +--- + +## Comparison + +Most options for synchronous replication available to +PGD allow for different levels of synchronization, offering different +tradeoffs between performance and protection against node or network +outages. + +The following table summarizes what a client can expect from a peer +node replicated to after receiving a COMMIT confirmation from +the origin node the transaction was issued to. For commit scopes the Mode +column refers to the confirmation requirements of the +[commit scope configuration](commit-scopes#configuration). + +| Variant | Mode | Received | Visible | Durable | +|---------------------|-----------------------|----------|---------|---------| +| PGD Async | off (default) | no | no | no | +| PGD Lag Control | 'ON received' nodes | no | no | no | +| PGD Lag Control | 'ON replicated' nodes | no | no | no | +| PGD Lag Control | 'ON durable' nodes | no | no | no | +| PGD Lag Control | 'ON visible' nodes | no | no | no | +| PGD Group Commit | 'ON received' nodes | yes | no | no | +| PGD Group Commit | 'ON replicated' nodes | yes | no | no | +| PGD Group Commit | 'ON durable' nodes | yes | no | yes | +| PGD Group Commit | 'ON visible' nodes | yes | yes | yes | +| PGD CAMO | 'ON received' nodes | yes | no | no | +| PGD CAMO | 'ON replicated' nodes | yes | no | no | +| PGD CAMO | 'ON durable' nodes | yes | no | yes | +| PGD CAMO | 'ON visible' nodes | yes | yes | yes | + +Reception ensures the peer operating normally can +eventually apply the transaction without requiring any further +communication, even in the face of a full or partial network +outage. A crash of a peer node might still require retransmission of +the transaction, as this confirmation doesn't involve persistent +storage. All modes considered synchronous provide this protection. + +Visibility implies the transaction was applied remotely. All other +clients see the results of the transaction on all nodes, providing +this guarantee immediately after the commit is confirmed by the origin +node. Without visibility, other clients connected might not see the +results of the transaction and experience stale reads. + +Durability relates to the peer node's storage and provides protection +against loss of data after a crash and recovery of the peer node. +This can either relate to the reception of the data (as with physical +streaming replication) or to visibility (as with Group Commit). +The former eliminates the need for retransmissions after +a crash, while the latter ensures visibility is maintained across +restarts. \ No newline at end of file diff --git a/product_docs/docs/pgd/5/durability/configuring.mdx b/product_docs/docs/pgd/5/durability/configuring.mdx new file mode 100644 index 00000000000..276efde6bd2 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/configuring.mdx @@ -0,0 +1,49 @@ +--- +title: Configuring Commit Scopes +navTitle: Configuring +--- + +## Configuration + +You configure commit scopes using an SQL function just like other administration +operations in PGD. + +For example, you might define a basic commit scope that does Group Commit on a majority +of nodes in the example_group PGD group: + +```sql +SELECT bdr.add_commit_scope( + commit_scope_name := 'example_scope', + origin_node_group := 'example_group', + rule := 'ANY MAJORITY (example_group) GROUP COMMIT', + wait_for_ready := true +); +``` + +You can then use the commit scope either by setting the configuration variable (GUC) +`bdr.commit_scope` either per transaction or globally to that commit scope: + +```sql +BEGIN; +SET LOCAL bdr.commit_scope = 'example_scope'; +... +COMMIT; +``` + +You can also set the default commit scope for a given PGD group: + +```sql +SELECT bdr.alter_node_group_option( + node_group_name := 'example_group', + config_key := 'default_commit_scope', + config_value := 'example_scope' +); +``` + +The `default_commit_scope` is checked in the group tree that the given origin +node belongs to from bottom to top. The `default_commit_scope` can't be set to +the special value `local`, which means there's no way for the commit scope to use +the `bdr.commit_scope` configuration parameter. + +For full details of the commit scope language with all the options described, +see [Commit scopes](commit-scopes). diff --git a/product_docs/docs/pgd/5/durability/group-commit.mdx b/product_docs/docs/pgd/5/durability/group-commit.mdx index 56a3683657b..185eb7eda82 100644 --- a/product_docs/docs/pgd/5/durability/group-commit.mdx +++ b/product_docs/docs/pgd/5/durability/group-commit.mdx @@ -2,6 +2,7 @@ title: Group Commit redirects: - /pgd/latest/bdr/group-commit/ +deepToC: true --- The goal of Group Commit is to protect against data loss diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index da69e0dbdfc..ddc4bc1c60a 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -2,132 +2,57 @@ title: Durability navigation: + - overview + - configuring + - comparing - commit-scopes - group-commit - camo - lag-control + - administering + - legacy-sync redirects: - /pgd/latest/bdr/durability/ - /pgd/latest/choosing_durability/ --- -## Overview - -EDB Postgres Distributed allows you to choose from several replication -configurations based on your durability, consistency, availability, and -performance needs using *commit scopes*. - -In its basic configuration, EDB Postgres Distributed uses asynchronous -replication. However, commit scopes can change both the default and the -per-transaction behavior. - -It's also possible to configure the legacy Postgres -synchronous replication using standard `synchronous_standby_names` in the same -way as the built-in physical or logical replication. However, commit scopes -provide much more flexibility and control over the replication behavior. - -The different synchronization settings affect three properties of interest -to applications that are related but can all be implemented individually: - -- Durability: Writing to multiple nodes increases crash resilience - and allows you to recover the data after a crash and restart. -- Visibility: With the commit confirmation to the client, the database - guarantees immediate visibility of the committed transaction on some - sets of nodes. -- Conflict handling: Conflicts can be handled optimistically - postcommit, with conflicts resolved when the transaction is replicated - based on commit timestamps. Or, they can be handled pessimistically - precommit. The client can rely on the transaction to eventually be - applied on all nodes without further conflicts or get an abort, directly - informing the client of an error. - -Commit scopes allow two ways of controlling durability of the transaction: - -- [Group Commit](group-commit). This option controls which and how many nodes - have to reach a consensus before the transaction is considered to be committable - and at what stage of replication it can be considered committed. This option also - allows you to control the visibility ordering of the transaction. -- [CAMO](camo). This option is a variant of Group Commit in which the client is part of the - consensus. -- [Lag Control](lag-control). This option controls how far behind nodes can - be in terms of replication before allowing commit to proceed. - - -!!! Note Legacy synchronization availability -For backward compatibility, PGD still supports configuring synchronous -replication with `synchronous_commit` and `synchronous_standby_names`. See -[Legacy synchronous replication](legacy-sync) for more on this option, -but consider using [Group Commit](group-commit) instead. -!!! - -## Terms and definitions - -PGD nodes take different roles during the replication of a transaction. -These are implicitly assigned per transaction and are unrelated even for -concurrent transactions. - -* The *origin* is the node that receives the transaction from the - client or application. It's the node processing the transaction - first, initiating replication to other PGD nodes and responding back - to the client with a confirmation or an error. - -* A *partner* node is a PGD node expected to confirm transactions - according to Group Commit requirements. - -* A *commit group* is the group of all PGD nodes involved in the - commit, that is, the origin and all of its partner nodes, which can be - just a few or all peer nodes. - -## Comparison - -Most options for synchronous replication available to -PGD allow for different levels of synchronization, offering different -tradeoffs between performance and protection against node or network -outages. - -The following table summarizes what a client can expect from a peer -node replicated to after receiving a COMMIT confirmation from -the origin node the transaction was issued to. For commit scopes the Mode -column refers to the confirmation requirements of the -[commit scope configuration](commit-scopes#configuration). - -| Variant | Mode | Received | Visible | Durable | -|---------------------|-----------------------|----------|---------|---------| -| PGD Async | off (default) | no | no | no | -| PGD Lag Control | 'ON received' nodes | no | no | no | -| PGD Lag Control | 'ON replicated' nodes | no | no | no | -| PGD Lag Control | 'ON durable' nodes | no | no | no | -| PGD Lag Control | 'ON visible' nodes | no | no | no | -| PGD Group Commit | 'ON received' nodes | yes | no | no | -| PGD Group Commit | 'ON replicated' nodes | yes | no | no | -| PGD Group Commit | 'ON durable' nodes | yes | no | yes | -| PGD Group Commit | 'ON visible' nodes | yes | yes | yes | -| PGD CAMO | 'ON received' nodes | yes | no | no | -| PGD CAMO | 'ON replicated' nodes | yes | no | no | -| PGD CAMO | 'ON durable' nodes | yes | no | yes | -| PGD CAMO | 'ON visible' nodes | yes | yes | yes | - -Reception ensures the peer operating normally can -eventually apply the transaction without requiring any further -communication, even in the face of a full or partial network -outage. A crash of a peer node might still require retransmission of -the transaction, as this confirmation doesn't involve persistent -storage. All modes considered synchronous provide this protection. - -Visibility implies the transaction was applied remotely. All other -clients see the results of the transaction on all nodes, providing -this guarantee immediately after the commit is confirmed by the origin -node. Without visibility, other clients connected might not see the -results of the transaction and experience stale reads. - -Durability relates to the peer node's storage and provides protection -against loss of data after a crash and recovery of the peer node. -This can either relate to the reception of the data (as with physical -streaming replication) or to visibility (as with Group Commit). -The former eliminates the need for retransmissions after -a crash, while the latter ensures visibility is maintained across -restarts. +EDB Postgres Distributed offers a range of synchronous modes to complement its +default asynchronous replication. These synchronous modes are configured through +commit scopes; rules that define how operations are handled and when the system +should consider a transaction committed. + +The [overview](overview) introduces these concepts and some of the essential +terminology which is used when discussing synchronous commits. + +[Comparing](comparing) compares how each option behaves. + +[Configuring](configuring) shows how you can use the PGD SQL interface to set up +commit scopes which manage the various synchronous commit options. + +[Commit Scopes](commit-scopes) is a more in-depth look at the syntax and structure +of commit scopes and how to define them for your needs. + +[Group Commit](group-commit) focuses on the Group Commit option, where you can +define a transaction as done when a group of nodes agrees its done. + +[CAMO](camo) focuses on the Commit At Most Once option, which allows applications +to participate in confirming a transaction has been done and in turn ensure that +their commits only happen at most once. + +[Lag Control](lag-control) looks at the commit scope mechanism which regulates how +far out of sync nodes may go when a database node goes out of service. + +[Administering](administering) addresses how a PGD cluster with Group Commit +in use should be managed. + +[Legacy Sync](legacy-sync) shows how traditional Postgres synchronous operations +can still be accessed under EDB Postgres Distributed. + + + + \ No newline at end of file diff --git a/product_docs/docs/pgd/5/durability/overview.mdx b/product_docs/docs/pgd/5/durability/overview.mdx new file mode 100644 index 00000000000..6904cd93bc2 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/overview.mdx @@ -0,0 +1,71 @@ +--- +title: An Overview of Durability options +navTitle: Overview +--- + +## Overview + +EDB Postgres Distributed allows you to choose from several replication +configurations based on your durability, consistency, availability, and +performance needs using *commit scopes*. + +In its basic configuration, EDB Postgres Distributed uses asynchronous +replication. However, commit scopes can change both the default and the +per-transaction behavior. + +It's also possible to configure the legacy Postgres +synchronous replication using standard `synchronous_standby_names` in the same +way as the built-in physical or logical replication. However, commit scopes +provide much more flexibility and control over the replication behavior. + +The different synchronization settings affect three properties of interest +to applications that are related but can all be implemented individually: + +- Durability: Writing to multiple nodes increases crash resilience + and allows you to recover the data after a crash and restart. +- Visibility: With the commit confirmation to the client, the database + guarantees immediate visibility of the committed transaction on some + sets of nodes. +- Conflict handling: Conflicts can be handled optimistically + postcommit, with conflicts resolved when the transaction is replicated + based on commit timestamps. Or, they can be handled pessimistically + precommit. The client can rely on the transaction to eventually be + applied on all nodes without further conflicts or get an abort, directly + informing the client of an error. + +Commit scopes allow two ways of controlling durability of the transaction: + +- [Group Commit](group-commit). This option controls which and how many nodes + have to reach a consensus before the transaction is considered to be committable + and at what stage of replication it can be considered committed. This option also + allows you to control the visibility ordering of the transaction. +- [CAMO](camo). This option is a variant of Group Commit in which the client is part of the + consensus. +- [Lag Control](lag-control). This option controls how far behind nodes can + be in terms of replication before allowing commit to proceed. + + +!!! Note Legacy synchronization availability +For backward compatibility, PGD still supports configuring synchronous +replication with `synchronous_commit` and `synchronous_standby_names`. See +[Legacy synchronous replication](legacy-sync) for more on this option, +but consider using [Group Commit](group-commit) instead. +!!! + +## Terms and definitions + +PGD nodes take different roles during the replication of a transaction. +These are implicitly assigned per transaction and are unrelated even for +concurrent transactions. + +* The *origin* is the node that receives the transaction from the + client or application. It's the node processing the transaction + first, initiating replication to other PGD nodes and responding back + to the client with a confirmation or an error. + +* A *partner* node is a PGD node expected to confirm transactions + according to Group Commit requirements. + +* A *commit group* is the group of all PGD nodes involved in the + commit, that is, the origin and all of its partner nodes, which can be + just a few or all peer nodes. From 105ba4d47ee64a1713969baf549aa8f516910b0b Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 29 Aug 2023 10:30:24 +0100 Subject: [PATCH 28/68] Brought out limitations Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/group-commit.mdx | 28 +---------------- product_docs/docs/pgd/5/durability/index.mdx | 4 +++ .../docs/pgd/5/durability/limitations.mdx | 31 +++++++++++++++++++ 3 files changed, 36 insertions(+), 27 deletions(-) create mode 100644 product_docs/docs/pgd/5/durability/limitations.mdx diff --git a/product_docs/docs/pgd/5/durability/group-commit.mdx b/product_docs/docs/pgd/5/durability/group-commit.mdx index 185eb7eda82..87c0fc5208c 100644 --- a/product_docs/docs/pgd/5/durability/group-commit.mdx +++ b/product_docs/docs/pgd/5/durability/group-commit.mdx @@ -76,30 +76,4 @@ transaction abort is requested. If the transaction is already decided to be committed at the time the abort request is sent, the transaction does eventually COMMIT even though the client might receive an abort message. -## Limitations - -Group Commit transactions can't yet execute DDL, -and they don't support explicit two-phase commit. Future releases might allow them. -However, the `TRUNCATE` command is allowed. - -You can combine only CAMO transactions with the `DEGRADE TO` clause for switching -to asynchronous operation in case of lowered availability. - -Eager and CAMO transactions aren't currently supported in combination -with the Decoding Worker feature or with transaction streaming. -Installations using Eager must keep `enable_wal_decoder` and `streaming_mode` -disabled for the PGD node group. - -Synchronous replication uses a mechanism for transaction confirmation -different from Group Commit. The two aren't compatible, and you must not use -them together. Therefore, whenever you Group Commit transactions, -make sure none of the PGD nodes are configured in -`synchronous_standby_names`. - -Currently, Raft commit decisions are extremely slow, producing very low TPS. We recommended -using them only with the `eager` conflict resolution setting -to get the Eager All-Node Replication behavior of PGD 4 and older. - -Combining different commit decision options in the same transaction isn't -supported. Combining different conflict resolution options in the same transaction also isn't -supported. +See also [Limitations](limitations). \ No newline at end of file diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index ddc4bc1c60a..6945cafce8a 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -11,6 +11,7 @@ navigation: - lag-control - administering - legacy-sync + - limitations redirects: - /pgd/latest/bdr/durability/ @@ -49,6 +50,9 @@ in use should be managed. [Legacy Sync](legacy-sync) shows how traditional Postgres synchronous operations can still be accessed under EDB Postgres Distributed. +[Limitations](limitations) lists the various combinations of durability options +which are not currently supported or not possible. Do refer to this before deciding +on a durability strategy. \ No newline at end of file diff --git a/product_docs/docs/pgd/5/durability/timing.mdx b/product_docs/docs/pgd/5/durability/timing.mdx new file mode 100644 index 00000000000..5c31ecef401 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/timing.mdx @@ -0,0 +1,48 @@ +--- +title: Internal Timing of Operations +--- + +For a better understanding of how the different modes work, it's helpful to know +that legacy physical streaming replication (PSR) and PGD apply transactions in +different ways. + +With Legacy PSR, the order of operations is: + +- Origin flushes a commit record to WAL, making the transaction + visible locally. +- Peer node receives changes and issues a write. +- Peer flushes the received changes to disk. +- Peer applies changes, making the transaction visible on the peer. + +Note that the change is written to the disk before applying the chnages. + +With PGD, by default and with Lag Control, the order of operations is different. +In these cases, the change becomes visible on the peer before the transaction is +flushed to the peer's disk: + +- Origin flushes a commit record to WAL, making the transaction + visible locally. +- Peer node receives changes into its apply queue in memory. +- Peer applies changes, making the transaction visible on the peer. +- Peer persists the transaction by flushing to disk. + +For PGD's Group Commit and CAMO, the origin node waits for a certain number of +confirmations prior to making the transaction visible locally. The order of +operations is: + +- Origin flushes a prepare or precommit record to WAL. +- Peer node receives changes into its apply queue in memory. +- Peer applies changes, making the transaction visible on the peer. +- Peer persists the transaction by flushing to disk. +- Origin commits and makes the transaction visible locally. + +The following table summarizes the differences. + +| Variant | Order of apply vs persist | Replication before or after commit | +|:-----------------|:-------------------------:|:----------------------------------:| +| PSR | persist first | after WAL flush of commit record | +| PGD Async | apply first | after WAL flush of commit record | +| PGD Lag Control | apply first | after WAL flush of commit record | +| PGD Group Commit | apply first | before COMMIT on origin | +| PGD CAMO | apply first | before COMMIT on origin | + From 7b5f731cf00895a525e0b71372c0df33f97b349b Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 Nov 2023 12:23:53 +0000 Subject: [PATCH 43/68] Various edits reworking the commit scopes Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/camo.mdx | 5 + .../docs/pgd/5/durability/comparing.mdx | 41 ++++--- .../docs/pgd/5/durability/configuring.mdx | 49 -------- .../docs/pgd/5/durability/group-commit.mdx | 5 + product_docs/docs/pgd/5/durability/index.mdx | 10 +- .../docs/pgd/5/durability/legacy-sync.mdx | 2 +- .../docs/pgd/5/durability/limitations.mdx | 116 +++++++++++++++--- product_docs/docs/pgd/5/index.mdx | 19 +-- product_docs/docs/pgd/5/limitations.mdx | 84 +++---------- .../docs/pgd/5/reference/commit-scopes.mdx | 8 +- .../docs/pgd/5/reference/functions.mdx | 2 +- product_docs/docs/pgd/5/reference/index.json | 2 +- product_docs/docs/pgd/5/reference/index.mdx | 2 +- product_docs/docs/pgd/5/twophase.mdx | 4 + 14 files changed, 161 insertions(+), 188 deletions(-) delete mode 100644 product_docs/docs/pgd/5/durability/configuring.mdx diff --git a/product_docs/docs/pgd/5/durability/camo.mdx b/product_docs/docs/pgd/5/durability/camo.mdx index 22dc1db4f63..618cd4a4972 100644 --- a/product_docs/docs/pgd/5/durability/camo.mdx +++ b/product_docs/docs/pgd/5/durability/camo.mdx @@ -40,6 +40,11 @@ To use CAMO, an application must issue an explicit `COMMIT` message as a separat Configuration of CAMO happens through [commit scopes](commit-scopes). +## Limitations + +See the CAMO section of the [Limitations](limitations#camo) section. + + ## Failure scenarios Different failure scenarios occur in different configurations. diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index 3aa0de8475b..69913b3fc80 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -11,26 +11,29 @@ tradeoffs between performance and protection against node or network outages. The table summarizes what a client can expect from a peer node that has -replicated a transaction after receiving a COMMIT confirmation from the origin node. -For commit scopes the Mode -column refers to the confirmation requirements of the -[commit scope configuration](commit-scopes#configuration). +replicated a transaction after receiving a COMMIT confirmation from the origin +node. For commit scopes the Mode column refers to the confirmation requirements +of the [commit scope configuration](commit-scopes#configuration). -| Variant | Mode | Received | Visible | Durable | -|---------------------|-----------------------|----------|---------|---------| -| PGD Async | off (default) | no | no | no | -| PGD Lag Control | 'ON received' nodes | no | no | no | -| PGD Lag Control | 'ON replicated' nodes | no | no | no | -| PGD Lag Control | 'ON durable' nodes | no | no | no | -| PGD Lag Control | 'ON visible' nodes | no | no | no | -| PGD Group Commit | 'ON received' nodes | yes | no | no | -| PGD Group Commit | 'ON replicated' nodes | yes | no | no | -| PGD Group Commit | 'ON durable' nodes | yes | no | yes | -| PGD Group Commit | 'ON visible' nodes | yes | yes | yes | -| PGD CAMO | 'ON received' nodes | yes | no | no | -| PGD CAMO | 'ON replicated' nodes | yes | no | no | -| PGD CAMO | 'ON durable' nodes | yes | no | yes | -| PGD CAMO | 'ON visible' nodes | yes | yes | yes | +| Variant | Mode | Received | Visible | Durable | +|------------------------|-----------------------|----------|---------|---------| +| PGD Async | off (default) | no | no | no | +| PGD Group Commit | 'ON received' nodes | yes | no | no | +| PGD Group Commit | 'ON replicated' nodes | yes | no | no | +| PGD Group Commit | 'ON durable' nodes | yes | no | yes | +| PGD Group Commit | 'ON visible' nodes | yes | yes | yes | +| PGD CAMO | 'ON received' nodes | yes | no | no | +| PGD CAMO | 'ON replicated' nodes | yes | no | no | +| PGD CAMO | 'ON durable' nodes | yes | no | yes | +| PGD CAMO | 'ON visible' nodes | yes | yes | yes | +| PGD Lag Control | 'ON received' nodes | no | no | no | +| PGD Lag Control | 'ON replicated' nodes | no | no | no | +| PGD Lag Control | 'ON durable' nodes | no | no | no | +| PGD Lag Control | 'ON visible' nodes | no | no | no | +| PGD Synchronous Commit | 'ON received' nodes | no | no | no | +| PGD Synchronous Commit | 'ON replicated' nodes | no | no | no | +| PGD Synchronous Commit | 'ON durable' nodes | no | no | no | +| PGD Synchronous Commit | 'ON visible' nodes | no | no | no | Reception ensures the peer operating normally can eventually apply the transaction without requiring any further diff --git a/product_docs/docs/pgd/5/durability/configuring.mdx b/product_docs/docs/pgd/5/durability/configuring.mdx deleted file mode 100644 index 276efde6bd2..00000000000 --- a/product_docs/docs/pgd/5/durability/configuring.mdx +++ /dev/null @@ -1,49 +0,0 @@ ---- -title: Configuring Commit Scopes -navTitle: Configuring ---- - -## Configuration - -You configure commit scopes using an SQL function just like other administration -operations in PGD. - -For example, you might define a basic commit scope that does Group Commit on a majority -of nodes in the example_group PGD group: - -```sql -SELECT bdr.add_commit_scope( - commit_scope_name := 'example_scope', - origin_node_group := 'example_group', - rule := 'ANY MAJORITY (example_group) GROUP COMMIT', - wait_for_ready := true -); -``` - -You can then use the commit scope either by setting the configuration variable (GUC) -`bdr.commit_scope` either per transaction or globally to that commit scope: - -```sql -BEGIN; -SET LOCAL bdr.commit_scope = 'example_scope'; -... -COMMIT; -``` - -You can also set the default commit scope for a given PGD group: - -```sql -SELECT bdr.alter_node_group_option( - node_group_name := 'example_group', - config_key := 'default_commit_scope', - config_value := 'example_scope' -); -``` - -The `default_commit_scope` is checked in the group tree that the given origin -node belongs to from bottom to top. The `default_commit_scope` can't be set to -the special value `local`, which means there's no way for the commit scope to use -the `bdr.commit_scope` configuration parameter. - -For full details of the commit scope language with all the options described, -see [Commit scopes](commit-scopes). diff --git a/product_docs/docs/pgd/5/durability/group-commit.mdx b/product_docs/docs/pgd/5/durability/group-commit.mdx index f658acfc1d6..605adbea10e 100644 --- a/product_docs/docs/pgd/5/durability/group-commit.mdx +++ b/product_docs/docs/pgd/5/durability/group-commit.mdx @@ -41,6 +41,11 @@ Transactions committed with Group Commit use two-phase commit underneath. Therefore, configure `max_prepared_transactions` high enough to handle all such transactions originating per node. +## Limitations + +See the Group Commit section of the [Limitations](limitations#group-commit) section. + + ## Configuration To use Group Commit, first define a [commit scope](commit-scopes). The commit scope diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 72febd2f312..d4f9f764c2b 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -4,8 +4,8 @@ title: Durability navigation: - overview - configuring - - comparing - commit-scopes + - comparing - '# Commit Scope kinds' - group-commit - camo @@ -31,14 +31,12 @@ should consider a transaction committed. * [Overview](overview) introduces these concepts and some of the essential terminology which is used when discussing synchronous commits. -* [Comparing](comparing) compares how each option behaves. - -* [Configuring](configuring) shows how you can use the PGD SQL interface to set up -commit scopes which manage the various synchronous commit options. - * [Commit Scopes](commit-scopes) is a more in-depth look at the syntax and structure of commit scopes and how to define them for your needs. +* [Comparing](comparing) compares how each option behaves. + + ## Commit scope kinds * [Group Commit](group-commit) focuses on the Group Commit option, where you can diff --git a/product_docs/docs/pgd/5/durability/legacy-sync.mdx b/product_docs/docs/pgd/5/durability/legacy-sync.mdx index 0f3e707ddcf..750b05056c4 100644 --- a/product_docs/docs/pgd/5/durability/legacy-sync.mdx +++ b/product_docs/docs/pgd/5/durability/legacy-sync.mdx @@ -4,7 +4,7 @@ navTitle: Legacy synchronous replication --- !!! Warning Important -Consider using [Group Commit](group-commit) instead. +Consider using [Group Commit](group-commit) or [Synchronous Commit](synchronous_commit) instead. !!! Postgres provides [physical streaming diff --git a/product_docs/docs/pgd/5/durability/limitations.mdx b/product_docs/docs/pgd/5/durability/limitations.mdx index f94caafef93..5603a0e53c7 100644 --- a/product_docs/docs/pgd/5/durability/limitations.mdx +++ b/product_docs/docs/pgd/5/durability/limitations.mdx @@ -4,28 +4,106 @@ title: Limitations The following limitations apply to the use of commit scopes and the various durability options they enable. +## General limitations -Group Commit transactions can't yet execute DDL, and they don't support [explicit -two-phase commit](../twophase). Future releases might allow them. However, the `TRUNCATE` -command is allowed. +- Replacing a node with its physical standby doesn't work for nodes that use + CAMO/Eager/Group Commit. We don't recommend combining physical standbys and EDB Postgres + Distributed, even if it's possible. -You can combine only CAMO transactions with the `DEGRADE TO` clause for -switching to asynchronous operation in case of lowered availability. +- [Legacy synchronous replication](legacy-sync) uses a mechanism for transaction confirmation + different from the one used by CAMO, Eager, and Group Commit. The two aren't + compatible, so don't use them together. Whenever you use Group Commit, CAMO + or Eager, make sure none of the PGD nodes are configured in + `synchronous_standby_names`. + +- Postgres two-phase commit (2PC) transactions (that is, [`PREPARE + TRANSACTION`](https://www.postgresql.org/docs/current/sql-prepare-transaction.html)) + can't be used with CAMO, Group Commit, or Eager because those + features use two-phase commit underneath. + +## Group Commit + +[Group Commit](group-commit) enables configurable synchronous commits over +nodes in a group. If you use this feature, take the following limitations into account: + +- Not all DDL can run when you use Group Commit. If you use unsupported DDL, a warning is logged, and the transactions commit scope is set to local. The only supported DDL operations are: + - Nonconcurrent `CREATE INDEX` + - Nonconcurrent `DROP INDEX` + - Nonconcurrent `REINDEX` of an individual table or index + - `CLUSTER` (of a single relation or index only) + - `ANALYZE` + - `TRUNCATE` + +- Explicit two-phase commit is not supported by Group Commit as it already uses two-phase commit. + +- Combining different commit decision options in the same transaction or + combining different conflict resolution options in the same transaction isn't + supported. + +- Currently, Raft commit decisions are extremely slow, producing very low TPS. + We recommended using them only with the `eager` conflict resolution setting + to get the Eager All-Node Replication behavior of PGD 4 and older. + +## Eager + +[Eager](../consistency/eager) is available through Group Commit. It avoids conflicts by eagerly aborting transactions that might clash. It's subject to the same limitations as Group Commit. + +- Eager doesn't allow the `NOTIFY` SQL command or the `pg_notify()` function. It + also don't allow `LISTEN` or `UNLISTEN`. -Group Commit, Eager and CAMO transactions aren't currently supported in -combination with the Decoding Worker feature or with transaction streaming. -Installations using these commit scopes must keep `enable_wal_decoder` and `streaming_mode` -disabled for the PGD node group. +## CAMO -[Legacy synchronous replication](legacy-sync) uses a mechanism for transaction -confirmation different from Group Commit. The two aren't compatible, and you -must not use them together. Therefore, whenever you Group Commit transactions, -make sure none of the PGD nodes are configured in `synchronous_standby_names`. +[Commit At Most Once](camo) (CAMO) is a feature that aims to prevent +applications committing more than once. If you use this feature, take +these limitations into account when planning: -Currently, Raft commit decisions are extremely slow, producing very low TPS. We -recommended using them only with the `eager` conflict resolution setting to get -the Eager All-Node Replication behavior of PGD 4 and older. +- CAMO is designed to query the results of a recently failed COMMIT on the +origin node. So, in case of disconnection, code the application to immediately +request the transaction status from the CAMO partner. Have as little delay as +possible after the failure before requesting the status. Applications must not +rely on CAMO decisions being stored for longer than 15 minutes. -Combining different commit decision options in the same transaction isn't -supported. Combining different conflict resolution options in the same -transaction also isn't supported. +- If the application forgets the global identifier assigned, for example, +as a result of a restart, there's no easy way to recover +it. Therefore, we recommend that applications wait for outstanding +transactions to end before shutting down. + +- For the client to apply proper checks, a transaction protected by CAMO +can't be a single statement with implicit transaction control. You also can't +use CAMO with a transaction-controlling procedure or +in a `DO` block that tries to start or end transactions. + +- CAMO resolves commit status but doesn't resolve pending +notifications on commit. CAMO doesn't +allow the `NOTIFY` SQL command or the `pg_notify()` function. +They also don't allow `LISTEN` or `UNLISTEN`. + +- When replaying changes, CAMO transactions might detect conflicts just +the same as other transactions. If timestamp-conflict detection is used, +the CAMO transaction uses the timestamp of the prepare-on-the-origin +node, which is before the transaction becomes visible on the origin +node itself. + +- CAMO isn't currently compatible with transaction streaming. +Be sure to disable transaction streaming when planning to use +CAMO. You can configure this option globally or in the PGD node group. See +[Transaction streaming configuration](../transaction-streaming#configuration). + +- CAMO isn't currently compatible with decoding worker. +Be sure to not enable decoding worker when planning to use +CAMO. You can configure this option in the PGD node group. See +[Decoding worker disabling](../node_management/decoding_worker#enabling). + +- Not all DDL can run when you use CAMO. If you use unsupported DDL, a warning is logged and the transactions commit scope is set to local only. The only supported DDL operations are: + - Nonconcurrent `CREATE INDEX` + - Nonconcurrent `DROP INDEX` + - Nonconcurrent `REINDEX` of an individual table or index + - `CLUSTER` (of a single relation or index only) + - `ANALYZE` + - `TRUNCATE` + + +- Explicit two-phase commit is not supported by CAMO as it already uses two-phase commit. + +- You can combine only CAMO transactions with the `DEGRADE TO` clause for +switching to asynchronous operation in case of lowered availability. diff --git a/product_docs/docs/pgd/5/index.mdx b/product_docs/docs/pgd/5/index.mdx index bf4c894d88f..953eb08d76e 100644 --- a/product_docs/docs/pgd/5/index.mdx +++ b/product_docs/docs/pgd/5/index.mdx @@ -7,10 +7,11 @@ redirects: navigation: - rel_notes - known_issues - - quickstart - "#Concepts" - terminology - overview + - "#Get Started" + - quickstart - "#Planning" - architectures - choosing_server @@ -68,19 +69,3 @@ Postgres 16 support is only available in EDB Postgres Distributed 5.3 and later !!! See the [compatibility matrix](/pgd/4/#compatibility-matrix) for previous versions - - diff --git a/product_docs/docs/pgd/5/limitations.mdx b/product_docs/docs/pgd/5/limitations.mdx index 08d283b6032..7003869a450 100644 --- a/product_docs/docs/pgd/5/limitations.mdx +++ b/product_docs/docs/pgd/5/limitations.mdx @@ -66,69 +66,18 @@ doing so incurs many immediate risks and current limitations: - CLI and OTEL integration (new with v5) assumes one database. -## CAMO - -[Commit At Most Once](durability/camo) (CAMO) is a feature that aims to prevent -applications committing more than once. If you use this feature, take -these limitations into account when planning: - -- CAMO is designed to query the results of a recently failed COMMIT on -the origin node. So, in case of disconnection, code the application -to immediately request the transaction status from the CAMO partner. -Have as little delay as possible after the failure before -requesting the status. Applications must not rely on CAMO decisions -being stored for longer than 15 minutes. - -- If the application forgets the global identifier assigned, for example, -as a result of a restart, there's no easy way to recover -it. Therefore, we recommend that applications wait for outstanding -transactions to end before shutting down. - -- For the client to apply proper checks, a transaction protected by CAMO -can't be a single statement with implicit transaction control. You also can't -use CAMO with a transaction-controlling procedure or -in a `DO` block that tries to start or end transactions. - -- CAMO resolves commit status but doesn't resolve pending -notifications on commit. CAMO and Eager Replication options don't -allow the `NOTIFY` SQL command or the `pg_notify()` function. -They also don't allow `LISTEN` or `UNLISTEN`. - -- When replaying changes, CAMO transactions might detect conflicts just -the same as other transactions. If timestamp-conflict detection is used, -the CAMO transaction uses the timestamp of the prepare-on-the-origin -node, which is before the transaction becomes visible on the origin -node itself. - -- CAMO isn't currently compatible with transaction streaming. -Be sure to disable transaction streaming when planning to use -CAMO. You can configure this option globally or in the PGD node group. See -[Transaction streaming configuration](transaction-streaming/#configuration). - -- Not all DDL can run when you use CAMO. If you use unsupported DDL, a warning is logged and the transactions commit scope is set to local only. The only supported DDL operations are: - - Nonconcurrent `CREATE INDEX` - - Nonconcurrent `DROP INDEX` - - Nonconcurrent `REINDEX` of an individual table or index - - `CLUSTER` (of a single relation or index only) - - `ANALYZE` - - `TRUNCATE` - -## Group Commit - -[Group Commit](durability/group-commit) enables configurable synchronous commits over -nodes in a group. If you use this feature, take the following limitations into account: - -- Not all DDL can run when you use Group Commit. If you use unsupported DDL, a warning is logged, and the transactions commit scope is set to local. The only supported DDL operations are: - - Nonconcurrent `CREATE INDEX` - - Nonconcurrent `DROP INDEX` - - Nonconcurrent `REINDEX` of an individual table or index - - `CLUSTER` (of a single relation or index only) - - `ANALYZE` - - `TRUNCATE` - -## Eager - -[Eager](consistency/eager) is available in Group Commit. It avoids conflicts by eagerly aborting transactions that might clash. It's subject to the same limitations as Group Commit. +## Durability options (Group Commit/CAMO) + +There are various limits on how the PGD durability options work; this covers +Group Commit and CAMO and how they operate with PGD features such as the WAL +decoder and transaction streaming. + +Also there limitations on interoperability with legacy synchronous replication, +interoperability with explicit two-phase commit and unsupported combinations +within commit scope rules. + +Consult the [Durability limitations](durability/limitations) section for a full +and current listing. ## Other limitations @@ -136,9 +85,6 @@ This noncomprehensive list includes other limitations that are expected and are by design. We don't expect to resolve them in the future. Consider these limitations when planning your deployment: -- Replacing a node with its physical standby doesn't work for nodes that use - CAMO/Eager/Group Commit. We don't recommend combining physical standbys and EDB Postgres - Distributed, even if it's possible. - A `galloc` sequence might skip some chunks if you create the sequence in a rolled back transaction and then create it again with the same name. Skipping chunks can @@ -149,8 +95,6 @@ Consider these limitations when planning your deployment: specify the starting value for the sequence as an argument to the `bdr.alter_sequence_set_kind()` function. -- Legacy synchronous replication uses a mechanism for transaction confirmation - different from the one used by CAMO, Eager, and Group Commit. The two aren't compatible, - so don't use them together. -- Postgres two-phase commit (2PC) transactions (that is, [`PREPARE TRANSACTION`](https://www.postgresql.org/docs/current/sql-prepare-transaction.html)) can't be used with CAMO, Group Commit, or Eager Replication because those features use two-phase commit underneath. + + diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index a2b84591c13..a9ec90be2f9 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -20,9 +20,9 @@ commit_scope: commit_scope_operation [AND ...] commit_scope_operation: - commit_scope_group commit_scope_visibility commit_scope_kind + commit_scope_group confirmation_level commit_scope_kind -commit_scope_visibility: +confirmation_level: [ ON { received|replicated|durable|visible } ] commit_scope_group: @@ -60,9 +60,9 @@ Example: `ALL left_dc` A transaction under this commit scope group will be considered commited if all of the nodes in the left_dc group confirm they have processed the transaction. -## Commit scope visibility +## Confirmation level -Visibility controls when a remote PGD node confirms that it has +The confirmation level sets when a remote PGD node confirms that it has reached a particular point in processing a transaction. ### ON received diff --git a/product_docs/docs/pgd/5/reference/functions.mdx b/product_docs/docs/pgd/5/reference/functions.mdx index cc7119c8423..7f65bf88f76 100644 --- a/product_docs/docs/pgd/5/reference/functions.mdx +++ b/product_docs/docs/pgd/5/reference/functions.mdx @@ -1087,7 +1087,7 @@ Alternatively, you can invoke it multiple times with the same `commit_scope_name bdr.add_commit_scope( commit_scope_name NAME, origin_node_group NAME, - rule TEXT + rule TEXT, wait_for_ready boolean DEFAULT true) ``` diff --git a/product_docs/docs/pgd/5/reference/index.json b/product_docs/docs/pgd/5/reference/index.json index baee31040b3..d28b86cfa20 100644 --- a/product_docs/docs/pgd/5/reference/index.json +++ b/product_docs/docs/pgd/5/reference/index.json @@ -294,7 +294,7 @@ "cs.any": "/pgd/latest/reference/commit-scopes#any", "cs.majority": "/pgd/latest/reference/commit-scopes#majority", "cs.all": "/pgd/latest/reference/commit-scopes#all", - "cs.commit-scope-visibility": "/pgd/latest/reference/commit-scopes#commit-scope-visibility", + "cs.confirmation-level": "/pgd/latest/reference/commit-scopes#confirmation-level", "cs.on-received": "/pgd/latest/reference/commit-scopes#on-received", "cs.on-replicated": "/pgd/latest/reference/commit-scopes#on-replicated", "cs.on-durable": "/pgd/latest/reference/commit-scopes#on-durable", diff --git a/product_docs/docs/pgd/5/reference/index.mdx b/product_docs/docs/pgd/5/reference/index.mdx index d0d9fde458f..df12a692546 100644 --- a/product_docs/docs/pgd/5/reference/index.mdx +++ b/product_docs/docs/pgd/5/reference/index.mdx @@ -408,7 +408,7 @@ The reference section is a definitive listing of all functions, views and comman * [ANY](commit-scopes#any) * [MAJORITY](commit-scopes#majority) * [ALL](commit-scopes#all) - * [Commit scope visibility](commit-scopes#commit-scope-visibility) + * [Confirmation level](commit-scopes#confirmation-level) * [ON received](commit-scopes#on-received) * [ON replicated](commit-scopes#on-replicated) * [ON durable](commit-scopes#on-durable) diff --git a/product_docs/docs/pgd/5/twophase.mdx b/product_docs/docs/pgd/5/twophase.mdx index 3e5f084e6d9..6d43066fdc6 100644 --- a/product_docs/docs/pgd/5/twophase.mdx +++ b/product_docs/docs/pgd/5/twophase.mdx @@ -6,6 +6,10 @@ redirects: --- +!!!Note +Two-phase commit is not available with Group Commit or CAMO. See [Durability limitations](durability/limitations). +!!! + An application can explicitly opt to use two-phase commit with PGD. See [Distributed Transaction Processing: The XA Specification](http://pubs.opengroup.org/onlinepubs/009680699/toc.pdf). From f2c6cb0857c6087d4bc4978a3732cc1461395882 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 23 Nov 2023 12:24:33 +0000 Subject: [PATCH 44/68] Actual commit (as missed file previously) Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/commit-scopes.mdx | 106 ++++++++++++------ 1 file changed, 74 insertions(+), 32 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/commit-scopes.mdx b/product_docs/docs/pgd/5/durability/commit-scopes.mdx index 890c8ea4096..0b6c7ff77e0 100644 --- a/product_docs/docs/pgd/5/durability/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scopes.mdx @@ -4,22 +4,69 @@ title: Commit scopes Commit scopes give applications granular control about durability and consistency of EDB Postgres Distributed. -A commit scope is a named rule that describes the behavior of COMMIT replication. The actual behavior depends on whether a commit scope uses [Group Commit](group-commit), [Commit At Most Once](camo), [Lag Control](lag-control), [Synchronous Commit](synchronous_commit) or combination of these. +A commit scope is a set of rules that describes the behavior of COMMIT replication. The actual behavior depends on whether a commit scope uses [Group Commit](group-commit), [Commit At Most Once](camo), [Lag Control](lag-control), [Synchronous Commit](synchronous_commit) or combination of these. -## Commit Scope structure +## Commit scope structure -Every commit scope has a commit_scope_name and origin_node_group which together uniquely identify the commit scope. +Every commit scope has a commit_scope_name. -The origin_node_group is a PGD group which defines the nodes which will apply this rule when they are the originators of a transaction. +Each commit scope has one or more rules. -A commit scope also has a rule which defines what commit scope kind, or combination of commit scope kinds should be applied to those transactions. +Each rule within the commit scope has an origin_node_group which together uniquely identify the commit scope rule. +The origin_node_group is a PGD group and it defines the nodes which will apply this rule when they are the originators of a transaction. +Finally there is the rule which defines what kind of commit scope or combination of commit scope kinds should be applied to those transactions. -## Configuration +So if a commit scope has a rule that reads -To use Group Commit, first define a commit scope, which determines the PGD nodes involved in the commit of a transaction. Once a scope is established, you can configure a transaction to use Group Commit as follows: + origin_node_group := 'example_bdr_group', + rule := 'ANY 2 (example_bdr_group) GROUP COMMIT', + +Then, the rule is applies when any node in the `example_bdr_group` makes a change. + +The rule itself specifies how many nodes of a specified group will need to confirm the change - `ANY 2 (example_bdr_group)` - followed by the commit scope kind itself - `GROUP COMMIT`. This translates to requiring that any two nodes in the example_bdr_group must confirm the change before the change can be considered as comitted. + +## How a commit scope is selected + +When any change takes place, PGD looks up which commit scope is should be used for the transaction or node. + +If a transaction specifies a commit scope, that scope will be used. + +Then the consulting the group tree is consulted. The node's group is looked up and then starting at the bottom of the group tree with that group and working up, any group which has a default_commit_scope setting defined. This commit scope will then be used. + +If no default_commit_scope is found then the node's GUC, bdr.commit_scope is used. And if that isn't set or is set to `local` then no commit scope applies and PGD's async replication is used. + + +## Creating a Commit Scope + +Use `bdr.add_commit_scope` to add our example rule to a commit scope. For example: +```sql +SELECT bdr.add_commit_scope( + commit_scope_name := 'example_scope', + origin_node_group := 'example_bdr_group', + rule := 'ANY 2 (example_bdr_group) GROUP COMMIT', + wait_for_ready := true +); +``` + +This will add the rule `ANY 2 (example_bdr_group) GROUP COMMIT` for any transaction originating from the `example_bdr_group` to a scope called `example_scope`. + +If no rules previously existed in `example_scope`, then adding this rule would make the scope exist. + +When a rule is added, the origin_node_group must already exist. If it does not, the whole add operation will be discarded with an error. + +The rule will then be evaluated. If the rule mentions groups that don't exist or the settings on the group are incompatible with other configuration setting on the group's nodes, a warning will be emitted, but the rule will be added. + +Once the rule is added the commit scope is available to be used. + +The `wait_for_ready` setting is default tryue + + +## Using a commit scope + +To use our example scope, we can set `bdr.commit_scope` within a transaction ```sql BEGIN; SET LOCAL bdr.commit_scope = 'example_scope'; @@ -29,20 +76,32 @@ COMMIT; You must set the commit scope before the transaction writes any data. -For this example, you might previously have defined the commit scope as: +A commit scope may be set as a default for a group or sub_group using `bdr.alter_node_group_option` ```sql -SELECT bdr.add_commit_scope( - commit_scope_name := 'example_scope', - origin_node_group := 'example_bdr_group', - rule := 'ANY 2 (example_bdr_group) GROUP COMMIT', - wait_for_ready := true +SELECT bdr.alter_node_group_option( + node_group_name := 'example_bdr_group', + config_key := 'default_commit_scope', + config_value := 'example_scope' ); ``` -This example assumes a *node group* named `example_bdr_group` exists and includes at least two PGD nodes as members, either directly or in subgroups. Any transaction committed in the `example_scope` requires one extra confirmation from a PGD node in the group. Together with the origin node, this accounts for "ANY 2" nodes out of the group, on which the transaction is guaranteed to be visible and durable after the commit. +You can also do make this change using PGD cli: -### Origin groups +``` +pgd set-group-options example-bdr-group --option default_commit_scope=example_scope +``` + +Finally, you can set the default commit_scope for a node using: + +```sql +SET bdr.commit_scope = 'example_scope'; +``` + +Set `bdr.commit_scope` to `local` to use the PGD default async replication. + + +## Origin groups Rules for commit scopes can depend on the node the transaction is committed on, that is, the node that acts as the origin for the transaction. To make this transparent for the application, PGD allows a commit scope to define different rules depending on where the transaction originates from. @@ -121,20 +180,3 @@ The `CAMO` commit scope kind is mostly syntactic sugar for `GROUP COMMIT (transa While the grammar for `synchronous_standby_names` and commit scopes can look similar, the former doesn't account for the origin node, but the latter does. Therefore, for example `synchronous_standby_names = 'ANY 1 (..)'` is equivalent to a commit scope of `ANY 2 (...)`. This choice makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction. !!! -### Adding a commit scope rule - -To add a commit scope rule, use the function [`bdr.add_commit_scope`](/pgd/latest/reference/functions#bdradd_commit_scope). This function will create a rule for the given commit scope name and origin node group. If the rule is the same for all nodes in the EDB Postgres Distributed cluster, invoking this function once for the top-level node group is enough to fully define the commit scope. - -Alternatively, you can invoke it multiple times with the same `commit_scope_name` but different origin node groups and rules for commit scopes that vary depending on the origin of the transaction. - -### Changing a commit scope rule - -To change a specific rule for a single origin node group in a commit scope, you can use the function [`bdr.alter_commit_scope`](/pgd/latest/reference/functions#bdralter_commit_scope). - -### Removing a commit scope rule - -To remove a commit scope rule, use [`bdr.remove_commit_scope`](/pgd/latest/reference/functions#bdrremove_commit_scope). This can drop a single rule in a commit scope. If you have defined multiple rules for the commit scope, you will have to invoke this function for each defined rule to remove the entire commit scope. - -!!! Note -Removing a commit scope that's still used as default by a node group isn't allowed. -!!! From 6c1c6d0fa5d034f468a931b11a5a1da27404d5f8 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Mon, 27 Nov 2023 08:23:53 +0000 Subject: [PATCH 45/68] Update product_docs/docs/pgd/5/durability/index.mdx Co-authored-by: Lenz Grimmer --- product_docs/docs/pgd/5/durability/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index d4f9f764c2b..3c9f0aed3f6 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -28,7 +28,7 @@ should consider a transaction committed. ## Introducing -* [Overview](overview) introduces these concepts and some of the essential +* [Overview](overview) introduces the concepts and some of the essential terminology which is used when discussing synchronous commits. * [Commit Scopes](commit-scopes) is a more in-depth look at the syntax and structure From c78bdd1e7e0252f6fd06a27f156ede8b4958c701 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Mon, 27 Nov 2023 09:46:56 +0000 Subject: [PATCH 46/68] Update product_docs/docs/pgd/5/durability/index.mdx Co-authored-by: Lenz Grimmer --- product_docs/docs/pgd/5/durability/index.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 3c9f0aed3f6..2b6a2624602 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -60,7 +60,7 @@ in use should be managed. * [Legacy Synchronous Replication](legacy-sync) shows how traditional Postgres synchronous operations can still be accessed under EDB Postgres Distributed. -* [Internal timing of operations](timing) compared Legacy replication with PGD's async and synchronous +* [Internal timing of operations](timing) compares legacy replication with PGD's async and synchronous operations, especially the difference in the order by which transactions are flushed to disk or made visible. * [Limitations](limitations) lists the various combinations of durability options From 39ea3c9f260a87c26958e4b84d587183e05534d4 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 27 Nov 2023 11:07:33 +0000 Subject: [PATCH 47/68] Review changes Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/camo.mdx | 7 ++++- .../docs/pgd/5/durability/commit-scopes.mdx | 4 --- .../5/durability/durabilityterminology.mdx | 29 +++++++++++++++++++ product_docs/docs/pgd/5/durability/index.mdx | 11 ++++--- 4 files changed, 42 insertions(+), 9 deletions(-) create mode 100644 product_docs/docs/pgd/5/durability/durabilityterminology.mdx diff --git a/product_docs/docs/pgd/5/durability/camo.mdx b/product_docs/docs/pgd/5/durability/camo.mdx index 618cd4a4972..531db7a0c37 100644 --- a/product_docs/docs/pgd/5/durability/camo.mdx +++ b/product_docs/docs/pgd/5/durability/camo.mdx @@ -30,7 +30,12 @@ If the application sends an explicit `COMMIT`, the protocol ensures that the app CAMO works by creating a pair of partner nodes that are two PGD nodes from the same PGD group. In this operation mode, each node in the pair knows the outcome of any recent transaction executed on the other peer and especially (for our need) knows the outcome of any transaction disconnected during `COMMIT`. The node that receives the transactions from the application might be referred to as "origin" and the node that confirms these transactions as "partner." However, there's no difference in the CAMO configuration for the nodes in the CAMO pair. The pair is symmetric. !!! Warning - CAMO requires changes to the user's application to take advantage of the advanced error handling. Enabling a parameter isn't enough to gain protection. Reference client implementations are provided to customers on request. +CAMO requires changes to the user's application to take advantage of the advanced error handling. Enabling a parameter isn't enough to gain protection. Reference client implementations are provided to customers on request. +!!! + +!!! Note +The `CAMO` commit scope kind is mostly an alias for `GROUP COMMIT (transaction_tracking = true, commit_decision = partner)` with an additional `DEGRADE ON` clause. +!!! ## Requirements diff --git a/product_docs/docs/pgd/5/durability/commit-scopes.mdx b/product_docs/docs/pgd/5/durability/commit-scopes.mdx index 0b6c7ff77e0..72b48e3cf47 100644 --- a/product_docs/docs/pgd/5/durability/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scopes.mdx @@ -172,10 +172,6 @@ In rules for commit scopes, you can append these confirmation levels to the node - `ALL (left_dc) ON received AND ANY 1 (right_dc) ON durable` -!!! Note -The `CAMO` commit scope kind is mostly syntactic sugar for `GROUP COMMIT (transaction_tracking = true, commit_decision = partner)` with an additional `DEGRADE ON` clause. It's expected that `GROUP COMMIT` will eventually gain the `DEGRADE ON` clause, making `CAMO` syntax deprecated. -!!! - !!! Note While the grammar for `synchronous_standby_names` and commit scopes can look similar, the former doesn't account for the origin node, but the latter does. Therefore, for example `synchronous_standby_names = 'ANY 1 (..)'` is equivalent to a commit scope of `ANY 2 (...)`. This choice makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction. !!! diff --git a/product_docs/docs/pgd/5/durability/durabilityterminology.mdx b/product_docs/docs/pgd/5/durability/durabilityterminology.mdx new file mode 100644 index 00000000000..e8143d3f210 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/durabilityterminology.mdx @@ -0,0 +1,29 @@ +--- +title: Durability Terminology +--- + +## Durability terminology + +This page covers terms and definitions directly related to PGD's durability options. +For other terms, see the main [Terminology](../terminology) section. + +### Nodes + +PGD nodes take different roles during the replication of a transaction. +These are implicitly assigned per transaction and are unrelated even for +concurrent transactions. + +* The *origin* is the node that receives the transaction from the + client or application. It's the node processing the transaction + first, initiating replication to other PGD nodes and responding back + to the client with a confirmation or an error. + +* The *origin node group* is a PGD group which includes the *origin*. + +* A *partner* node is a PGD node expected to confirm transactions + according to Group Commit requirements. + +* A *commit group* is the group of all PGD nodes involved in the + commit, that is, the origin and all of its partner nodes, which can be + just a few or all peer nodes. + diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 2b6a2624602..1e916331ca4 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -31,12 +31,14 @@ should consider a transaction committed. * [Overview](overview) introduces the concepts and some of the essential terminology which is used when discussing synchronous commits. +* [Durability terminology](durabilityterminology) lists terms used around PGD's +durability options, including how nodes are referred to in replication. + * [Commit Scopes](commit-scopes) is a more in-depth look at the syntax and structure of commit scopes and how to define them for your needs. * [Comparing](comparing) compares how each option behaves. - ## Commit scope kinds * [Group Commit](group-commit) focuses on the Group Commit option, where you can @@ -57,14 +59,15 @@ in a similar fashion to legacy synchronous replication, but from within the comm * [Administering](administering) addresses how a PGD cluster with Group Commit in use should be managed. +* [Limitations](limitations) lists the various combinations of durability options +which are not currently supported or not possible. Do refer to this before deciding +on a durability strategy. + * [Legacy Synchronous Replication](legacy-sync) shows how traditional Postgres synchronous operations can still be accessed under EDB Postgres Distributed. * [Internal timing of operations](timing) compares legacy replication with PGD's async and synchronous operations, especially the difference in the order by which transactions are flushed to disk or made visible. -* [Limitations](limitations) lists the various combinations of durability options -which are not currently supported or not possible. Do refer to this before deciding -on a durability strategy. From 1dfeb39e056a7026c684460afc41caca14f1cdd2 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 27 Nov 2023 11:09:24 +0000 Subject: [PATCH 48/68] Remove terms from overview, update reference Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/overview.mdx | 19 ------------------- .../docs/pgd/5/reference/commit-scopes.mdx | 2 +- 2 files changed, 1 insertion(+), 20 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/overview.mdx b/product_docs/docs/pgd/5/durability/overview.mdx index 2e3a70e819f..98545c1e564 100644 --- a/product_docs/docs/pgd/5/durability/overview.mdx +++ b/product_docs/docs/pgd/5/durability/overview.mdx @@ -57,23 +57,4 @@ replication with `synchronous_commit` and `synchronous_standby_names`. See but consider using [Group Commit](group-commit) or [Synchronous_Commit](synchronous_commit). !!! -## Terms and definitions - -PGD nodes take different roles during the replication of a transaction. -These are implicitly assigned per transaction and are unrelated even for -concurrent transactions. - -* The *origin* is the node that receives the transaction from the - client or application. It's the node processing the transaction - first, initiating replication to other PGD nodes and responding back - to the client with a confirmation or an error. - -* The *origin node group* is a PGD group which includes the *origin*. - -* A *partner* node is a PGD node expected to confirm transactions - according to Group Commit requirements. - -* A *commit group* is the group of all PGD nodes involved in the - commit, that is, the origin and all of its partner nodes, which can be - just a few or all peer nodes. diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index a9ec90be2f9..e5cc05b75d6 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -62,7 +62,7 @@ A transaction under this commit scope group will be considered commited if all o ## Confirmation level -The confirmation level sets when a remote PGD node confirms that it has reached a particular point in processing a transaction. +The confirmation level sets the point in time when a remote PGD node confirms that it has reached a particular point in processing a transaction. ### ON received From d5039520f37132a5dd6ce4a1030a757018612d3f Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Tue, 28 Nov 2023 10:34:47 +0000 Subject: [PATCH 49/68] Update product_docs/docs/pgd/5/durability/lag-control.mdx Co-authored-by: Lenz Grimmer --- product_docs/docs/pgd/5/durability/lag-control.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/lag-control.mdx b/product_docs/docs/pgd/5/durability/lag-control.mdx index 9ad3b2be095..76a581ae475 100644 --- a/product_docs/docs/pgd/5/durability/lag-control.mdx +++ b/product_docs/docs/pgd/5/durability/lag-control.mdx @@ -38,7 +38,7 @@ such as RPO, RCO, and GEO at risk. !!! -To allow an organization to achieve their objectives, PGD offers Lag Control, +To allow organizations to achieve their objectives, PGD offers Lag Control, also known as Replication Lag Control. This feature provides a means to precisely regulate the potential imbalance without intruding on applications, by transparently introducing a delay to READ WRITE transactions that modify data. From b0e8edc4958379c1a57fedad62df7284dd98cc13 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 27 Nov 2023 14:21:43 +0000 Subject: [PATCH 50/68] WIP Commit Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/lag-control.mdx | 41 ++++++++++--------- .../docs/pgd/5/reference/commit-scopes.mdx | 37 +++++++++++++++-- 2 files changed, 56 insertions(+), 22 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/lag-control.mdx b/product_docs/docs/pgd/5/durability/lag-control.mdx index 76a581ae475..9d5ab666a4d 100644 --- a/product_docs/docs/pgd/5/durability/lag-control.mdx +++ b/product_docs/docs/pgd/5/durability/lag-control.mdx @@ -22,22 +22,17 @@ rate at which committed data can replicate to downstream peer nodes. If this imbalance persists, it can put satisfying organizational objectives such as RPO, RCO, and GEO at risk. -!!! information - -- RPO - Recovery point objective specifies the maximum-tolerated amount of data +- **RPO** (Recovery point objective) specifies the maximum-tolerated amount of data that can be lost due to unplanned events, usually expressed as an amount of time. In PGD, RPO determines the acceptable amount of committed data that hasn't been applied to one or more peer nodes. -- RCO - Resource constraint objective acknowledges that there is finite storage - available. In PGD, as lag increases the demands on these storage resources - also increase. +- **RCO** (Resource constraint objective) acknowledges that there is finite storage + available. In PGD, the demands on these storage resources increase as lag increases. -- GEO - Group elasticity objective ensures that any node isn't originating new +- **GEO** (Group elasticity objective) ensures that any node isn't originating new data at a rate that can't be saved to its peer nodes. -!!! - To allow organizations to achieve their objectives, PGD offers Lag Control, also known as Replication Lag Control. This feature provides a means to precisely regulate the potential imbalance without intruding on applications, by @@ -96,10 +91,9 @@ coordinated parameter settings across the nodes spanned by the commit scope rule. You can include a lag control specification in the default commit scope of a top group or as part of an origin group commit scope. -Given a configuration with two datacenters, represented as sub-groups: +As in example, take a configuration with two datacenters, `left_dc` and `right_dc`, represented as sub-groups: -``` --- create subgroups +```sql SELECT bdr.create_node_group( node_group_name := 'left_dc', parent_group_name := 'top_group', @@ -112,11 +106,9 @@ SELECT bdr.create_node_group( ); ``` -This example shows Lag Control rules for two data centers: +The code below adds Lag Control rules for those two data centers, using individual rules for each subgroup: ```sql --- create a Lag Control commit scope with individual rules --- for each sub-group SELECT bdr.add_commit_scope( commit_scope_name := 'example_scope', origin_node_group := 'left_dc', @@ -131,14 +123,25 @@ SELECT bdr.add_commit_scope( ); ``` -The LAG CONTROL syntax is - You can add a lag control commit scope rule to existing commit scope rules that also include group commit and CAMO rule specifications. -The `max_commit_delay` is typically specified in milliseconds (1ms). Using -fractional values for sub-millisecond precision is supported. +The `max_commit_delay` is interval, typically specified in milliseconds (1ms). +Using fractional values for sub-millisecond precision is supported. +The `max_lag_size` is an integer which specifies the maximum allowed lag in +terms of WAL bytes. + +The `max_lag_time` is an interval, typically specified in seconds, that +specifies the maximum allowed lag in terms of time. + +The maximum commit delay (`max_commit_delay`) is a ceiling value representing a hard limit, +which means that a commit delay never exceeds the configured value. + +The maximum lag size and time (`max_lag_size` and `max_lag_time`) are soft limits that can be exceeded. +When the maximum commit delay is reached, +there's no additional back pressure on the lag measures to prevent their +continued increase. ## Transaction application diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index e5cc05b75d6..462e4836783 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -175,9 +175,40 @@ LAG CONTROL [ ( lag_control_parameter = value [, ... ] ) ] Parameter | Type | Default | Description --- | --- | --- | --- - `max_lag_size` | int | 0 | Maximum allowed lag based on WAL bytes. (0 means not set) - `max_lag_time` | interval | 0 | Maximum allowed lag based on wall clock sampling. (0 means not set) - `max_commit_delay` | interval | 0 | Maximum delay that can be injected to commit to try to keep within the lag limits. (0 means not set) + `max_lag_size` | int | 0 | The maximum lag in kB that a given node can have in the replication connection to another node. When the lag exceeds this maximum scaled by `max_commit_delay`, lag control adjusts the commit delay. + `max_lag_time` | interval | 0 | The maximum replication lag in milliseconds that the given origin can have wrt to a replication connection to a given downstream node. + `max_commit_delay` | interval | 0 | Configures the maximum delay each commit can take, in fractional milliseconds. If set to 0, it disables lag control. After each commit delay adjustment (e.g. if the replication is lagging more than `max_lag_size` or `max_lag_time`), the commit delay is recalculated with the weight of the `bdr.lag_control_commit_delay_adjust` GUC. The `max_commit_delay` is a ceiling for the commit delay. + +* If `max_lag_size` and `max_lag_time` are set to 0, the LAG CONTROL is disabled. +* If `max_commit_delay` is not set or set to 0, the LAG CONTROL is disabled. + +The lag size is derived from the delta of the send_ptr of the walsender to the apply_ptr of the receiver. + +The lag time is calculated according to the following formula: + +``` + lag_time = (lag_size / apply_rate) * 1000; +``` + +Where `lag_size` is the delta between the send_ptr and apply_ptr (as used for +`max_lag_size`) and `apply_rate` is a weighted exponential moving average, +following the simplified formula: + +``` + apply_rate = prev_apply_rate * (1 - apply_rate_weight) + + ((apply_ptr_diff * apply_rate_weight) / diff_secs); +``` + +Where: + + `prev_apply_rate` was the previously configured `apply_rate`, before + recalculating the new rate + + `apply_rate_weight` is the value of the GUC `bdr.lag_tracker_apply_rate_weight` + + `apply_ptr_diff` is the difference between the current apply_ptr and + the apply_ptr at the point in time which the apply rate was last + computed + + `diff_secs` is the delta in seconds from the last time the apply rate + was calculated + ## SYNCHRONOUS_COMMIT From 7a3ce6da68569ca769f1f424f038239477ac7a98 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 28 Nov 2023 15:37:19 +0000 Subject: [PATCH 51/68] WIP Commit Signed-off-by: Dj Walker-Morgan --- .../pgd/5/durability/commit-scope-rules.mdx | 60 +++++++++++++++++++ .../docs/pgd/5/durability/commit-scopes.mdx | 23 +------ product_docs/docs/pgd/5/durability/index.mdx | 2 +- .../docs/pgd/5/reference/commit-scopes.mdx | 5 +- 4 files changed, 66 insertions(+), 24 deletions(-) create mode 100644 product_docs/docs/pgd/5/durability/commit-scope-rules.mdx diff --git a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx new file mode 100644 index 00000000000..1b07d4ca4c4 --- /dev/null +++ b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx @@ -0,0 +1,60 @@ +--- +title: Commit scope rules +--- + +Commit scope rules are at the core of the commit scope mechanism. They define what the commit scope will enforce. + +Commit scope rules are composed of one or more operations combined with an `AND`. + +Each operation is made up of two (or three) parts, the commit scope group, an optional confirmation level, and the kind of commit scope (which may have it's own parameters). + +``` +ANY 2 (group) GROUP COMMIT +``` + +The `ANY 2 (group)` is the commit scope group, specifying, for the rule, of which nodes need to respond, confirming they have processed the transaction. Specifically, here, any two nodes from the named group must confirm. + +There is no confirmation level here, but that only means that the default is used. You can think of the rule in full as: + +``` +ANY 2 (group) ON visible GROUP COMMIT +``` + +The visible setting means the nodes are able to confirm once the all the transaction's changes are flushed to disk and visible to other transactions. + +The last part of this operation is the commit scope kind, which here is `GROUP COMMIT`, a synchronous two-phase commit which will be confirmed when any two nodes in the named group confirm they have flushed and made visible the transactions changes. + +## The commit scope group + +There are three kinds of commit scope group, `ANY`, `ALL` and `MAJORITY`. They are all followed by a parenthesed list of one or more groups, which combine to make a pool of nodes that this operation will apply to. + +- `ANY` &mdash is followed by an integer value, "n". It translates to any "n" nodes in the listed groups nodes. +- `ALL` &mdash is followed by the groups and translates to all nodes in the listed groups noes. +- `MAJORITY` &mdsh is followed by the groups and translates to requiring a half, plus one, of the listed groups nodes to confirm to give a majority. + +## The Confirmation Level + +PGD nodes can send confirmations for a transaction at different times. In increasing levels of protection, from the perspective of the confirming node, these are: + +- `received` — A remote PGD node confirms the transaction immediately after receiving it, prior to starting the local application. +- `replicated` — Confirms after applying changes of the transaction but before flushing them to disk. +- `durable` — Confirms the transaction after all of its changes are flushed to disk. +- `visible` (default) — Confirms the transaction after all of its changes are flushed to disk and it's visible to concurrent transactions. + +In rules for commit scopes, you can append these confirmation levels to the node group definition in parentheses with `ON` as follows: + +- `ANY 2 (right_dc) ON replicated` +- `ALL (left_dc) ON visible` (default) +- `ALL (left_dc) ON received AND ANY 1 (right_dc) ON durable` + +!!! Note +If you are familiar with Postgresql's `synchronous_standby_names` feature, be aware that while the grammar for `synchronous_standby_names` and commit scopes can look similar, there is a subtle difference. The former doesn't account for the origin node, but the latter does. For example `synchronous_standby_names = 'ANY 1 (..)'` is equivalent to a commit scope of `ANY 2 (...)`. This difference makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction. +!!! + +## The Commit Scope kind + +There are, currently, four commit scope kinds. Each of them has their own page, so we'll be summarizing and linking to them here: + +### `GROUP COMMIT` + +Group commit is a synchronous two-phase commit which will be confirmed according to the diff --git a/product_docs/docs/pgd/5/durability/commit-scopes.mdx b/product_docs/docs/pgd/5/durability/commit-scopes.mdx index 72b48e3cf47..d5c8ea4a1f0 100644 --- a/product_docs/docs/pgd/5/durability/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scopes.mdx @@ -59,9 +59,9 @@ When a rule is added, the origin_node_group must already exist. If it does not, The rule will then be evaluated. If the rule mentions groups that don't exist or the settings on the group are incompatible with other configuration setting on the group's nodes, a warning will be emitted, but the rule will be added. -Once the rule is added the commit scope is available to be used. +Once the rule is added, the commit scope will be available for use. -The `wait_for_ready` setting is default tryue +The `wait_for_ready` controls whether the `bdr.add_commit_scope()` call blocks until the rule has been added to the relevant nodes. The setting defaults to true and can be omitted. ## Using a commit scope @@ -156,23 +156,4 @@ SELECT bdr.alter_node_group_option( ); ``` -### Confirmation levels - -PGD nodes can send confirmations for a transaction at different times. In increasing levels of protection, from the perspective of the confirming node, these are: - -- `received` — A remote PGD node confirms the transaction immediately after receiving it, prior to starting the local application. -- `replicated` — Confirms after applying changes of the transaction but before flushing them to disk. -- `durable` — Confirms the transaction after all of its changes are flushed to disk. -- `visible` (default) — Confirms the transaction after all of its changes are flushed to disk and it's visible to concurrent transactions. - -In rules for commit scopes, you can append these confirmation levels to the node group definition in parentheses with `ON` as follows: - -- `ANY 2 (right_dc) ON replicated` -- `ALL (left_dc) ON visible` (default) -- `ALL (left_dc) ON received AND ANY 1 (right_dc) ON durable` - - -!!! Note -While the grammar for `synchronous_standby_names` and commit scopes can look similar, the former doesn't account for the origin node, but the latter does. Therefore, for example `synchronous_standby_names = 'ANY 1 (..)'` is equivalent to a commit scope of `ANY 2 (...)`. This choice makes reasoning about majority easier and reflects that the origin node also contributes to the durability and visibility of the transaction. -!!! diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 1e916331ca4..52aefee6884 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -3,7 +3,7 @@ title: Durability navigation: - overview - - configuring + - durabilityterminology - commit-scopes - comparing - '# Commit Scope kinds' diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index 462e4836783..954fddbd1e9 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -22,14 +22,15 @@ commit_scope: commit_scope_operation: commit_scope_group confirmation_level commit_scope_kind -confirmation_level: - [ ON { received|replicated|durable|visible } ] commit_scope_group: { ANY num (node_group [, ...]) | MAJORITY (node_group [, ...]) | ALL (node_group [, ...]) } +confirmation_level: + [ ON { received|replicated|durable|visible } ] + commit_scope_kind: { GROUP COMMIT [ ( group_commit_parameter = value [, ...] ) ] [ ABORT ON ( abort_on_parameter = value ) ] | CAMO [ DEGRADE ON ( degrade_on_parameter = value ) TO ASYNC ] From 2ccccd82c5f32c597740f17fdf39202179c5708c Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 28 Nov 2023 16:45:07 +0000 Subject: [PATCH 52/68] Added commit scope kinds Signed-off-by: Dj Walker-Morgan --- .../pgd/5/durability/commit-scope-rules.mdx | 19 ++++++++++++++++++- 1 file changed, 18 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx index 1b07d4ca4c4..d0a0902127b 100644 --- a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx @@ -57,4 +57,21 @@ There are, currently, four commit scope kinds. Each of them has their own page, ### `GROUP COMMIT` -Group commit is a synchronous two-phase commit which will be confirmed according to the +Group commit is a synchronous two-phase commit which will be confirmed according to the requirements of the commit scope group. `GROUP COMMIT` has a number of options which control whether transactions should be tracked over interruptions (boolean, defaults to off), how conflicts should be resolved (`async` or `eager`, defaults to `async`) and how a consensus is obtained (`group`, `partner` or `raft`, defaults to `group`). + +For further details see [`GROUP COMMIT`](group-commit). + +### `CAMO` + +Camo, Commit At Most Once, allows the client/application, origin node and partner node to ensure that a transaction is committed to the database at most once. Because the client is involved in the process, the application will require modifications to participate in the CAMO process. + +For further details see [`CAMO`](camo). + +### `LAG CONTROL` + +With Lag control, when the system's replication performance exceeds specified limits, a delay can be automatically injected into client interaction with the database, providing a back pressure on clients. + + +### `SYNCHRONOUS_COMMIT` + + From 329cffae53033abadfbf84a38acfcc726142e45b Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 28 Nov 2023 16:47:10 +0000 Subject: [PATCH 53/68] Update index for rules Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/index.mdx | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index 52aefee6884..da8fd4d2ac4 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -5,6 +5,7 @@ navigation: - overview - durabilityterminology - commit-scopes + - commit-scope-rules - comparing - '# Commit Scope kinds' - group-commit @@ -34,9 +35,12 @@ terminology which is used when discussing synchronous commits. * [Durability terminology](durabilityterminology) lists terms used around PGD's durability options, including how nodes are referred to in replication. -* [Commit Scopes](commit-scopes) is a more in-depth look at the syntax and structure +* [Commit scopes](commit-scopes) is a more in-depth look at the structure of commit scopes and how to define them for your needs. +* [Commit scope rules](commit-scope-rules) looks at the syntax of and how to formulate +a commit scope rule. + * [Comparing](comparing) compares how each option behaves. ## Commit scope kinds From 9b79c30716d634e86269d0e16ede0069c6652ba4 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 29 Nov 2023 11:14:19 +0000 Subject: [PATCH 54/68] Review edits Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/commit-scopes.mdx | 13 +++++++------ product_docs/docs/pgd/5/durability/index.mdx | 18 ++++++++++-------- .../docs/pgd/5/durability/overview.mdx | 19 ++++++++++--------- .../docs/pgd/5/reference/commit-scopes.mdx | 2 -- 4 files changed, 27 insertions(+), 25 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/commit-scopes.mdx b/product_docs/docs/pgd/5/durability/commit-scopes.mdx index d5c8ea4a1f0..bbd3f34425d 100644 --- a/product_docs/docs/pgd/5/durability/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scopes.mdx @@ -4,11 +4,13 @@ title: Commit scopes Commit scopes give applications granular control about durability and consistency of EDB Postgres Distributed. -A commit scope is a set of rules that describes the behavior of COMMIT replication. The actual behavior depends on whether a commit scope uses [Group Commit](group-commit), [Commit At Most Once](camo), [Lag Control](lag-control), [Synchronous Commit](synchronous_commit) or combination of these. +A commit scope is a set of rules that describes the behavior of the system as transactions are committed. The actual behavior depends on which a kind of commit scope a commit scope's rule uses [Group Commit](group-commit), [Commit At Most Once](camo), [Lag Control](lag-control), [Synchronous Commit](synchronous_commit) or combination of these. + +While most commit scope kinds control the processing of the transaction, Lag Control is the exception as it dynamically regulates the performance of the system in response to replication operations being slow or queued up. It is typically used, though, in combination with other commit scope kinds ## Commit scope structure -Every commit scope has a commit_scope_name. +Every commit scope has a name (a `commit_scope_name`). Each commit scope has one or more rules. @@ -25,7 +27,7 @@ So if a commit scope has a rule that reads Then, the rule is applies when any node in the `example_bdr_group` makes a change. -The rule itself specifies how many nodes of a specified group will need to confirm the change - `ANY 2 (example_bdr_group)` - followed by the commit scope kind itself - `GROUP COMMIT`. This translates to requiring that any two nodes in the example_bdr_group must confirm the change before the change can be considered as comitted. +The rule itself specifies how many nodes of a specified group will need to confirm the change - `ANY 2 (example_bdr_group)` - followed by the commit scope kind itself - `GROUP COMMIT`. This translates to requiring that any two nodes in `example_bdr_group` must confirm the change before the change can be considered as comitted. ## How a commit scope is selected @@ -33,11 +35,10 @@ When any change takes place, PGD looks up which commit scope is should be used f If a transaction specifies a commit scope, that scope will be used. -Then the consulting the group tree is consulted. The node's group is looked up and then starting at the bottom of the group tree with that group and working up, any group which has a default_commit_scope setting defined. This commit scope will then be used. +If not specified, the system will search for a default commit scope. Default commit scopes are a group level setting. The system consults the group tree. Starting at the bottom of the group tree with the node's group and working up, it searches for any group which has a default_commit_scope setting defined. This commit scope will then be used. If no default_commit_scope is found then the node's GUC, bdr.commit_scope is used. And if that isn't set or is set to `local` then no commit scope applies and PGD's async replication is used. - ## Creating a Commit Scope Use `bdr.add_commit_scope` to add our example rule to a commit scope. For example: @@ -137,7 +138,7 @@ SELECT bdr.add_commit_scope( ); ``` -Now, using the `example_scope` on any node that's part of `left_dc` uses the first scope. Using the same scope on a node that's part of `right_dc` uses the second scope. This is an effective way of creating an inverted scope without having to juggle scope names in the application. +Now, using the `example_scope` on any node that's part of `left_dc` uses the first scope. Using the same scope on a node that's part of `right_dc` uses the second scope. By combining the `left_dc` and `right_dc` origin rules under one commit scope name, an application can simply use `example_scope` on either data center and get the appropriate behavior for that data center. Each group can also have a default commit scope specified using the `bdr.alter_node_group_option` admin interface. diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index da8fd4d2ac4..c76efccc521 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -22,7 +22,7 @@ redirects: - /pgd/latest/choosing_durability/ --- -EDB Postgres Distributed offers a range of synchronous modes to complement its +EDB Postgres Distributed (PGD) offers a range of synchronous modes to complement its default asynchronous replication. These synchronous modes are configured through commit scopes; rules that define how operations are handled and when the system should consider a transaction committed. @@ -46,14 +46,16 @@ a commit scope rule. ## Commit scope kinds * [Group Commit](group-commit) focuses on the Group Commit option, where you can -define a transaction as done when a group of nodes agrees its done. +define a transaction as done when a group of nodes agrees it's done. -* [CAMO](camo) focuses on the Commit At Most Once option, which allows applications -to participate in confirming a transaction has been done and in turn ensure that -their commits only happen at most once. +* [CAMO](camo) focuses on the Commit At Most Once option, which allows +applications to take on the responsibility of verifying that a commit has been +successful before retying. This ensures that their commits only happen at most +once. -* [Lag Control](lag-control) looks at the commit scope mechanism which regulates how -far out of sync nodes may go when a database node goes out of service. +* [Lag Control](lag-control) looks at the commit scope mechanism which +dynamically throttle nodes according to the slowest node and regulates how far +out of sync nodes may go when a database node goes out of service. * [Synchronous Commit](synchronous_commit) examines a commit scope mechanism which works in a similar fashion to legacy synchronous replication, but from within the commit scope framework. @@ -68,7 +70,7 @@ which are not currently supported or not possible. Do refer to this before decid on a durability strategy. * [Legacy Synchronous Replication](legacy-sync) shows how traditional Postgres synchronous operations -can still be accessed under EDB Postgres Distributed. +can still be accessed under PGD. * [Internal timing of operations](timing) compares legacy replication with PGD's async and synchronous operations, especially the difference in the order by which transactions are flushed to disk or made visible. diff --git a/product_docs/docs/pgd/5/durability/overview.mdx b/product_docs/docs/pgd/5/durability/overview.mdx index 98545c1e564..b6b5f3ba7da 100644 --- a/product_docs/docs/pgd/5/durability/overview.mdx +++ b/product_docs/docs/pgd/5/durability/overview.mdx @@ -5,18 +5,17 @@ navTitle: Overview ## Overview -EDB Postgres Distributed allows you to choose from several replication +EDB Postgres Distributed (PGD) allows you to choose from several replication configurations based on your durability, consistency, availability, and performance needs using *commit scopes*. -In its basic configuration, EDB Postgres Distributed uses asynchronous -replication. However, commit scopes can change both the default and the -per-transaction behavior. +In its basic configuration, PGD uses asynchronous replication. However, commit +scopes can change both the default and the per-transaction behavior. -It's also possible to configure the legacy Postgres -synchronous replication using standard `synchronous_standby_names` in the same -way as the built-in physical or logical replication. However, commit scopes -provide much more flexibility and control over the replication behavior. +It's also possible to configure the legacy Postgres synchronous replication +using standard `synchronous_standby_names` in the same way as the built-in +physical or logical replication. However, commit scopes provide much more +flexibility and control over the replication behavior. The different synchronization settings affect three properties of interest to applications that are related but can all be implemented individually: @@ -42,7 +41,9 @@ Commit scopes allow four kinds of controlling durability of the transaction: and at what stage of replication it can be considered committed. This option also allows you to control the visibility ordering of the transaction. -- [CAMO](camo): This kind of commit scope is a variant of Group Commit in which the client is part of the consensus. +- [CAMO](camo): This kind of commit scope is a variant of Group Commit in +which enables the client take on the responsibility of verifying a commit went +through before retrying. - [Lag Control](lag-control): This kind of commit scope controls how far behind nodes can be in terms of replication before allowing commit to proceed. diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index 954fddbd1e9..9f9995551d9 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -22,7 +22,6 @@ commit_scope: commit_scope_operation: commit_scope_group confirmation_level commit_scope_kind - commit_scope_group: { ANY num (node_group [, ...]) | MAJORITY (node_group [, ...]) @@ -221,4 +220,3 @@ SYNCHRONOUS_COMMIT The `SYNCHRONOUS_COMMIT` commit scope kind has no parameters. It is effectively configured only by the commit scope group and commit scope visibility. - From 50ae64210ed951575e30d3c16b8b2d2234c2ff46 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Wed, 29 Nov 2023 12:56:56 +0000 Subject: [PATCH 55/68] Review edits cont. Signed-off-by: Dj Walker-Morgan --- .../pgd/5/durability/commit-scope-rules.mdx | 15 ++++-- .../docs/pgd/5/durability/group-commit.mdx | 47 +++++++++++-------- .../docs/pgd/5/durability/lag-control.mdx | 12 ++--- product_docs/docs/pgd/5/durability/timing.mdx | 26 +++++----- 4 files changed, 57 insertions(+), 43 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx index d0a0902127b..705f6975eaf 100644 --- a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx @@ -6,7 +6,15 @@ Commit scope rules are at the core of the commit scope mechanism. They define wh Commit scope rules are composed of one or more operations combined with an `AND`. -Each operation is made up of two (or three) parts, the commit scope group, an optional confirmation level, and the kind of commit scope (which may have it's own parameters). +Each operation is made up of two (or three) parts, the commit scope group, an optional confirmation level, and the kind of commit scope (which may have it's own parameters). + +``` +commit_scope_group [ confimation_level ] commit_scope_kind +``` + +A full formal syntax diagram is available in the [commit scope reference section.](/pgd/latest/reference/commit-scopes/#commit-scope-syntax). + +If we look at a typical commit scope rule, we can now break it down into its components: ``` ANY 2 (group) GROUP COMMIT @@ -32,7 +40,7 @@ There are three kinds of commit scope group, `ANY`, `ALL` and `MAJORITY`. They a - `ALL` &mdash is followed by the groups and translates to all nodes in the listed groups noes. - `MAJORITY` &mdsh is followed by the groups and translates to requiring a half, plus one, of the listed groups nodes to confirm to give a majority. -## The Confirmation Level +## The confirmation Level PGD nodes can send confirmations for a transaction at different times. In increasing levels of protection, from the perspective of the confirming node, these are: @@ -69,8 +77,7 @@ For further details see [`CAMO`](camo). ### `LAG CONTROL` -With Lag control, when the system's replication performance exceeds specified limits, a delay can be automatically injected into client interaction with the database, providing a back pressure on clients. - +With Lag control, when the system's replication performance exceeds specified limits, a commit delay can be automatically injected into client interaction with the database, providing a back pressure on clients. Lag control has parameters to set the maximum commit delay that can be exerted, and limits in terms of time to process or queue size which will trigger increases in that commit delay. ### `SYNCHRONOUS_COMMIT` diff --git a/product_docs/docs/pgd/5/durability/group-commit.mdx b/product_docs/docs/pgd/5/durability/group-commit.mdx index 605adbea10e..9e07b75a2b2 100644 --- a/product_docs/docs/pgd/5/durability/group-commit.mdx +++ b/product_docs/docs/pgd/5/durability/group-commit.mdx @@ -11,7 +11,9 @@ Commit scope kind: `GROUP COMMIT` The goal of Group Commit is to protect against data loss in case of single node failures or temporary outages. You achieve this by requiring more than one PGD -node to successfully receive and confirm a transaction at COMMIT time. +node to successfully confirm a transaction at COMMIT time. Confirmation can be sent +at a number of points in the transaction processing, but defaults to "visible" when +the transaction has been flushed to disk and is visible to all other transactions. ## Example @@ -37,9 +39,10 @@ activity currently happens only when either the origin node recovers or when it's parted from the cluster. This behavior is the same as with Postgres legacy built-in synchronous replication. -Transactions committed with Group Commit use two-phase commit underneath. -Therefore, configure `max_prepared_transactions` high enough to handle all such -transactions originating per node. +Transactions committed with Group Commit use [two-phase +commit](../terminology/#two-phase-commit-2pc) underneath. Therefore, configure +`max_prepared_transactions` high enough to handle all such transactions +originating per node. ## Limitations @@ -61,10 +64,10 @@ scope. You can configure Group Commit to decide commits in three different ways: `group`, `partner`, and `raft`. -The `group` decision is done through the consensus specified using the same -commit scope group settings used for the durability purposes. The difference -is that the commit decision is made based on PREPARE replication while the -durability checks COMMIT (PREPARED) replication. +The `group` decision is the default. It specifies that the commit is confirmed +by the origin node upon it recieving as many confirmations as required by the +commit scope group. The difference is that the commit decision is made based on +PREPARE replication while the durability checks COMMIT (PREPARED) replication. The `partner` decision is what [Commit At Most Once](camo) uses. This approach works only when there are two data nodes in the node group. These two nodes are @@ -73,23 +76,27 @@ to commit something. This approach requires application changes to use the CAMO transaction protocol to work correctly, as the application is in some way part of the consensus. For more on this approach, see [CAMO](camo). -The `raft` option uses the built-in Raft consensus to decide whether commit can -happen. Currently, the global Raft is used. For this to work, the majority of -nodes across the whole cluster must work. +The `raft` option uses the PGD's built-in Raft consensus to decide whether +commit can happen. Currently, the global Raft is used. For this to work, the +majority of nodes across the whole cluster must work. This option means that all +commits are effectively ordered and every node moves forward in step with those +commits at the same rate. ### Conflict resolution Conflict resolution can be `async` or `eager`. -Async means that PGD does optimistic conflict resolution during replication using the row-level resolution as configured for given node. This happens regardless of whether the origin transaction committed or is still in progress. See -[Conflicts](../consistency/conflicts) for details about -how the asynchronous conflict resolution works. - -Eager means that conflicts are resolved eagerly (as part of agreement on COMMIT), -and conflicting transactions get aborted with a serialization error. This approach -provides greater isolation than the asynchronous resolution at the price of -performance. For the details about how Eager conflict resolution works, see -[Eager conflict resolution](../consistency/eager). +Async means that PGD does optimistic conflict resolution during replication +using the row-level resolution as configured for given node. This happens +regardless of whether the origin transaction committed or is still in progress. +See [Conflicts](../consistency/conflicts) for details about how the asynchronous +conflict resolution works. + +Eager means that conflicts are resolved eagerly (as part of agreement on +COMMIT), and conflicting transactions get aborted with a serialization error. +This approach provides greater isolation than the asynchronous resolution at the +price of performance. For the details about how Eager conflict resolution works, +see [Eager conflict resolution](../consistency/eager). ### Aborts diff --git a/product_docs/docs/pgd/5/durability/lag-control.mdx b/product_docs/docs/pgd/5/durability/lag-control.mdx index 9d5ab666a4d..5ca17cb96a5 100644 --- a/product_docs/docs/pgd/5/durability/lag-control.mdx +++ b/product_docs/docs/pgd/5/durability/lag-control.mdx @@ -33,11 +33,10 @@ such as RPO, RCO, and GEO at risk. - **GEO** (Group elasticity objective) ensures that any node isn't originating new data at a rate that can't be saved to its peer nodes. -To allow organizations to achieve their objectives, PGD offers Lag Control, -also known as Replication Lag Control. This feature provides a means to -precisely regulate the potential imbalance without intruding on applications, by -transparently introducing a delay to READ WRITE transactions that modify data. -This delay, the PGD Commit Delay, starts at 0ms. +To allow organizations to achieve their objectives, PGD offers Lag Control. This +feature provides a means to precisely regulate the potential imbalance without +intruding on applications, by transparently introducing a delay to READ WRITE +transactions that modify data. This delay, the PGD Commit Delay, starts at 0ms. Using the LAG CONTROL commit scope kind, you can set a maximum time that commits commits can be delayed between nodes in a group, maximum lag time or maximum lag @@ -220,4 +219,5 @@ transactions from observing or modifying its values or acquiring its resources. The same guarantee can't be made for external resources managed by Postgres extensions. Regardless of extension dependencies, the same guarantee can be made if the PGD extension is listed before extension-based resource managers in -postgresql.conf. \ No newline at end of file +postgresql.conf. + diff --git a/product_docs/docs/pgd/5/durability/timing.mdx b/product_docs/docs/pgd/5/durability/timing.mdx index 5c31ecef401..8938be4132c 100644 --- a/product_docs/docs/pgd/5/durability/timing.mdx +++ b/product_docs/docs/pgd/5/durability/timing.mdx @@ -8,11 +8,11 @@ different ways. With Legacy PSR, the order of operations is: -- Origin flushes a commit record to WAL, making the transaction +1. Origin flushes a commit record to WAL, making the transaction visible locally. -- Peer node receives changes and issues a write. -- Peer flushes the received changes to disk. -- Peer applies changes, making the transaction visible on the peer. +1. Peer node receives changes and issues a write. +1. Peer flushes the received changes to disk. +1. Peer applies changes, making the transaction visible on the peer. Note that the change is written to the disk before applying the chnages. @@ -20,21 +20,21 @@ With PGD, by default and with Lag Control, the order of operations is different. In these cases, the change becomes visible on the peer before the transaction is flushed to the peer's disk: -- Origin flushes a commit record to WAL, making the transaction +1. Origin flushes a commit record to WAL, making the transaction visible locally. -- Peer node receives changes into its apply queue in memory. -- Peer applies changes, making the transaction visible on the peer. -- Peer persists the transaction by flushing to disk. +1. Peer node receives changes into its apply queue in memory. +1. Peer applies changes, making the transaction visible on the peer. +1. Peer persists the transaction by flushing to disk. For PGD's Group Commit and CAMO, the origin node waits for a certain number of confirmations prior to making the transaction visible locally. The order of operations is: -- Origin flushes a prepare or precommit record to WAL. -- Peer node receives changes into its apply queue in memory. -- Peer applies changes, making the transaction visible on the peer. -- Peer persists the transaction by flushing to disk. -- Origin commits and makes the transaction visible locally. +1. Origin flushes a prepare or precommit record to WAL. +1. Peer node receives changes into its apply queue in memory. +1. Peer applies changes, making the transaction visible on the peer. +1. Peer persists the transaction by flushing to disk. +1. Origin commits and makes the transaction visible locally. The following table summarizes the differences. From 07052f41d3f262f20cbdacd893a0c3ac68ed8c38 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 30 Nov 2023 14:12:47 +0000 Subject: [PATCH 56/68] Shorter version focussing on GC Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/comparing.mdx | 103 ++++++++++-------- 1 file changed, 57 insertions(+), 46 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index 69913b3fc80..cf2e3cd1a78 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -5,53 +5,64 @@ navTitle: Comparing ## Comparison -Most options for synchronous replication available to -PGD allow for different levels of synchronization, offering different -tradeoffs between performance and protection against node or network -outages. - -The table summarizes what a client can expect from a peer node that has -replicated a transaction after receiving a COMMIT confirmation from the origin -node. For commit scopes the Mode column refers to the confirmation requirements -of the [commit scope configuration](commit-scopes#configuration). - -| Variant | Mode | Received | Visible | Durable | -|------------------------|-----------------------|----------|---------|---------| -| PGD Async | off (default) | no | no | no | -| PGD Group Commit | 'ON received' nodes | yes | no | no | -| PGD Group Commit | 'ON replicated' nodes | yes | no | no | -| PGD Group Commit | 'ON durable' nodes | yes | no | yes | -| PGD Group Commit | 'ON visible' nodes | yes | yes | yes | -| PGD CAMO | 'ON received' nodes | yes | no | no | -| PGD CAMO | 'ON replicated' nodes | yes | no | no | -| PGD CAMO | 'ON durable' nodes | yes | no | yes | -| PGD CAMO | 'ON visible' nodes | yes | yes | yes | -| PGD Lag Control | 'ON received' nodes | no | no | no | -| PGD Lag Control | 'ON replicated' nodes | no | no | no | -| PGD Lag Control | 'ON durable' nodes | no | no | no | -| PGD Lag Control | 'ON visible' nodes | no | no | no | -| PGD Synchronous Commit | 'ON received' nodes | no | no | no | -| PGD Synchronous Commit | 'ON replicated' nodes | no | no | no | -| PGD Synchronous Commit | 'ON durable' nodes | no | no | no | -| PGD Synchronous Commit | 'ON visible' nodes | no | no | no | - -Reception ensures the peer operating normally can +Most options for synchronous replication available to PGD allow for different +levels of synchronization, offering different tradeoffs between performance and +protection against node or network outages. + +The following list of [confirmation levels](commit-scopes-rules#the-confirmation-level) explains +what a user should expect to see when that confirmation level in in effect and how that can effect performance, durability, and consistency. + +### ON RECIEVED + +**Expect**: The peer node has recieved the changes. Nothing has been updated in +the peer nodes tuple store or written to storage. + +Confirmation on reception means that the peer operating normally can eventually apply the transaction without requiring any further communication, even in the face of a full or partial network outage. A crash of a peer node might still require retransmission of the transaction, as this confirmation doesn't involve persistent -storage. All modes considered synchronous provide this protection. - -Visibility implies the transaction was applied remotely. All other -clients see the results of the transaction on all nodes, providing -this guarantee immediately after the commit is confirmed by the origin -node. Without visibility, other clients connected might not see the -results of the transaction and experience stale reads. - -Durability relates to the peer node's storage and provides protection -against loss of data after a crash and recovery of the peer node. -This can either relate to the reception of the data (as with physical -streaming replication) or to visibility (as with Group Commit). -The former eliminates the need for retransmissions after -a crash, while the latter ensures visibility is maintained across -restarts. \ No newline at end of file +storage. + +**For**: Good performance. +**Against**: Less robust when a crash occurs. + +### ON REPLICATED + +**Expect**: The peer node has recieved the changes and applied them to the tuple +store. The changes have not been persisted to storage. + +Confirmation on replication means the peer has recieved and applied the changes, but +has yet to persist them to disk. + +**For**: A better checkpoint than recieved with less performance +**Against**: Less robust when a crash occurs. + +### ON DURABLE + +**Expect**: The peer node has recieved the changes, applied them to the tuple store and persisted the changes to storage. It has yet to make the changes available to other concurrent transactions. + +Durable confirmation means that the transaction has been written to the peer +node's storage. This provides protection against loss of data after a crash and +recovery of the peer node. With Group Commit, this also means the changes are +visible. + +**For**: More robust, able to recover without retransmission in the event of a crash. +**Against**: Doesn't guarantee consistency. + +### ON VISIBLE + +**Expect**: The peer node has recieved and applied the changes, and persisted them to storage, and has now made them visible to other concurrent transactions. + +Confirmation on visibility means that the transaction was applied remotely. All +other clients see the results of the transaction on all nodes, providing this +guarantee immediately after the commit is confirmed by the origin node. Without +visibility, other clients connected might not see the results of the transaction +and experience stale reads. + +**For**: Robust and consistent. +**Against**: Lower performance. + + + + From 7a54ea78515e3810a2ff67bf834eb8518915e5c4 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 30 Nov 2023 14:41:49 +0000 Subject: [PATCH 57/68] Format and typo fix Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/comparing.mdx | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index cf2e3cd1a78..77ba48b99fd 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -12,9 +12,9 @@ protection against node or network outages. The following list of [confirmation levels](commit-scopes-rules#the-confirmation-level) explains what a user should expect to see when that confirmation level in in effect and how that can effect performance, durability, and consistency. -### ON RECIEVED +### ON RECEIVED -**Expect**: The peer node has recieved the changes. Nothing has been updated in +**Expect**: The peer node has received the changes. Nothing has been updated in the peer nodes tuple store or written to storage. Confirmation on reception means that the peer operating normally can @@ -25,22 +25,24 @@ the transaction, as this confirmation doesn't involve persistent storage. **For**: Good performance. + **Against**: Less robust when a crash occurs. ### ON REPLICATED -**Expect**: The peer node has recieved the changes and applied them to the tuple +**Expect**: The peer node has received the changes and applied them to the tuple store. The changes have not been persisted to storage. -Confirmation on replication means the peer has recieved and applied the changes, but +Confirmation on replication means the peer has received and applied the changes, but has yet to persist them to disk. -**For**: A better checkpoint than recieved with less performance +**For**: A better checkpoint than received with less performance. + **Against**: Less robust when a crash occurs. ### ON DURABLE -**Expect**: The peer node has recieved the changes, applied them to the tuple store and persisted the changes to storage. It has yet to make the changes available to other concurrent transactions. +**Expect**: The peer node has received the changes, applied them to the tuple store and persisted the changes to storage. It has yet to make the changes available to other concurrent transactions. Durable confirmation means that the transaction has been written to the peer node's storage. This provides protection against loss of data after a crash and @@ -48,6 +50,7 @@ recovery of the peer node. With Group Commit, this also means the changes are visible. **For**: More robust, able to recover without retransmission in the event of a crash. + **Against**: Doesn't guarantee consistency. ### ON VISIBLE @@ -61,6 +64,7 @@ visibility, other clients connected might not see the results of the transaction and experience stale reads. **For**: Robust and consistent. + **Against**: Lower performance. From 9e685fd3703352f879ca58e894d2a288c3893233 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 30 Nov 2023 15:37:57 +0000 Subject: [PATCH 58/68] Add deepToC Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/comparing.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index 77ba48b99fd..c6ce7506778 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -1,6 +1,7 @@ --- title: Comparing Durability Options navTitle: Comparing +deepToC: true --- ## Comparison From 168ce4e18b3ee4c74106007691c168d4cea7b36f Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 30 Nov 2023 16:19:31 +0000 Subject: [PATCH 59/68] wrapped sync commit wordage - verification needed Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/commit-scope-rules.mdx | 6 ++++++ product_docs/docs/pgd/5/durability/synchronous_commit.mdx | 3 ++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx index 705f6975eaf..6f3fb3f4bd5 100644 --- a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx @@ -79,6 +79,12 @@ For further details see [`CAMO`](camo). With Lag control, when the system's replication performance exceeds specified limits, a commit delay can be automatically injected into client interaction with the database, providing a back pressure on clients. Lag control has parameters to set the maximum commit delay that can be exerted, and limits in terms of time to process or queue size which will trigger increases in that commit delay. +For further details see [`LAG CONTROL`](lag-control) + ### `SYNCHRONOUS_COMMIT` +Synchronous Commit is a commit scope option which is designed to be like the legacy `synchronous_commit` option, but accessible within the commit scope environment. Unlike `GROUP COMMIT` it is a synchronous non-two-phase commit operation, with no parameters. The preceding commit scope group controls what groups and confirmation requirements the `SYNCHRONOUS_COMMIT` will use. + +For further details see [`SYNCHRONOUS_COMMIT`](synchronous_commit) + diff --git a/product_docs/docs/pgd/5/durability/synchronous_commit.mdx b/product_docs/docs/pgd/5/durability/synchronous_commit.mdx index 67d5571d885..7460120a60c 100644 --- a/product_docs/docs/pgd/5/durability/synchronous_commit.mdx +++ b/product_docs/docs/pgd/5/durability/synchronous_commit.mdx @@ -26,4 +26,5 @@ In this example, any node in the `left_dc` group, when this commit scope is in u ## Details -TODO: \ No newline at end of file +Currently `SYNCHRONOUS_COMMIT` does not use the confirmation levels of the commit scope rule syntax. + From e434a679290276b9e2902722e9237c883828a107 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 30 Nov 2023 17:57:49 +0000 Subject: [PATCH 60/68] Clear end of doc Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/durability/timing.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/product_docs/docs/pgd/5/durability/timing.mdx b/product_docs/docs/pgd/5/durability/timing.mdx index 8938be4132c..dc7605b4dc8 100644 --- a/product_docs/docs/pgd/5/durability/timing.mdx +++ b/product_docs/docs/pgd/5/durability/timing.mdx @@ -46,3 +46,4 @@ The following table summarizes the differences. | PGD Group Commit | apply first | before COMMIT on origin | | PGD CAMO | apply first | before COMMIT on origin | + From f91d96157f7be6dc4106c7e87ddb56878cff426d Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 30 Nov 2023 18:50:51 +0000 Subject: [PATCH 61/68] Fix reference commit scope entry Signed-off-by: Dj Walker-Morgan --- package.json | 2 +- product_docs/docs/pgd/5/reference/index.mdx | 2 +- product_docs/docs/pgd/5/reference/pgd-settings.mdx | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/package.json b/package.json index 7e83f88a581..1a5638a03aa 100644 --- a/package.json +++ b/package.json @@ -111,7 +111,7 @@ }, "lint-staged": { "*.js": "prettier --write" - }, + }, "repository": { "type": "git", "url": "https://github.com/EnterpriseDB/docs" diff --git a/product_docs/docs/pgd/5/reference/index.mdx b/product_docs/docs/pgd/5/reference/index.mdx index df12a692546..106bbe98e13 100644 --- a/product_docs/docs/pgd/5/reference/index.mdx +++ b/product_docs/docs/pgd/5/reference/index.mdx @@ -183,7 +183,7 @@ The reference section is a definitive listing of all functions, views and comman * [`bdr.min_worker_backoff_delay`](pgd-settings#bdrmin_worker_backoff_delay) ### [CRDTs](pgd-settings#crdts) * [`bdr.crdt_raw_value`](pgd-settings#bdrcrdt_raw_value) -### [Commit Scopes (current/default)](pgd-settings#commit-scopes-(current/default)) +### [Commit Scopes](pgd-settings#commit-scopes) * [`bdr.commit_scope`](pgd-settings#bdrcommit_scope) ### [Commit At Most Once](pgd-settings#commit-at-most-once) * [`bdr.camo_local_mode_delay`](pgd-settings#bdrcamo_local_mode_delay) diff --git a/product_docs/docs/pgd/5/reference/pgd-settings.mdx b/product_docs/docs/pgd/5/reference/pgd-settings.mdx index d69fc7d671a..d663c4121f4 100644 --- a/product_docs/docs/pgd/5/reference/pgd-settings.mdx +++ b/product_docs/docs/pgd/5/reference/pgd-settings.mdx @@ -366,7 +366,7 @@ value of the base CRDT type (for example, a bigint for `crdt_pncounter`). When set to `on`, the returned value represents the full representation of the CRDT value, which can, for example, include the state from multiple nodes. -## Commit Scopes (current/default) +## Commit Scopes ### `bdr.commit_scope` From add4876a2fbf69653a5003d1d139ca3b783c6c5b Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Thu, 30 Nov 2023 19:51:53 +0000 Subject: [PATCH 62/68] Link fixes Signed-off-by: Dj Walker-Morgan --- product_docs/docs/pgd/5/choosing_server.mdx | 2 +- product_docs/docs/pgd/5/durability/comparing.mdx | 2 +- product_docs/docs/pgd/5/durability/legacy-sync.mdx | 4 +--- product_docs/docs/pgd/5/durability/overview.mdx | 2 +- product_docs/docs/pgd/5/reference/commit-scopes.mdx | 4 ++-- 5 files changed, 6 insertions(+), 8 deletions(-) diff --git a/product_docs/docs/pgd/5/choosing_server.mdx b/product_docs/docs/pgd/5/choosing_server.mdx index 83629330505..0bb2dd938dd 100644 --- a/product_docs/docs/pgd/5/choosing_server.mdx +++ b/product_docs/docs/pgd/5/choosing_server.mdx @@ -23,7 +23,7 @@ The following table lists features of EDB Postgres Distributed that are dependen | [Transform triggers](striggers/#transform-triggers) | Y | Y | Y | | [Conflict triggers](striggers/#conflict-triggers) | Y | Y | Y | | [Asynchronous replication](durability/) | Y | Y | Y | -| [Legacy synchronous replication](durability/#legacy-synchronous-replication-using-pgd) | Y | Y | Y | +| [Legacy synchronous replication](durability/legacy-sync) | Y | Y | Y | | [Group Commit](durability/group-commit/) | N | Y | 14+ | | [Commit At Most Once (CAMO)](durability/camo/) | N | Y | 14+ | | [Eager Conflict Resolution](consistency/eager/) | N | Y | 14+ | diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index c6ce7506778..4015c5673dc 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -10,7 +10,7 @@ Most options for synchronous replication available to PGD allow for different levels of synchronization, offering different tradeoffs between performance and protection against node or network outages. -The following list of [confirmation levels](commit-scopes-rules#the-confirmation-level) explains +The following list of [confirmation levels](commit-scope-rules#the-confirmation-level) explains what a user should expect to see when that confirmation level in in effect and how that can effect performance, durability, and consistency. ### ON RECEIVED diff --git a/product_docs/docs/pgd/5/durability/legacy-sync.mdx b/product_docs/docs/pgd/5/durability/legacy-sync.mdx index 750b05056c4..70edc908521 100644 --- a/product_docs/docs/pgd/5/durability/legacy-sync.mdx +++ b/product_docs/docs/pgd/5/durability/legacy-sync.mdx @@ -13,9 +13,7 @@ replication](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING variant](https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION). For backward compatibility, PGD still supports configuring synchronous -replication with `synchronous_commit` and `synchronous_standby_names`. See -[Legacy synchronous replication](#legacy-synchronous-replication-using-pgd), but -consider using [Group Commit](group-commit) instead. +replication with `synchronous_commit` and `synchronous_standby_names`. Consider using [Group Commit](group-commit) or [Synchronous Commit](synchronous_commit) instead instead. Unlike PGD replication options, PSR sync will persist first, replicating after the WAL flush of commit record. diff --git a/product_docs/docs/pgd/5/durability/overview.mdx b/product_docs/docs/pgd/5/durability/overview.mdx index b6b5f3ba7da..a71f4b51b57 100644 --- a/product_docs/docs/pgd/5/durability/overview.mdx +++ b/product_docs/docs/pgd/5/durability/overview.mdx @@ -48,7 +48,7 @@ through before retrying. - [Lag Control](lag-control): This kind of commit scope controls how far behind nodes can be in terms of replication before allowing commit to proceed. -- [Synchronous Commit](synchronous-commit): This kind of commit scope allows for a behaviour where the origin node awaits a majority of nodes to confirm and behaves more like a native Postgres synchronous commit. +- [Synchronous Commit](synchronous_commit): This kind of commit scope allows for a behaviour where the origin node awaits a majority of nodes to confirm and behaves more like a native Postgres synchronous commit. !!! Note Legacy synchronization availability diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index 9f9995551d9..4454b5c0582 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -127,7 +127,7 @@ The value `async` means resolve conflicts asynchronously during replication usin The value `eager` means that conflicts are resolved eagerly during COMMIT by aborting one of the conflicting transactions. -See [Group Commit/Conflict Resolution](group-commit#conflict_resolution) +See [Group Commit/Conflict Resolution](../durability/group-commit/#conflict-resolution) ### commit_decision settings @@ -137,7 +137,7 @@ The value `partner` means the partner node decides whether transactions can be The value `raft` means the COMMIT decision is done using Raft consensus independent of `commit_scope_group` consensus. -See [Group Commit/Commit Decision](group-commit#commit_decision) +See [Group Commit/Commit Decision](../durability/group-commit/#commit-decisions) From 56810c313c193f2f8fd1190a5b743b565c620ad5 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 4 Dec 2023 11:46:45 +0000 Subject: [PATCH 63/68] Review Edits Signed-off-by: Dj Walker-Morgan --- .../pgd/5/durability/commit-scope-rules.mdx | 24 ++++++-- .../docs/pgd/5/reference/commit-scopes.mdx | 34 ++++++++--- product_docs/docs/pgd/5/reference/index.json | 49 ++++++++-------- product_docs/docs/pgd/5/reference/index.mdx | 56 ++++++++++--------- .../docs/pgd/5/reference/index.mdx.src | 1 + .../docs/pgd/5/reference/pgd-settings.mdx | 3 +- 6 files changed, 103 insertions(+), 64 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx index 6f3fb3f4bd5..a11506714d1 100644 --- a/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx +++ b/product_docs/docs/pgd/5/durability/commit-scope-rules.mdx @@ -4,7 +4,7 @@ title: Commit scope rules Commit scope rules are at the core of the commit scope mechanism. They define what the commit scope will enforce. -Commit scope rules are composed of one or more operations combined with an `AND`. +Commit scope rules are composed of one or more operations combined with an `AND`. Each operation is made up of two (or three) parts, the commit scope group, an optional confirmation level, and the kind of commit scope (which may have it's own parameters). @@ -34,11 +34,15 @@ The last part of this operation is the commit scope kind, which here is `GROUP C ## The commit scope group -There are three kinds of commit scope group, `ANY`, `ALL` and `MAJORITY`. They are all followed by a parenthesed list of one or more groups, which combine to make a pool of nodes that this operation will apply to. +There are three kinds of commit scope group, `ANY`, `ALL` and `MAJORITY`. They are all followed by a parenthesed list of one or more groups, which combine to make a pool of nodes that this operation will apply to. This list can be preceded by `NOT` which inverts to pool to all other groups apart from those in the list. + +- `ANY n` — is followed by an integer value, "n". It translates to any "n" nodes in the listed groups nodes. +- `ALL` — is followed by the groups and translates to all nodes in the listed groups nodes. +- `MAJORITY` — is followed by the groups and translates to requiring a half, plus one, of the listed groups nodes to confirm to give a majority. +- `ANY n NOT` — is followed by an integer value, "n". It translates to any "n" nodes that are not in the listed groups nodes. +- `ALL NOT` — is followed by the groups and translates to all nodes that are not in the listed group's nodes. +- `MAJORITY NOT` — is followed by the groups and translates to requiring a half, plus one, of the nodes that are not in the listed groups nodes to confirm to give a majority. -- `ANY` &mdash is followed by an integer value, "n". It translates to any "n" nodes in the listed groups nodes. -- `ALL` &mdash is followed by the groups and translates to all nodes in the listed groups noes. -- `MAJORITY` &mdsh is followed by the groups and translates to requiring a half, plus one, of the listed groups nodes to confirm to give a majority. ## The confirmation Level @@ -87,4 +91,14 @@ Synchronous Commit is a commit scope option which is designed to be like the leg For further details see [`SYNCHRONOUS_COMMIT`](synchronous_commit) +## Combining rules + +A rule can have multiple operations, combined with an `AND` to form a single rule. For example, + +``` +MAJORITY (Region_A) SYNCHRONOUS_COMMIT AND ANY 1 (Region_A) LAG CONTROL (MAX_LAG_SIZE = '50MB') +``` + +The first operation sets up a synchronous commit against a majority of `Region_A`, the second operation adds lag control which will start pushing the commit delay up when any one of the nodes in `Region_A` has more than 50MB of lag. This combination of operations allows the lag control to operate when any node is lagging. + diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index 4454b5c0582..5c700b88a4b 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -23,9 +23,9 @@ commit_scope_operation: commit_scope_group confirmation_level commit_scope_kind commit_scope_group: -{ ANY num (node_group [, ...]) - | MAJORITY (node_group [, ...]) - | ALL (node_group [, ...]) } +{ ANY num [NOT] (node_group [, ...]) + | MAJORITY [NOT] (node_group [, ...]) + | ALL [NOT] (node_group [, ...]) } confirmation_level: [ ON { received|replicated|durable|visible } ] @@ -44,21 +44,39 @@ where `node_group` is the name of a PGD data node group. ### ANY -Example: `ANY 2 left_dc` +Example: `ANY 2 (left_dc)` -A transaction under this commit scope group will be considered commited if any two nodes in the left_dc group confirm they have processed the transaction. +A transaction under this commit scope group will be considered commited if any two nodes in the left_dc group confirm they have processed the transaction. + +### ANY NOT + +Example: `ANY 2 NOT (left_dc)` + +A transaction under this commit scope group will be considered commited if any two nodes that are not in the left_dc group confirm they have processed the transaction. ### MAJORITY -Example: `MAJORITY left_dc` +Example: `MAJORITY (left_dc)` A transaction under this commit scope group will be considered commited if a majority of the nodes in the left_dc group confirm they have processed the transaction. +### MAJORITY NOT + +Example: `MAJORITY NOT (left_dc)` + +A transaction under this commit scope group will be considered commited if a majority of the nodes that are not in the left_dc group confirm they have processed the transaction. + ### ALL -Example: `ALL left_dc` +Example: `ALL (left_dc)` + +A transaction under this commit scope group will be considered commited if all of the nodes in the left_dc group confirm they have processed the transaction. + +### ALL NOT + +Example: `ALL NOT (left_dc)` -A transaction under this commit scope group will be considered commited if all of the nodes in the left_dc group confirm they have processed the transaction. +A transaction under this commit scope group will be considered commited if all of the nodes that are not in the left_dc group confirm they have processed the transaction. ## Confirmation level diff --git a/product_docs/docs/pgd/5/reference/index.json b/product_docs/docs/pgd/5/reference/index.json index d28b86cfa20..7f7761f87ff 100644 --- a/product_docs/docs/pgd/5/reference/index.json +++ b/product_docs/docs/pgd/5/reference/index.json @@ -188,6 +188,31 @@ "bdralter_proxy_option": "/pgd/latest/reference/routing#bdralter_proxy_option", "bdrdrop_proxy": "/pgd/latest/reference/routing#bdrdrop_proxy", "bdrrouting_leadership_transfer": "/pgd/latest/reference/routing#bdrrouting_leadership_transfer", + "cs.commit-scope-syntax": "/pgd/latest/reference/commit-scopes#commit-scope-syntax", + "cs.commit-scope-groups": "/pgd/latest/reference/commit-scopes#commit-scope-groups", + "cs.any": "/pgd/latest/reference/commit-scopes#any", + "cs.any-not": "/pgd/latest/reference/commit-scopes#any-not", + "cs.majority": "/pgd/latest/reference/commit-scopes#majority", + "cs.majority-not": "/pgd/latest/reference/commit-scopes#majority-not", + "cs.all": "/pgd/latest/reference/commit-scopes#all", + "cs.all-not": "/pgd/latest/reference/commit-scopes#all-not", + "cs.confirmation-level": "/pgd/latest/reference/commit-scopes#confirmation-level", + "cs.on-received": "/pgd/latest/reference/commit-scopes#on-received", + "cs.on-replicated": "/pgd/latest/reference/commit-scopes#on-replicated", + "cs.on-durable": "/pgd/latest/reference/commit-scopes#on-durable", + "cs.on-visible": "/pgd/latest/reference/commit-scopes#on-visible", + "cs.commit-scope-kinds": "/pgd/latest/reference/commit-scopes#commit-scope-kinds", + "cs.group-commit": "/pgd/latest/reference/commit-scopes#group-commit", + "cs.group-commit-parameters": "/pgd/latest/reference/commit-scopes#group-commit-parameters", + "cs.abort-on-parameters": "/pgd/latest/reference/commit-scopes#abort-on-parameters", + "cs.transaction_tracking-settings": "/pgd/latest/reference/commit-scopes#transaction_tracking-settings", + "cs.conflict_resolution-settings": "/pgd/latest/reference/commit-scopes#conflict_resolution-settings", + "cs.commit_decision-settings": "/pgd/latest/reference/commit-scopes#commit_decision-settings", + "cs.camo": "/pgd/latest/reference/commit-scopes#camo", + "cs.degrade-on-parameters": "/pgd/latest/reference/commit-scopes#degrade-on-parameters", + "cs.lag-control": "/pgd/latest/reference/commit-scopes#lag-control", + "cs.lag-control-parameters": "/pgd/latest/reference/commit-scopes#lag-control-parameters", + "cs.synchronous_commit": "/pgd/latest/reference/commit-scopes#synchronous_commit", "bdrcreate_replication_set": "/pgd/latest/reference/repsets-management#bdrcreate_replication_set", "bdralter_replication_set": "/pgd/latest/reference/repsets-management#bdralter_replication_set", "bdrdrop_replication_set": "/pgd/latest/reference/repsets-management#bdrdrop_replication_set", @@ -288,27 +313,5 @@ "bdrbdr_get_commit_decisions": "/pgd/latest/reference/functions-internal#bdrbdr_get_commit_decisions", "bdrshow_workers": "/pgd/latest/reference/functions-internal#bdrshow_workers", "bdrshow_writers": "/pgd/latest/reference/functions-internal#bdrshow_writers", - "bdrnode_kind_name": "/pgd/latest/reference/functions-internal#bdrnode_kind_name", - "cs.commit-scope-syntax": "/pgd/latest/reference/commit-scopes#commit-scope-syntax", - "cs.commit-scope-groups": "/pgd/latest/reference/commit-scopes#commit-scope-groups", - "cs.any": "/pgd/latest/reference/commit-scopes#any", - "cs.majority": "/pgd/latest/reference/commit-scopes#majority", - "cs.all": "/pgd/latest/reference/commit-scopes#all", - "cs.confirmation-level": "/pgd/latest/reference/commit-scopes#confirmation-level", - "cs.on-received": "/pgd/latest/reference/commit-scopes#on-received", - "cs.on-replicated": "/pgd/latest/reference/commit-scopes#on-replicated", - "cs.on-durable": "/pgd/latest/reference/commit-scopes#on-durable", - "cs.on-visible": "/pgd/latest/reference/commit-scopes#on-visible", - "cs.commit-scope-kinds": "/pgd/latest/reference/commit-scopes#commit-scope-kinds", - "cs.group-commit": "/pgd/latest/reference/commit-scopes#group-commit", - "cs.group-commit-parameters": "/pgd/latest/reference/commit-scopes#group-commit-parameters", - "cs.abort-on-parameters": "/pgd/latest/reference/commit-scopes#abort-on-parameters", - "cs.transaction_tracking-settings": "/pgd/latest/reference/commit-scopes#transaction_tracking-settings", - "cs.conflict_resolution-settings": "/pgd/latest/reference/commit-scopes#conflict_resolution-settings", - "cs.commit_decision-settings": "/pgd/latest/reference/commit-scopes#commit_decision-settings", - "cs.camo": "/pgd/latest/reference/commit-scopes#camo", - "cs.degrade-on-parameters": "/pgd/latest/reference/commit-scopes#degrade-on-parameters", - "cs.lag-control": "/pgd/latest/reference/commit-scopes#lag-control", - "cs.lag-control-parameters": "/pgd/latest/reference/commit-scopes#lag-control-parameters", - "cs.synchronous_commit": "/pgd/latest/reference/commit-scopes#synchronous_commit" + "bdrnode_kind_name": "/pgd/latest/reference/functions-internal#bdrnode_kind_name" } \ No newline at end of file diff --git a/product_docs/docs/pgd/5/reference/index.mdx b/product_docs/docs/pgd/5/reference/index.mdx index 106bbe98e13..01f828b885d 100644 --- a/product_docs/docs/pgd/5/reference/index.mdx +++ b/product_docs/docs/pgd/5/reference/index.mdx @@ -11,6 +11,7 @@ navigation: - nodes - nodes-management-interfaces - routing +- commit-scopes - repsets-management - repsets-membership - repsets-ddl-filtering @@ -183,7 +184,7 @@ The reference section is a definitive listing of all functions, views and comman * [`bdr.min_worker_backoff_delay`](pgd-settings#bdrmin_worker_backoff_delay) ### [CRDTs](pgd-settings#crdts) * [`bdr.crdt_raw_value`](pgd-settings#bdrcrdt_raw_value) -### [Commit Scopes](pgd-settings#commit-scopes) +### [Commit Scope](pgd-settings#commit-scope) * [`bdr.commit_scope`](pgd-settings#bdrcommit_scope) ### [Commit At Most Once](pgd-settings#commit-at-most-once) * [`bdr.camo_local_mode_delay`](pgd-settings#bdrcamo_local_mode_delay) @@ -262,6 +263,34 @@ The reference section is a definitive listing of all functions, views and comman * [`bdr.routing_leadership_transfer`](routing#bdrrouting_leadership_transfer) +## [Commit scopes](commit-scopes) + * [Commit scope syntax](commit-scopes#commit-scope-syntax) + * [Commit scope groups](commit-scopes#commit-scope-groups) + * [ANY](commit-scopes#any) + * [ANY NOT](commit-scopes#any-not) + * [MAJORITY](commit-scopes#majority) + * [MAJORITY NOT](commit-scopes#majority-not) + * [ALL](commit-scopes#all) + * [ALL NOT](commit-scopes#all-not) + * [Confirmation level](commit-scopes#confirmation-level) + * [ON received](commit-scopes#on-received) + * [ON replicated](commit-scopes#on-replicated) + * [ON durable](commit-scopes#on-durable) + * [ON visible](commit-scopes#on-visible) + * [Commit Scope kinds](commit-scopes#commit-scope-kinds) + * [GROUP COMMIT](commit-scopes#group-commit) + * [GROUP COMMIT parameters](commit-scopes#group-commit-parameters) + * [ABORT ON parameters](commit-scopes#abort-on-parameters) + * [transaction_tracking settings](commit-scopes#transaction_tracking-settings) + * [conflict_resolution settings](commit-scopes#conflict_resolution-settings) + * [commit_decision settings](commit-scopes#commit_decision-settings) + * [CAMO](commit-scopes#camo) + * [Degrade On parameters](commit-scopes#degrade-on-parameters) + * [LAG CONTROL](commit-scopes#lag-control) + * [LAG CONTROL parameters](commit-scopes#lag-control-parameters) + * [SYNCHRONOUS_COMMIT](commit-scopes#synchronous_commit) + + ## [Replication set management](repsets-management) * [`bdr.create_replication_set`](repsets-management#bdrcreate_replication_set) * [`bdr.alter_replication_set`](repsets-management#bdralter_replication_set) @@ -400,28 +429,3 @@ The reference section is a definitive listing of all functions, views and comman * [`bdr.show_workers`](functions-internal#bdrshow_workers) * [`bdr.show_writers`](functions-internal#bdrshow_writers) * [`bdr.node_kind_name`](functions-internal#bdrnode_kind_name) - - -## [Commit scopes](commit-scopes) - * [Commit scope syntax](commit-scopes#commit-scope-syntax) - * [Commit scope groups](commit-scopes#commit-scope-groups) - * [ANY](commit-scopes#any) - * [MAJORITY](commit-scopes#majority) - * [ALL](commit-scopes#all) - * [Confirmation level](commit-scopes#confirmation-level) - * [ON received](commit-scopes#on-received) - * [ON replicated](commit-scopes#on-replicated) - * [ON durable](commit-scopes#on-durable) - * [ON visible](commit-scopes#on-visible) - * [Commit Scope kinds](commit-scopes#commit-scope-kinds) - * [GROUP COMMIT](commit-scopes#group-commit) - * [GROUP COMMIT parameters](commit-scopes#group-commit-parameters) - * [ABORT ON parameters](commit-scopes#abort-on-parameters) - * [transaction_tracking settings](commit-scopes#transaction_tracking-settings) - * [conflict_resolution settings](commit-scopes#conflict_resolution-settings) - * [commit_decision settings](commit-scopes#commit_decision-settings) - * [CAMO](commit-scopes#camo) - * [Degrade On parameters](commit-scopes#degrade-on-parameters) - * [LAG CONTROL](commit-scopes#lag-control) - * [LAG CONTROL parameters](commit-scopes#lag-control-parameters) - * [SYNCHRONOUS_COMMIT](commit-scopes#synchronous_commit) diff --git a/product_docs/docs/pgd/5/reference/index.mdx.src b/product_docs/docs/pgd/5/reference/index.mdx.src index b95a72d417c..191d7650595 100644 --- a/product_docs/docs/pgd/5/reference/index.mdx.src +++ b/product_docs/docs/pgd/5/reference/index.mdx.src @@ -11,6 +11,7 @@ navigation: - nodes - nodes-management-interfaces - routing +- commit-scopes - repsets-management - repsets-membership - repsets-ddl-filtering diff --git a/product_docs/docs/pgd/5/reference/pgd-settings.mdx b/product_docs/docs/pgd/5/reference/pgd-settings.mdx index d663c4121f4..01d62c59211 100644 --- a/product_docs/docs/pgd/5/reference/pgd-settings.mdx +++ b/product_docs/docs/pgd/5/reference/pgd-settings.mdx @@ -366,7 +366,7 @@ value of the base CRDT type (for example, a bigint for `crdt_pncounter`). When set to `on`, the returned value represents the full representation of the CRDT value, which can, for example, include the state from multiple nodes. -## Commit Scopes +## Commit Scope ### `bdr.commit_scope` @@ -381,7 +381,6 @@ The commit delay that applies in CAMO's asynchronous mode to emulate the overhead that normally occurs with the CAMO partner having to confirm transactions. Defaults to 5 ms. Set to `0` to disable this feature. - - ### `bdr.camo_enable_client_warnings` Emit warnings if an activity is carried out in the database for which CAMO From f3e17cf26a7be3e0363dd04f3cc5fc3e4766a6e8 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Mon, 4 Dec 2023 12:13:24 +0000 Subject: [PATCH 64/68] GH review comments resolved Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/comparing.mdx | 2 +- product_docs/docs/pgd/5/durability/index.mdx | 7 +++--- .../docs/pgd/5/durability/legacy-sync.mdx | 24 +++++++++---------- .../docs/pgd/5/durability/limitations.mdx | 8 +++---- .../docs/pgd/5/durability/overview.mdx | 4 ++-- 5 files changed, 21 insertions(+), 24 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index 4015c5673dc..9cdb157d93f 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -1,6 +1,6 @@ --- title: Comparing Durability Options -navTitle: Comparing +navTitle: Comparing Durability Options deepToC: true --- diff --git a/product_docs/docs/pgd/5/durability/index.mdx b/product_docs/docs/pgd/5/durability/index.mdx index c76efccc521..9fafe023e5b 100644 --- a/product_docs/docs/pgd/5/durability/index.mdx +++ b/product_docs/docs/pgd/5/durability/index.mdx @@ -48,10 +48,9 @@ a commit scope rule. * [Group Commit](group-commit) focuses on the Group Commit option, where you can define a transaction as done when a group of nodes agrees it's done. -* [CAMO](camo) focuses on the Commit At Most Once option, which allows -applications to take on the responsibility of verifying that a commit has been -successful before retying. This ensures that their commits only happen at most -once. +* [CAMO](camo) focuses on the Commit At Most Once option, in which applications +take responsibility for verifying that a transaction has been committed before +retrying. This ensures that their commits only happen at most once. * [Lag Control](lag-control) looks at the commit scope mechanism which dynamically throttle nodes according to the slowest node and regulates how far diff --git a/product_docs/docs/pgd/5/durability/legacy-sync.mdx b/product_docs/docs/pgd/5/durability/legacy-sync.mdx index 70edc908521..83886df24ab 100644 --- a/product_docs/docs/pgd/5/durability/legacy-sync.mdx +++ b/product_docs/docs/pgd/5/durability/legacy-sync.mdx @@ -13,7 +13,7 @@ replication](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING variant](https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION). For backward compatibility, PGD still supports configuring synchronous -replication with `synchronous_commit` and `synchronous_standby_names`. Consider using [Group Commit](group-commit) or [Synchronous Commit](synchronous_commit) instead instead. +replication with `synchronous_commit` and `synchronous_standby_names`. Consider using [Group Commit](group-commit) or [Synchronous Commit](synchronous_commit) instead. Unlike PGD replication options, PSR sync will persist first, replicating after the WAL flush of commit record. @@ -64,9 +64,9 @@ Postgres crashes.* ### Postgres configuration parameters -The following table provides an overview of the configuration -settings that you must set to a non-default value (req) and those that are -optional (opt) but affecting a specific variant. +The following table provides an overview of the configuration settings that you +must set to a non-default value (req) and those that are optional (opt) but +affecting a specific variant. | Setting (GUC) | Group Commit | Lag Control | PSR | Legacy Sync | |--------------------------------------|:------------:|:-----------:|:-------:|:-----------:| @@ -82,13 +82,11 @@ You configure the Group Commit feature of PGD independent of instead of configuring `synchronous_standby_names` on each node individually, Group Commit uses globally synchronized commit scopes. -!!! Note -While the grammar for `synchronous_standby_names` and commit -scopes looks similar, the former -doesn't account for the origin node, but the latter does. -Therefore, for example, `synchronous_standby_names = 'ANY 1 (..)'` -is equivalent to a commit scope of `ANY 2 (...)`. This choice -makes reasoning about majority easier and reflects that the origin -node also contributes to the durability and visibility of the -transaction. +!!! Note +While the grammar for `synchronous_standby_names` and commit scopes +looks similar, the former doesn't account for the origin node, but the latter +does. Therefore, for example, `synchronous_standby_names = 'ANY 1 (..)'` is +equivalent to a commit scope of `ANY 2 (...)`. This choice makes reasoning about +majority easier and reflects that the origin node also contributes to the +durability and visibility of the transaction. !!! diff --git a/product_docs/docs/pgd/5/durability/limitations.mdx b/product_docs/docs/pgd/5/durability/limitations.mdx index 5603a0e53c7..d1ee0319903 100644 --- a/product_docs/docs/pgd/5/durability/limitations.mdx +++ b/product_docs/docs/pgd/5/durability/limitations.mdx @@ -58,10 +58,10 @@ applications committing more than once. If you use this feature, take these limitations into account when planning: - CAMO is designed to query the results of a recently failed COMMIT on the -origin node. So, in case of disconnection, code the application to immediately -request the transaction status from the CAMO partner. Have as little delay as -possible after the failure before requesting the status. Applications must not -rely on CAMO decisions being stored for longer than 15 minutes. +origin node. In case of disconnection the application must request the +transaction status from the CAMO partner. Ensure that you have as little delay +as possible after the failure before requesting the status. Applications must +not rely on CAMO decisions being stored for longer than 15 minutes. - If the application forgets the global identifier assigned, for example, as a result of a restart, there's no easy way to recover diff --git a/product_docs/docs/pgd/5/durability/overview.mdx b/product_docs/docs/pgd/5/durability/overview.mdx index a71f4b51b57..55c58fa6b2c 100644 --- a/product_docs/docs/pgd/5/durability/overview.mdx +++ b/product_docs/docs/pgd/5/durability/overview.mdx @@ -42,8 +42,8 @@ Commit scopes allow four kinds of controlling durability of the transaction: allows you to control the visibility ordering of the transaction. - [CAMO](camo): This kind of commit scope is a variant of Group Commit in -which enables the client take on the responsibility of verifying a commit went -through before retrying. + which the client takes on the responsibility for verifying that a transaction + has been committed before retrying. - [Lag Control](lag-control): This kind of commit scope controls how far behind nodes can be in terms of replication before allowing commit to proceed. From 7f5a3aeae93c5bc1d1b054e1e3eec5e8949d45c7 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Tue, 5 Dec 2023 10:57:11 +0000 Subject: [PATCH 65/68] Fixes for review comments from pjmodos Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/comparing.mdx | 41 +++++++++++-------- .../docs/pgd/5/durability/group-commit.mdx | 8 ++-- .../docs/pgd/5/durability/legacy-sync.mdx | 13 +++--- .../docs/pgd/5/durability/overview.mdx | 8 ++-- .../pgd/5/durability/synchronous_commit.mdx | 4 +- .../docs/pgd/5/reference/commit-scopes.mdx | 2 +- 6 files changed, 42 insertions(+), 34 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index 9cdb157d93f..369fb0811a4 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -27,42 +27,49 @@ storage. **For**: Good performance. -**Against**: Less robust when a crash occurs. +**Against**: Not robust; Data can be lost when either a Postgres server or operating system crash occurs. ### ON REPLICATED **Expect**: The peer node has received the changes and applied them to the tuple -store. The changes have not been persisted to storage. +store. The changes have been written to storage, but the storage has not been flushed +to disk. -Confirmation on replication means the peer has received and applied the changes, but +Confirmation on replication means the peer has received and applied the changes. Those changes +have been written to storage, but will still be in operating system caches and buffers. The system has yet to persist them to disk. -**For**: A better checkpoint than received with less performance. +**For**: A better checkpoint than received with less performance. -**Against**: Less robust when a crash occurs. +**Against**: Slightly more robust than RECEIVED; Replicated data can survive a Postgres crash but will not survive an operating system crash. ### ON DURABLE -**Expect**: The peer node has received the changes, applied them to the tuple store and persisted the changes to storage. It has yet to make the changes available to other concurrent transactions. +**Expect**: The peer node has received the changes, applied them to the tuple +store and persisted the changes to storage. It has yet to make the changes +available to other sessions. -Durable confirmation means that the transaction has been written to the peer -node's storage. This provides protection against loss of data after a crash and -recovery of the peer node. With Group Commit, this also means the changes are -visible. +Durable confirmation means that the transaction has been written and flushed to +the peer node's storage. This provides protection against loss of data after a +crash and recovery of the peer node. But, if a session commits a transaction +with an ON DURABLE rule before disconnecting and reconnects, the transaction's +changes are not guaranteed to be visible to the reconnected session. + +When used with the Group Commit commit scope kind, this also means the changes +are visible. **For**: More robust, able to recover without retransmission in the event of a crash. -**Against**: Doesn't guarantee consistency. +**Against**: Doesn't guarantee consistency in cases of failover. ### ON VISIBLE -**Expect**: The peer node has recieved and applied the changes, and persisted them to storage, and has now made them visible to other concurrent transactions. +**Expect**: The peer node has recieved and applied the changes, persisted and flushed those changes to storage. -Confirmation on visibility means that the transaction was applied remotely. All -other clients see the results of the transaction on all nodes, providing this -guarantee immediately after the commit is confirmed by the origin node. Without -visibility, other clients connected might not see the results of the transaction -and experience stale reads. +Confirmation on visibility means that the transaction was fully applied +remotely. If a session commits a transaction with an ON VISIBLE rule before +disconnecting and reconnects, the transaction's changes are guaranteed to be +visible to the reconnected session. **For**: Robust and consistent. diff --git a/product_docs/docs/pgd/5/durability/group-commit.mdx b/product_docs/docs/pgd/5/durability/group-commit.mdx index 9e07b75a2b2..6a50a091ef9 100644 --- a/product_docs/docs/pgd/5/durability/group-commit.mdx +++ b/product_docs/docs/pgd/5/durability/group-commit.mdx @@ -76,11 +76,9 @@ to commit something. This approach requires application changes to use the CAMO transaction protocol to work correctly, as the application is in some way part of the consensus. For more on this approach, see [CAMO](camo). -The `raft` option uses the PGD's built-in Raft consensus to decide whether -commit can happen. Currently, the global Raft is used. For this to work, the -majority of nodes across the whole cluster must work. This option means that all -commits are effectively ordered and every node moves forward in step with those -commits at the same rate. +The `raft` option is for backwards compatibility only. It uses PGD's built-in Raft +consensus for commit decisions. It is slow and only useful with eager conflict +resolution. ### Conflict resolution diff --git a/product_docs/docs/pgd/5/durability/legacy-sync.mdx b/product_docs/docs/pgd/5/durability/legacy-sync.mdx index 83886df24ab..9deaf40b0f1 100644 --- a/product_docs/docs/pgd/5/durability/legacy-sync.mdx +++ b/product_docs/docs/pgd/5/durability/legacy-sync.mdx @@ -4,20 +4,23 @@ navTitle: Legacy synchronous replication --- !!! Warning Important -Consider using [Group Commit](group-commit) or [Synchronous Commit](synchronous_commit) instead. +We highly recommend [PGD Synchronous Commit](synchronous-commit) instead of +legacy synchronous replication. !!! + Postgres provides [physical streaming replication](https://www.postgresql.org/docs/current/warm-standby.html#STREAMING-REPLICATION) (PSR), which is unidirectional but offers a [synchronous variant](https://www.postgresql.org/docs/current/warm-standby.html#SYNCHRONOUS-REPLICATION). For backward compatibility, PGD still supports configuring synchronous -replication with `synchronous_commit` and `synchronous_standby_names`. Consider using [Group Commit](group-commit) or [Synchronous Commit](synchronous_commit) instead. - -Unlike PGD replication options, PSR sync will persist first, replicating after the WAL flush of commit record. - +replication with `synchronous_commit` and `synchronous_standby_names`. Consider +using [Group Commit](group-commit) or [Synchronous Commit](synchronous_commit) +instead. +Unlike PGD replication options, PSR sync will persist first, replicating after +the WAL flush of commit record. ### Usage diff --git a/product_docs/docs/pgd/5/durability/overview.mdx b/product_docs/docs/pgd/5/durability/overview.mdx index 55c58fa6b2c..50760607ce4 100644 --- a/product_docs/docs/pgd/5/durability/overview.mdx +++ b/product_docs/docs/pgd/5/durability/overview.mdx @@ -48,14 +48,14 @@ Commit scopes allow four kinds of controlling durability of the transaction: - [Lag Control](lag-control): This kind of commit scope controls how far behind nodes can be in terms of replication before allowing commit to proceed. -- [Synchronous Commit](synchronous_commit): This kind of commit scope allows for a behaviour where the origin node awaits a majority of nodes to confirm and behaves more like a native Postgres synchronous commit. +- [PGD Synchronous Commit](synchronous_commit): This kind of commit scope allows for a behaviour where the origin node awaits a majority of nodes to confirm and behaves more like a native Postgres synchronous commit. -!!! Note Legacy synchronization availability +!!! Note Legacy synchronization availability For backward compatibility, PGD still supports configuring synchronous replication with `synchronous_commit` and `synchronous_standby_names`. See -[Legacy synchronous replication](legacy-sync) for more on this option, -but consider using [Group Commit](group-commit) or [Synchronous_Commit](synchronous_commit). +[Legacy synchronous replication](legacy-sync) for more on this option. We +recommend users instead use [PGD Synchronous_Commit](synchronous_commit). !!! diff --git a/product_docs/docs/pgd/5/durability/synchronous_commit.mdx b/product_docs/docs/pgd/5/durability/synchronous_commit.mdx index 7460120a60c..6ff9ad7c43f 100644 --- a/product_docs/docs/pgd/5/durability/synchronous_commit.mdx +++ b/product_docs/docs/pgd/5/durability/synchronous_commit.mdx @@ -1,5 +1,5 @@ --- -title: Synchronous Commit +title: PGD Synchronous Commit deepToC: true --- @@ -7,7 +7,7 @@ Commit scope kind: `SYNCHRONOUS_COMMIT` ## Overview -`SYNCHRONOUS_COMMIT` is a commit scope kind that works in a way that is more like PostgreSQL's [Synchronous commit](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) option in its underlying operation. Unlike the PostgreSQL option though, it is configured as a commit scope and is easier to configure and interact with within PGD. +PGD's `SYNCHRONOUS_COMMIT` is a commit scope kind that works in a way that is more like PostgreSQL's [Synchronous commit](https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-SYNCHRONOUS-COMMIT) option in its underlying operation. Unlike the PostgreSQL option though, it is configured as a commit scope and is easier to configure and interact with within PGD. Unlike other commit scope kinds such as GROUP COMMIT and CAMO, the transactions in a SYNCHRONOUS_COMMIT operation will not be transformed into a two phase commit (2PC) transaction, but work more like a Postgres Synchronous commit. diff --git a/product_docs/docs/pgd/5/reference/commit-scopes.mdx b/product_docs/docs/pgd/5/reference/commit-scopes.mdx index 5c700b88a4b..19be9ee301a 100644 --- a/product_docs/docs/pgd/5/reference/commit-scopes.mdx +++ b/product_docs/docs/pgd/5/reference/commit-scopes.mdx @@ -153,7 +153,7 @@ The value `group` means the preceding `commit_scope_group` specification also af The value `partner` means the partner node decides whether transactions can be committed. This value is allowed only on groups with 2 data nodes. -The value `raft` means the COMMIT decision is done using Raft consensus independent of `commit_scope_group` consensus. +The value `raft` means the COMMIT decision is done using Raft consensus of all the nodes in the cluster, independent of any `commit_scope_group` consensus. This option has low performance and is for backwards compatibility only. See [Group Commit/Commit Decision](../durability/group-commit/#commit-decisions) From 3ab277a24bc1cb59611cb6bf155556b8487458db Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan Date: Fri, 8 Dec 2023 09:51:02 +0000 Subject: [PATCH 66/68] Edits to comparing Signed-off-by: Dj Walker-Morgan --- .../docs/pgd/5/durability/comparing.mdx | 35 +++++++++++-------- 1 file changed, 21 insertions(+), 14 deletions(-) diff --git a/product_docs/docs/pgd/5/durability/comparing.mdx b/product_docs/docs/pgd/5/durability/comparing.mdx index 369fb0811a4..b132313effe 100644 --- a/product_docs/docs/pgd/5/durability/comparing.mdx +++ b/product_docs/docs/pgd/5/durability/comparing.mdx @@ -11,7 +11,7 @@ levels of synchronization, offering different tradeoffs between performance and protection against node or network outages. The following list of [confirmation levels](commit-scope-rules#the-confirmation-level) explains -what a user should expect to see when that confirmation level in in effect and how that can effect performance, durability, and consistency. +what a user should expect to see when that confirmation level is in effect and how that can affect performance, durability, and consistency. ### ON RECEIVED @@ -19,15 +19,19 @@ what a user should expect to see when that confirmation level in in effect and h the peer nodes tuple store or written to storage. Confirmation on reception means that the peer operating normally can -eventually apply the transaction without requiring any further +eventually, apply the transaction without requiring any further communication, even in the face of a full or partial network outage. A crash of a peer node might still require retransmission of the transaction, as this confirmation doesn't involve persistent storage. -**For**: Good performance. +**For**: The origin node in the transaction only has to wait for the reception +of the transaction. Where transactions are large, it may improve the TPS +performance of the system. -**Against**: Not robust; Data can be lost when either a Postgres server or operating system crash occurs. +**Against**: An increased likelihood of stale reads. Overall, ON RECEIVED is not robust +because data can be lost when either a Postgres server or operating system crash +occurs. ### ON REPLICATED @@ -35,13 +39,16 @@ storage. store. The changes have been written to storage, but the storage has not been flushed to disk. -Confirmation on replication means the peer has received and applied the changes. Those changes -have been written to storage, but will still be in operating system caches and buffers. The system -has yet to persist them to disk. +Confirmation on replication means the peer has received and applied the changes. +Those changes have been written to storage, but will still be in operating +system caches and buffers. The system has yet to persist them to disk. -**For**: A better checkpoint than received with less performance. +**For**: This checkpoint is further down the timeline of transaction processing. +The origin node only waits for the transaction to be applied, but not persisted. -**Against**: Slightly more robust than RECEIVED; Replicated data can survive a Postgres crash but will not survive an operating system crash. +**Against**: There's a slightly lower chance of stale reads over ON RECEIVED. +Also, with ON REPLICATED data can survive a Postgres crash but will still not +survive an operating system crash. ### ON DURABLE @@ -50,9 +57,9 @@ store and persisted the changes to storage. It has yet to make the changes available to other sessions. Durable confirmation means that the transaction has been written and flushed to -the peer node's storage. This provides protection against loss of data after a +the peer node's storage. This protects against loss of data after a crash and recovery of the peer node. But, if a session commits a transaction -with an ON DURABLE rule before disconnecting and reconnects, the transaction's +with an ON DURABLE rule before disconnecting and reconnecting, the transaction's changes are not guaranteed to be visible to the reconnected session. When used with the Group Commit commit scope kind, this also means the changes @@ -64,11 +71,11 @@ are visible. ### ON VISIBLE -**Expect**: The peer node has recieved and applied the changes, persisted and flushed those changes to storage. +**Expect**: The peer node has received and applied the changes, persisted and flushed those changes to storage. -Confirmation on visibility means that the transaction was fully applied +Confirmation of visibility means that the transaction was fully applied remotely. If a session commits a transaction with an ON VISIBLE rule before -disconnecting and reconnects, the transaction's changes are guaranteed to be +disconnecting and reconnecting, the transaction's changes are guaranteed to be visible to the reconnected session. **For**: Robust and consistent. From 976d86a5030166de993a4a60502cb3f030d9bb75 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Tue, 12 Dec 2023 11:12:57 +0100 Subject: [PATCH 67/68] fix synchronous_commit link --- product_docs/docs/pgd/5/durability/legacy-sync.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/durability/legacy-sync.mdx b/product_docs/docs/pgd/5/durability/legacy-sync.mdx index 9deaf40b0f1..bc43cd89a1d 100644 --- a/product_docs/docs/pgd/5/durability/legacy-sync.mdx +++ b/product_docs/docs/pgd/5/durability/legacy-sync.mdx @@ -4,7 +4,7 @@ navTitle: Legacy synchronous replication --- !!! Warning Important -We highly recommend [PGD Synchronous Commit](synchronous-commit) instead of +We highly recommend [PGD Synchronous Commit](synchronous_commit) instead of legacy synchronous replication. !!! From 12377a1a70ab746aef8546ed8d51a26b99eef9c5 Mon Sep 17 00:00:00 2001 From: Dj Walker-Morgan <126472455+djw-m@users.noreply.github.com> Date: Tue, 12 Dec 2023 11:20:56 +0100 Subject: [PATCH 68/68] Fix reference link to como_partner_queue --- product_docs/docs/pgd/5/reference/functions.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/product_docs/docs/pgd/5/reference/functions.mdx b/product_docs/docs/pgd/5/reference/functions.mdx index 7f65bf88f76..7b7b99e85f8 100644 --- a/product_docs/docs/pgd/5/reference/functions.mdx +++ b/product_docs/docs/pgd/5/reference/functions.mdx @@ -294,7 +294,7 @@ function to prevent stale reads. For convenience, PGD provides a variant of this function for CAMO and the CAMO partner node. See -[bdr.wait_for_camo_partner_queue](../durability/camo#wait-for-consumption-of-the-apply-queue-from-the-camo-partner). +[bdr.wait_for_camo_partner_queue](#bdrwait_for_camo_partner_queue). In case a specific LSN is given, that's the point in the recovery stream from which the peer waits. You can use this