From 3a90b6fa7ec7e28c2d71601f90dcf74874a79fda Mon Sep 17 00:00:00 2001 From: edb-slonik Date: Thu, 19 May 2022 17:24:25 +0000 Subject: [PATCH 1/3] [create-pull-request] automated change --- product_docs/docs/bdr/3.7/backup.mdx | 50 +++++++++++++++++- product_docs/docs/bdr/3.7/configuration.mdx | 38 ++++++++++++++ product_docs/docs/bdr/3.7/nodes.mdx | 4 ++ product_docs/docs/bdr/3.7/release-notes.mdx | 58 +++++++++++++++++++++ product_docs/docs/bdr/3.7/repsets.mdx | 34 ++++++------ 5 files changed, 165 insertions(+), 19 deletions(-) diff --git a/product_docs/docs/bdr/3.7/backup.mdx b/product_docs/docs/bdr/3.7/backup.mdx index ad5f2b01ae5..963d4b3b609 100644 --- a/product_docs/docs/bdr/3.7/backup.mdx +++ b/product_docs/docs/bdr/3.7/backup.mdx @@ -20,7 +20,7 @@ Recovery (DR), such as in the following situations: ## Backup -### `pg_dump` +### pg_dump `pg_dump`, sometimes referred to as "logical backup", can be used normally with BDR. @@ -232,6 +232,54 @@ of a single BDR node, optionally plus WAL archives: - Add further BDR nodes with the standard procedure based on the `bdr.join_node_group()` function call. +### pg_dump restore + +The restore procedure using pg_dump is somehow simpler (but not as consistent +as phisycal backups because it only makes a snapshot of a given moment in time). +The dump only takes personal data and not catalog data (BDR). + +Creating a new cluster from a dump needs only a new Postgresql instance where +to restore normally the dump. +The result is a plain postgres instance with all personal data. + +**No catalog changes are restored from the original BDR node.** +You need to manually create the BDR cluster from ths instance. +Create the BDR extension , create the node, create the bdr group, etc. and +eventually join other nodes to this instance. + +There is only one caveat to keep in mind when restoring a BDR dump. +BDR creates two policies on the `bdr.conflict_history table`. Postgresql normal +behavior does not allow to tie policies to an extension so these will be dumped +together with your data. These policies, unless removed from the dump, will drag +also the BDR extension and grant some permissions. + +You need to manually remove those statements from the dump prior restoring it. + +The statements you need to delete from the dump are the following: + +```sql +CREATE EXTENSION IF NOT EXISTS bdr WITH SCHEMA pg_catalog; +COMMENT ON EXTENSION bdr IS 'Bi-Directional Replication for PostgreSQL'; +CREATE POLICY conflict_filter ON bdr.conflict_history FOR SELECT USING (((( SELECT c.relowner + FROM pg_class c + WHERE (c.oid = conflict_history.reloid)) = ( SELECT pg_roles.oid + FROM pg_roles + WHERE (pg_roles.rolname = CURRENT_USER))) AND ( SELECT (NOT (c.relrowsecurity AND c.relforcerowsecurity)) + FROM pg_class c + WHERE (c.oid = conflict_history.reloid)))); +ALTER TABLE bdr.conflict_history ENABLE ROW LEVEL SECURITY; +CREATE POLICY conflict_read_all ON bdr.conflict_history FOR SELECT TO bdr_read_all_conflicts USING (true); +REVOKE ALL ON TABLE bdr.conflict_history FROM postgres; +REVOKE SELECT ON TABLE bdr.conflict_history FROM PUBLIC; +GRANT ALL ON TABLE bdr.conflict_history TO bdr_superuser; +GRANT SELECT ON TABLE bdr.conflict_history TO PUBLIC; +``` + +!!! Note + In the example, `postgres` is the initdb user, if you create the PostgreSQL + instance with a different user, then that user will be shown instead in the + dump, related to the `REVOKE` on `bdr.conflict_history` table. + #### Cleanup BDR Metadata The cleaning of leftover BDR metadata is achieved as follows: diff --git a/product_docs/docs/bdr/3.7/configuration.mdx b/product_docs/docs/bdr/3.7/configuration.mdx index 80fa0a3e973..2770d197839 100644 --- a/product_docs/docs/bdr/3.7/configuration.mdx +++ b/product_docs/docs/bdr/3.7/configuration.mdx @@ -104,6 +104,44 @@ pglogical internally to implement the basic replication. - `pglogical.max_writers_per_subscription` - Maximum number of writers per subscription (sets upper limit for the setting above). +### `pglogical.min_worker_backoff_delay` and `pglogical.max_worker_backoff_delay` + +Rate limit BDR background worker launches by preventing a given worker +from being relaunched more often than every +`pglogical.min_worker_backoff_delay` milliseconds. On repeated errors, the back-off +increases exponentially with added jitter up to maximum of +`pglogical.max_worker_backoff_delay`. + +Time-unit suffixes are supported. + +!!! Note + This setting currently only affects receiver worker, which means it + primarily affects how fast a subscription will try to reconnect on error + or connection failure. + +The default for `pglogical.min_worker_backoff_delay` 1 second, for +`pglogical.max_worker_backoff_delay` it is 1 minute. + +If the backoff delay setting is changed and the PostgreSQL configuration is +reloaded then all current backoff waits will be reset. Additionally, the +`pglogical.worker_task_reset_backoff_all()` function is provided to allow the +administrator to force all backoff intervals to immediately expire. + +A tracking table in shared memory is maintained to remember the last launch +time of each type of worker. This tracking table is not persistent; it is +cleared by PostgreSQL restarts, including soft-restarts during crash recovery +after an unclean backend exit. + +The view `pglogical.worker_tasks` may be used to inspect this state so the +administrator can see any backoff rate-limiting currently in effect. + +For rate limiting purposes, workers are classified by "task". This key consists +of the worker role, database oid, subscription id, subscription writer id, +extension library name and function name, extension-supplied worker name, and +the remote relation id for sync writers. `NULL` is used where a given +classifier does not apply, e.g. manager workers don't have a subscription ID +and receivers don't have a writer id. + ## BDR Specific Settings There are also BDR specific configuration settings that can be set. diff --git a/product_docs/docs/bdr/3.7/nodes.mdx b/product_docs/docs/bdr/3.7/nodes.mdx index e03768f10cd..3bfd34739e7 100644 --- a/product_docs/docs/bdr/3.7/nodes.mdx +++ b/product_docs/docs/bdr/3.7/nodes.mdx @@ -1311,6 +1311,10 @@ worker stopping depends on the value of `immediate`; if set to `true`, the workers will be stopped immediately; if set to `false`, they will be stopped at the `COMMIT` time. +!!! Note + With the parameter `immediate` set to `true`, the stop will however wait + for the workers to finish current work. + ## Node Management Commands BDR also provides a command line utility for adding nodes to the BDR group via diff --git a/product_docs/docs/bdr/3.7/release-notes.mdx b/product_docs/docs/bdr/3.7/release-notes.mdx index c0d320f7584..ff77655a4fd 100644 --- a/product_docs/docs/bdr/3.7/release-notes.mdx +++ b/product_docs/docs/bdr/3.7/release-notes.mdx @@ -5,6 +5,57 @@ originalFilePath: release-notes.md --- +## BDR 3.7.16 + +This is a maintenance release for BDR 3.7 which includes minor +improvements as well as fixes for issues identified in previous +versions. + +Check also release notes for pglogical 3.7.16 for resolved issues +which affect BDR as well. + +### Resolved Issues + +- Make ALTER TABLE lock the underlying relation only once (RT80204) + This avoids the ALTER TABLE operation falling behind in the queue + when it released the lock in between internal operations. With this + fix, concurrent transactions trying to acquire the same lock after + the ALTER TABLE command will properly wait for the ALTER TABLE to + finish. + +- Show a proper wait event for CAMO / Eager confirmation waits (BDR-1899, RT75900) + Show correct "BDR Prepare Phase"/"BDR Commit Phase" in `bdr.stat_activity` + instead of the default “unknown wait event”. + +- Correct `bdr.monitor_local_replslots` for down nodes (BDR-2080) + This function mistakenly returned an okay result for down nodes + before. + +- Reduce log for bdr.run_on_nodes (BDR-2153, RT80973) + Don't log when setting `bdr.ddl_replication` to off if it's + done with the "run_on_nodes" variants of function. This eliminates + the flood of logs for monitoring functions. + +- Correct an SDW decoder restart edge case (BDR-2109) + Internal testing revealed a possible error during WAL decoder + recovery about mismatch between confirmed_flush LSN of WAL decoder + slot also stating: "some LCR segments might be missing". This could + happen before in case the WAL decoder exited immediately after + processing a "Standby" WAL record other than "RUNNING_XACTS" and + would lead to a halt of replication with the decoder processes + continuing to restart. + +### Improvements + +- Use 64 bits for calculating lag size in bytes (BDR-2215) + +### Upgrades + +This release supports upgrading from the following versions of BDR: + +- 3.7.9 and higher +- 3.6.29 and higher + ## BDR 3.7.15 This is a maintenance release for BDR 3.7 which includes minor @@ -78,6 +129,13 @@ which affect BDR as well. requests or time consuming consensus requests, for example during join of a new node. +### Upgrades + +This release supports upgrading from the following versions of BDR: + +- 3.7.9 and higher +- 3.6.29 and higher + ## BDR 3.7.14 This is a maintenance release for BDR 3.7 which includes minor diff --git a/product_docs/docs/bdr/3.7/repsets.mdx b/product_docs/docs/bdr/3.7/repsets.mdx index 44dc54d5d13..2c70f019088 100644 --- a/product_docs/docs/bdr/3.7/repsets.mdx +++ b/product_docs/docs/bdr/3.7/repsets.mdx @@ -461,16 +461,15 @@ Use the following SQL to show those replication sets that the current node publishes and subscribes from: ```sql -SELECT s.node_id, - s.node_name, + SELECT node_id, + node_name, COALESCE( - i.pub_repsets, s.pub_repsets + pub_repsets, pub_repsets ) AS pub_repsets, COALESCE( - i.sub_repsets, s.sub_repsets + sub_repsets, sub_repsets ) AS sub_repsets -FROM bdr.local_node_summary s -INNER JOIN bdr.node_local_info i ON i.node_id = s.node_id; + FROM bdr.local_node_summary; ``` This produces output like this: @@ -490,17 +489,16 @@ the following query: WITH node_repsets AS ( SELECT jsonb_array_elements( bdr.run_on_all_nodes($$ - SELECT s.node_id, - s.node_name, - COALESCE( - i.pub_repsets, s.pub_repsets - ) AS pub_repsets, - COALESCE( - i.sub_repsets, s.sub_repsets - ) AS sub_repsets - FROM bdr.local_node_summary s - INNER JOIN bdr.node_local_info i - ON i.node_id = s.node_id; + SELECT + node_id, + node_name, + COALESCE( + pub_repsets, pub_repsets + ) AS pub_repsets, + COALESCE( + sub_repsets, sub_repsets + ) AS sub_repsets + FROM bdr.local_node_summary; $$)::jsonb ) AS j ) @@ -508,7 +506,7 @@ SELECT j->'response'->'command_tuples'->0->>'node_id' AS node_id, j->'response'->'command_tuples'->0->>'node_name' AS node_name, j->'response'->'command_tuples'->0->>'pub_repsets' AS pub_repsets, j->'response'->'command_tuples'->0->>'sub_repsets' AS sub_repsets -FROM node_repsets;; +FROM node_repsets; ``` This will show, for example: From d66c6fe9310458700327c91f2a36fd3ccd62dbc2 Mon Sep 17 00:00:00 2001 From: edb-slonik Date: Thu, 19 May 2022 17:49:28 +0000 Subject: [PATCH 2/3] [create-pull-request] automated change --- .../docs/pglogical/3.7/configuration.mdx | 25 ++++++--- .../docs/pglogical/3.7/release-notes.mdx | 51 +++++++++++++++++-- .../pglogical/3.7/subscriptions/index.mdx | 5 +- 3 files changed, 67 insertions(+), 14 deletions(-) diff --git a/product_docs/docs/pglogical/3.7/configuration.mdx b/product_docs/docs/pglogical/3.7/configuration.mdx index 3e84b5bfcef..5b3f2027dc2 100644 --- a/product_docs/docs/pglogical/3.7/configuration.mdx +++ b/product_docs/docs/pglogical/3.7/configuration.mdx @@ -127,6 +127,12 @@ Collecting statistics requires additional CPU resources on the subscriber. The default is off. +### `pglogical.track_apply_lock_timing` + +This tracks lock timing when tracking statistics for relations. + +The default is off. + ### `pglogical.temp_directory` This defines system path for where to put temporary files needed for schema @@ -256,16 +262,23 @@ should be enough. The default is 32kB and the maximum allowed size is 1MB. While any storage size specifier can be used to set the GUC, the default is kB. -### `pglogical.min_worker_backoff_delay` +### `pglogical.min_worker_backoff_delay` and `pglogical.max_worker_backoff_delay` Rate limit pglogical background worker launches by preventing a given worker from being relaunched more often than every -`pglogical.min_worker_backoff_delay` milliseconds. Time-unit suffixes are -supported. +`pglogical.min_worker_backoff_delay` milliseconds. On repeated errors, the back-off +increases exponentially with added jitter up to maximum of +`pglogical.max_worker_backoff_delay`. + +Time-unit suffixes are supported. + +!!! Note + This setting currently only affects receiver worker, which means it + primarily affects how fast a subscription will try to reconnect on error + or connection failure. -The default is 0, meaning no rate limit. The delay is a time limit applied from -launch-to-launch, so a value of `'500ms'` would limit all types of workers to -at most 2 (re)launches per second. +The default for `pglogical.min_worker_backoff_delay` is 1 second, for +`pglogical.max_worker_backoff_delay` it is 1 minute. If the backoff delay setting is changed and the PostgreSQL configuration is reloaded then all current backoff waits will be reset. Additionally, the diff --git a/product_docs/docs/pglogical/3.7/release-notes.mdx b/product_docs/docs/pglogical/3.7/release-notes.mdx index 83b8cf0e276..05cce8d7044 100644 --- a/product_docs/docs/pglogical/3.7/release-notes.mdx +++ b/product_docs/docs/pglogical/3.7/release-notes.mdx @@ -5,6 +5,39 @@ originalFilePath: release-notes.md --- +## pglogical 3.7.16 + +This is a maintenance release for pglogical 3.7 which includes minor +improvements as well as fixes for issues identified previously. + +### Resolved Issues + +- Keep the `lock_timeout` as configured on non-CAMO-partner BDR nodes (BDR-1916) + A CAMO partner uses a low `lock_timeout` when applying transactions + from its origin node. This was inadvertently done for all BDR nodes + rather than just the CAMO partner, which may have led to spurious + `lock_timeout` errors on pglogical writer processes on normal BDR + nodes. + +- Prevent walsender processes spinning when facing lagging standby slots (RT80295, RT78290) + Correct signaling to reset a latch so that a walsender process does + consume 100% of a CPU in case one of the standby slots is lagging + behind. + +- Provide a proper error when attempting to use `pglogical.use_spi` (RT76368) + +- Reduce log information when switching between writer processes (BDR-2196) + +- Eliminate a memory leak when replicating partitioned tables (RT80981, BDR-2194) + +### Upgrades + +This release supports upgrading from following versions of pglogical: + +- 3.7.9 and higher +- 3.6.29 and higher +- 2.4.0 and 2.4.1 + ## pglogical 3.7.15 This is a maintenance release for pglogical 3.7 which includes minor @@ -12,10 +45,10 @@ improvements as well as fixes for issues identified previously. ### Improvements -- Add `bdr.max_worker_backoff_delay` (BDR-1767) +- Add `pglogical.max_worker_backoff_delay` (BDR-1767) This changes the handling of the backoff delay to exponentially - increase from `bdr.min_worker_backoff_delay` to - `bdr.max_worker_backoff_delay` in presence of repeated errors. This + increase from `pglogical.min_worker_backoff_delay` to + `pglogical.max_worker_backoff_delay` in presence of repeated errors. This reduces log spam and in some cases also prevents unnecessary connection attempts. @@ -29,15 +62,23 @@ improvements as well as fixes for issues identified previously. The `wal_receiver_timeout` has not been triggered correctly due to a regression in BDR 3.7 and 4.0. -- Limit the `bdr.standby_slot_names` check when reporting flush position only to +- Limit the `pglogical.standby_slot_names` check when reporting flush position only to physical slots (RT77985, RT78290) Otherwise flush progress is not reported in presence of disconnected nodes when - using `bdr.standby_slot_names`. + using `pglogical.standby_slot_names`. - Confirm LSN of LCR slot progress records when recovering LCR segments (BDR-1264) - Fix replication of data types created during bootstrap (BDR-1784) +### Upgrades + +This release supports upgrading from following versions of pglogical: + +- 3.7.9 and higher +- 3.6.29 and higher +- 2.4.0 and 2.4.1 + ## pglogical 3.7.14 This is a maintenance release for pglogical 3.7 which includes minor diff --git a/product_docs/docs/pglogical/3.7/subscriptions/index.mdx b/product_docs/docs/pglogical/3.7/subscriptions/index.mdx index fe4789b3075..3d929cbcf73 100644 --- a/product_docs/docs/pglogical/3.7/subscriptions/index.mdx +++ b/product_docs/docs/pglogical/3.7/subscriptions/index.mdx @@ -223,9 +223,8 @@ alternately choose your own replication slot name instead of using pglogical.writers_per_subscription. Valid values are either -1 or a positive integer. - `writer` - which writer to use for writing the data from the replication - stream. Available writers currently are `local`, `HeapWriter` and - `SPIWriter`; the local is an alias that automatically selects either - `HeapWriter` or `SPIWriter` based on the version of PostgreSQL being used. + stream. Only `HeapWriter` is supported in this version of PGLogical. + `local` is an alias for backwards compatibility. - `writer_options` - writer-specific options as an array of keys and values ### `pglogical_create_subscriber` From 24ae207aacfd2c809f55f40f50e1c07d77b72eea Mon Sep 17 00:00:00 2001 From: drothery-edb Date: Thu, 19 May 2022 13:57:51 -0400 Subject: [PATCH 3/3] Update index.mdx --- product_docs/docs/pgd/4/deployments/tpaexec/index.mdx | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/product_docs/docs/pgd/4/deployments/tpaexec/index.mdx b/product_docs/docs/pgd/4/deployments/tpaexec/index.mdx index 8218a2d3dc6..f0c60a36ca4 100644 --- a/product_docs/docs/pgd/4/deployments/tpaexec/index.mdx +++ b/product_docs/docs/pgd/4/deployments/tpaexec/index.mdx @@ -20,13 +20,18 @@ TPAexec packages are only available to EDB customers with an active BDR subscrip ### Configuration The `tpaexec configure` command generates a simple YAML configuration file to describe a cluster, based on the options you select. The configuration is ready for immediate use, and you can modify it to better suit your needs. Editing the configuration file is the usual way to make any configuration changes to your cluster, both before and after it's created. -The configuration options include: +The syntax is: + +``` +tpaexec configure --architecture --layout --bdr-version -- --2Q-repositories --extra-packages edb-livecompare edb-lasso --platform +``` + +The primary configuration options include: | Flags | Description | | ---------------------------- | ----------- | | --architecture BDR-Always-ON | Required. Set to 'BDR-Always-ON' for EDB Postgres Distributed deployments | | --layout layoutname | Required. Specify one of the four supported architectures: bronze, silver, gold, and platinum. See [Choosing your architecture][/pgd/4/architectures] for more information. | - | --harp-consensus-protocol | Required. `bdr` is recommended if your layout is bronze or silver. `etcd` is recommended if your layout is gold or platinum. See [Choosing your DCS]/pgd/4/dcs] for more information. | | --bdr-node-group groupname | Optional. Set the name of the BDR node group. The default is `bdrgroup`. | | --bdr-database dbname | Optional. Set the name of the database with BDR enabled. The default is `bdrdb`. |