Refactor Barman scripts and add a sub-command to switch Barman config (…

…patroni#3016)

We currently have a script named `patroni_barman_recover` in Patroni, which is intended to be used as a custom bootstrap method, or as a custom replica creation method.

Now there is need of one more Barman related script in Patroni to handle switching of config models in Barman upon `on_role_change` events.

However, instead of creating another Patroni script, let's say `patroni_barman_config_switch`, and duplicating a lot of logic in the code, we decided to refactor the code so:

* Instead of two separate scripts (`patroni_barman_recover` and `patroni_barman_config_switch`), we have a single script (`patroni_barman`) with 2 sub-commands (`recover` and `config-switch`)

This is the overview of changes that have been performed:

* File `patroni.scripts.barman_recover` has been removed, and its logic has been split into a few files:
  * `patroni.scripts.barman.cli`: handles the entrypoint of the new `patroni_barman` command, exposing the argument parser and calling the appropriate functions depending on the sub-command
  * `patroni.scripts.barman.utils`: implements utilitary enums, functions and classes wich can be used by `cli` and by sub-commands implementation:
    * retry mechanism
    * logging set up
    * communication with pg-backup-api
  * `patroni.scripts.barman.recover`: implements the `recover` sub-command only
* File `patroni.tests.test_barman_recover` has been renamed as `patroni.tests.test_barman`
* File `patroni.scripts.barman.config_switch` was created to implement the `config-switch` sub-command only
* `setup.py` has been changed so it generates a `patroni_barman` application instead of `patroni_barman_recover`
* Docs and unit tests were updated accordingly

References: PAT-154.

docs/index.rst

@@ -34,6 +34,7 @@ Currently supported PostgreSQL versions: 9.3 to 16.
    kubernetes
    citus
    existing_data
+   tools_integration
    security
    ha_multi_dc
    faq

docs/replica_bootstrap.rst

@@ -1,3 +1,5 @@
+.. _replica_imaging_and_bootstrap:
+
 Replica imaging and bootstrap
 =============================
 
@@ -79,14 +81,13 @@ As an example, you are able to bootstrap a fresh Patroni cluster from a Barman b
         method: barman
         barman:
             keep_existing_recovery_conf: true
-            command: patroni_barman_recover
-            api-url: https://barman-host:7480
+            command: patroni_barman --api-url https://barman-host:7480 recover
             barman-server: my_server
             ssh-command: ssh postgres@patroni-host
 
 .. note::
-    ``patroni_barman_recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
-    The above example uses a subset of the available parameters. You can get more information running ``patroni_barman_recover --help``.
+    ``patroni_barman recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
+    The above example uses a subset of the available parameters. You can get more information running ``patroni_barman recover --help``.
 
 .. _custom_replica_creation:
 
@@ -150,16 +151,15 @@ example: Barman
             - barman
             - basebackup
         barman:
-            command: patroni_barman_recover
-            api-url: https://barman-host:7480
+            command: patroni_barman --api-url https://barman-host:7480 recover
             barman-server: my_server
             ssh-command: ssh postgres@patroni-host
         basebackup:
             max-rate: '100M'
 
 .. note::
-    ``patroni_barman_recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
-    The above example uses a subset of the available parameters. You can get more information running ``patroni_barman_recover --help``.
+    ``patroni_barman recover`` requires that you have both Barman and ``pg-backup-api`` configured in the Barman host, so it can execute a remote ``barman recover`` through the backup API.
+    The above example uses a subset of the available parameters. You can get more information running ``patroni_barman recover --help``.
 
 The ``create_replica_methods`` defines available replica creation methods and the order of executing them. Patroni will
 stop on the first one that returns 0. Each method should define a separate section in the configuration file, listing the command

docs/tools_integration.rst

@@ -0,0 +1,62 @@
+Integration with other tools
+============================
+
+Patroni is able to integrate with other tools in your stack. In this section you
+will find a list of examples, which although not an exhaustive list, might
+provide you with ideas on how Patroni can integrate with other tools.
+
+Barman
+------
+
+Patroni delivers an application named ``patroni_barman`` which has logic to
+communicate with ``pg-backup-api``, so you are able to perform Barman operations
+remotely.
+
+This application currently has a couple of sub-commands: ``recover`` and
+``config-switch``.
+
+patroni_barman recover
+^^^^^^^^^^^^^^^^^^^^^^
+
+The ``recover`` sub-command can be used as a custom bootstrap or custom replica
+creation method. You can find more information about that in
+:ref:`replica_imaging_and_bootstrap`.
+
+patroni_barman config-switch
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The ``config-switch`` sub-command is designed to be used as an ``on_role_change``
+callback in Patroni. As an example, assume you are streaming WALs from your
+current primary to your Barman host. In the event of a failover in the cluster
+you might want to start streaming WALs from the new primary. You can accomplish
+this by using ``patroni_barman config-switch`` as the ``on_role_change`` callback.
+
+.. note::
+    That sub-command relies on the ``barman config-switch`` command, which is in
+    charge of overriding the configuration of a Barman server by applying a
+    pre-defined model on top of it. This command is available since Barman 3.10.
+    Please consult the Barman documentation for more details.
+
+This is an example of how you can configure Patroni to apply a configuration
+model in case this Patroni node is promoted to primary:
+
+.. code:: YAML
+
+    postgresql:
+        callbacks:
+            on_role_change: >
+                patroni_barman
+                    --api-url YOUR_API_URL
+                    config-switch
+                    --barman-server YOUR_BARMAN_SERVER_NAME
+                    --barman-model YOUR_BARMAN_MODEL_NAME
+                    --switch-when promoted
+
+.. note::
+    ``patroni_barman config-switch`` requires that you have both Barman and
+    ``pg-backup-api`` configured in the Barman host, so it can execute a remote
+    ``barman config-switch`` through the backup API. Also, it requires that you
+    have pre-configured Barman models to be applied. The above example uses a
+    subset of the available parameters. You can get more information running
+    ``patroni_barman config-switch --help``, and by consulting the Barman
+    documentation.

patroni/scripts/barman/__init__.py

@@ -0,0 +1 @@
+"""Create :mod:`patroni.scripts.barman`."""

patroni/scripts/barman/cli.py

@@ -0,0 +1,240 @@
+#!/usr/bin/env python
+
+"""Perform operations on Barman through ``pg-backup-api``.
+
+The actual operations are implemented by separate modules. This module only
+builds the CLI that makes an interface with the actual commands.
+
+.. note::
+    See :class:ExitCode` for possible exit codes of this main script.
+"""
+
+from argparse import ArgumentParser
+from enum import IntEnum
+import logging
+import sys
+
+from .config_switch import run_barman_config_switch
+from .recover import run_barman_recover
+from .utils import ApiNotOk, PgBackupApi, set_up_logging
+
+
+class ExitCode(IntEnum):
+    """Possible exit codes of this script.
+
+    :cvar NO_COMMAND: if no sub-command of ``patroni_barman`` application has
+        been selected by the user.
+    :cvar API_NOT_OK: ``pg-backup-api`` status is not ``OK``.
+    """
+
+    NO_COMMAND = -1
+    API_NOT_OK = -2
+
+
+def main() -> None:
+    """Entry point of ``patroni_barman`` application.
+
+    Implements the parser for the application and for its sub-commands.
+
+    The script exit code may be one of:
+
+    * :attr:`ExitCode.NO_COMMAND`: if no sub-command was specified in the
+        ``patroni_barman`` call;
+    * :attr:`ExitCode.API_NOT_OK`: if ``pg-backup-api`` is not correctly up and
+        running;
+    * Value returned by :func:`~patroni.scripts.barman.config_switch.run_barman_config_switch`,
+        if running ``patroni_barman config-switch``;
+    * Value returned by :func:`~patroni.scripts.barman.recover.run_barman_recover`,
+        if running ``patroni_barman recover``.
+
+    The called sub-command is expected to exit execution once finished using
+    its own set of exit codes.
+    """
+    parser = ArgumentParser(
+        description=(
+            "Wrapper application for pg-backup-api. Communicate with the API "
+            "running at the given URL to perform remote Barman operations."
+        ),
+    )
+    parser.add_argument(
+        "--api-url",
+        type=str,
+        required=True,
+        help="URL to reach the pg-backup-api, e.g. 'http://localhost:7480'",
+        dest="api_url",
+    )
+    parser.add_argument(
+        "--cert-file",
+        type=str,
+        required=False,
+        help="Certificate to authenticate against the API, if required.",
+        dest="cert_file",
+    )
+    parser.add_argument(
+        "--key-file",
+        type=str,
+        required=False,
+        help="Certificate key to authenticate against the API, if required.",
+        dest="key_file",
+    )
+    parser.add_argument(
+        "--retry-wait",
+        type=int,
+        required=False,
+        default=2,
+        help="How long in seconds to wait before retrying a failed "
+             "pg-backup-api request (default: '%(default)s')",
+        dest="retry_wait",
+    )
+    parser.add_argument(
+        "--max-retries",
+        type=int,
+        required=False,
+        default=5,
+        help="Maximum number of retries when receiving malformed responses "
+             "from the pg-backup-api (default: '%(default)s')",
+        dest="max_retries",
+    )
+    parser.add_argument(
+        "--log-file",
+        type=str,
+        required=False,
+        help="File where to log messages produced by this application, if any.",
+        dest="log_file",
+    )
+
+    subparsers = parser.add_subparsers(title="Sub-commands")
+
+    recover_parser = subparsers.add_parser(
+        "recover",
+        help="Remote 'barman recover'",
+        description="Restore a Barman backup of a given Barman server"
+    )
+    recover_parser.add_argument(
+        "--barman-server",
+        type=str,
+        required=True,
+        help="Name of the Barman server from which to restore the backup.",
+        dest="barman_server",
+    )
+    recover_parser.add_argument(
+        "--backup-id",
+        type=str,
+        required=False,
+        default="latest",
+        help="ID of the Barman backup to be restored. You can use any value "
+             "supported by 'barman recover' command "
+             "(default: '%(default)s')",
+        dest="backup_id",
+    )
+    recover_parser.add_argument(
+        "--ssh-command",
+        type=str,
+        required=True,
+        help="Value to be passed as '--remote-ssh-command' to 'barman recover'.",
+        dest="ssh_command",
+    )
+    recover_parser.add_argument(
+        "--data-directory",
+        "--datadir",
+        type=str,
+        required=True,
+        help="Destination path where to restore the barman backup in the "
+             "local host.",
+        dest="data_directory",
+    )
+    recover_parser.add_argument(
+        "--loop-wait",
+        type=int,
+        required=False,
+        default=10,
+        help="How long to wait before checking again the status of the "
+             "recovery process, in seconds. Use higher values if your "
+             "recovery is expected to take long (default: '%(default)s')",
+        dest="loop_wait",
+    )
+    recover_parser.set_defaults(func=run_barman_recover)
+
+    config_switch_parser = subparsers.add_parser(
+        "config-switch",
+        help="Remote 'barman config-switch'",
+        description="Switch the configuration of a given Barman server. "
+                    "Intended to be used as a 'on_role_change' callback."
+    )
+    config_switch_parser.add_argument(
+        "action",
+        type=str,
+        choices=["on_role_change"],
+        help="Name of the callback (automatically filled by Patroni)",
+    )
+    config_switch_parser.add_argument(
+        "role",
+        type=str,
+        choices=["master", "primary", "promoted", "standby_leader", "replica",
+                 "demoted"],
+        help="Name of the new role of this node (automatically filled by "
+             "Patroni)",
+    )
+    config_switch_parser.add_argument(
+        "cluster",
+        type=str,
+        help="Name of the Patroni cluster involved in the callback "
+             "(automatically filled by Patroni)",
+    )
+    config_switch_parser.add_argument(
+        "--barman-server",
+        type=str,
+        required=True,
+        help="Name of the Barman server which config is to be switched.",
+        dest="barman_server",
+    )
+    group = config_switch_parser.add_mutually_exclusive_group(required=True)
+    group.add_argument(
+        "--barman-model",
+        type=str,
+        help="Name of the Barman config model to be applied to the server.",
+        dest="barman_model",
+    )
+    group.add_argument(
+        "--reset",
+        action="store_true",
+        help="Unapply the currently active model for the server, if any.",
+        dest="reset",
+    )
+    config_switch_parser.add_argument(
+        "--switch-when",
+        type=str,
+        required=True,
+        default="promoted",
+        choices=["promoted", "demoted", "always"],
+        help="Controls under which circumstances the 'on_role_change' callback "
+             "should actually switch config in Barman. 'promoted' means the "
+             "'role' is either 'master', 'primary' or 'promoted'. 'demoted' "
+             "means the 'role' is either 'replica' or 'demoted' "
+             "(default: '%(default)s')",
+        dest="switch_when",
+    )
+    config_switch_parser.set_defaults(func=run_barman_config_switch)
+
+    args, _ = parser.parse_known_args()
+
+    set_up_logging(args.log_file)
+
+    if not hasattr(args, "func"):
+        parser.print_help()
+        sys.exit(ExitCode.NO_COMMAND)
+
+    api = None
+
+    try:
+        api = PgBackupApi(args.api_url, args.cert_file, args.key_file,
+                          args.retry_wait, args.max_retries)
+    except ApiNotOk as exc:
+        logging.error("pg-backup-api is not working: %r", exc)
+        sys.exit(ExitCode.API_NOT_OK)
+
+    sys.exit(args.func(api, args))
+
+
+if __name__ == "__main__":
+    main()