Merge pull request #327 from sloanebernstein/block-on-unacknowledged-…

…postgresql-dbs

Block when cP accts have DBs on distro PgSQL

docs-website-src/content/blockers.md

@@ -30,8 +30,7 @@ At any given time, the upgrade process may use at or more than 5 GB. If you have
 
 The following software is known to lead to a corrupt install if this script is used. We block elevation when it is detected:
 
-* **cPanel CCS Calendar Server** - Requires Postgresql < 10.0
-* **Postgresql** - ELevate upgrades you to Postgresql 10.x which makes it impossible to downgrade to a 9.x Postgresql.
+* **cPanel CCS Calendar Server** - Requires Postgresql older than 10.0
 
 ## Things you need to upgrade first.
 
@@ -64,6 +63,21 @@ You can discover many of these issues by downloading `elevate-cpanel` and runnin
 
 The following is a list of other known issues that could prevent your server's successful elevation.
 
+## PostgreSQL
+
+If you are using the PostgreSQL software provided by your distro (which includes PostgreSQL as installed by cPanel), ELevate will upgrade the software packages. However, your PostgreSQL service is unlikely to start properly. The reason for this is that ELevate will **not** attempt to update the data directory being used by your PostgreSQL instance to store settings and databases; and PostgreSQL will detect this condition and refuse to start, to protect your data from corruption, until you have performed this update.
+
+To ensure that you are aware of this requirement, if it detects that one or more cPanel accounts have associated PostgreSQL databases, ELevate will block you from beginning the upgrade process until you have created a file at `/var/cpanel/acknowledge_postgresql_for_elevate`.
+
+### Updating the PostgreSQL data directory
+
+Once ELevate has completed, you should then perform the update to the PostgreSQL data directory. Although we defer to the information [in the PostgreSQL documentation itself](https://www.postgresql.org/docs/10/pgupgrade.html), and although [Red Hat has provided steps in their documentation](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deploying_different_types_of_servers/using-databases#migrating-to-a-rhel-8-version-of-postgresql_using-postgresql) which should be mostly applicable to all distros derived from RHEL 8, we found that the following steps worked in our testing to update the PostgreSQL data directory. (Please note that these steps assume that your server's data directory is located at `/var/lib/pgsql/data`; your server may be different. You should also consider making a backup copy of your data directory before starting, because **cPanel cannot guarantee the correctness of these steps for any arbitrary PostgreSQL installation**.)
+
+1. Install the `postgresql-upgrade` package: `dnf install postgresql-upgrade`
+2. Within your PostgreSQL config file at `/var/lib/pgsql/data/postgresql.conf`, if there exists an active option `unix_socket_directories`, change that phrase to read `unix_socket_directory`. This is necessary to work around a difference between the CentOS 7 PostgreSQL 9.2 and the PostgreSQL 9.2 helpers packaged by your new operating system's `postgresql-upgrade` package.
+3. Invoke the `postgresql-setup` tool: `/usr/bin/postgresql-setup --upgrade`.
+4. In the root user's WHM, navigate to the "Configure PostgreSQL" area and click on "Install Config". This should restore the additions cPanel makes to the PostgreSQL access controls in order to allow phpPgAdmin to function.
+
 ## Using OVH proactive intervention monitoring
 
 If you are using a dedicated server hosted at OVH, you should **disable the `proactive monitoring` before starting** the elevation process.

elevate-cpanel

@@ -467,8 +467,10 @@ BEGIN {    # Suppress load of all of these at earliest point.
 
     use cPstrict;
 
-    use Cpanel::OS            ();
-    use Cpanel::Version::Tiny ();
+    use Cpanel::OS              ();
+    use Cpanel::Version::Tiny   ();
+    use Cpanel::JSON            ();
+    use Cpanel::SafeRun::Simple ();
 
     # use Elevate::Blockers::Base();
     our @ISA;
@@ -477,9 +479,13 @@ BEGIN {    # Suppress load of all of these at earliest point.
     # use Log::Log4perl qw(:easy);
     INIT { Log::Log4perl->import(qw{:easy}); }
 
+    use constant POSTGRESQL_ACK_TOUCH_FILE => q[/var/cpanel/acknowledge_postgresql_for_elevate];
+
     sub check ($self) {
+        my $ok = 1;
         $self->_warning_if_postgresql_installed;
-        my $ok = $self->_blocker_old_mysql;
+        $ok = 0 unless $self->_blocker_acknowledge_postgresql_datadir;
+        $ok = 0 unless $self->_blocker_old_mysql;
         $ok = 0 unless $self->_blocker_mysql_upgrade_in_progress;
         $self->_warning_mysql_not_enabled();
         return $ok;
@@ -498,6 +504,68 @@ BEGIN {    # Suppress load of all of these at earliest point.
         return 2;
     }
 
+    sub _blocker_acknowledge_postgresql_datadir ($self) {
+
+        return 0 unless Cpanel::Pkgr::is_installed('postgresql-server');
+
+        my $touch_file = POSTGRESQL_ACK_TOUCH_FILE;
+        return 0 if -e $touch_file;
+
+        my @users_with_dbs = $self->_has_mapped_postgresql_dbs();
+        return 0 unless scalar @users_with_dbs;
+
+        my $message = <<~"EOS";
+    One or more users on your system have associated PostgreSQL databases.
+    ELevate may upgrade the software packages associated with PostgreSQL
+    automatically, but if it does, it will *NOT* automatically update the
+    PostgreSQL data directory to work with the new version. Without an update
+    to the data directory, the upgraded PostgreSQL software will not start, in
+    order to ensure that your data does not become corrupted.
+
+    For more information about PostgreSQL upgrades, please consider the
+    following resources:
+
+    https://cpanel.github.io/elevate/blockers/#postgresql
+    https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deploying_different_types_of_servers/using-databases#migrating-to-a-rhel-8-version-of-postgresql_using-postgresql
+    https://www.postgresql.org/docs/10/pgupgrade.html
+
+    When you are ready to acknowledge that you have prepared to update the
+    PostgreSQL data directory, or that this warning does not apply to you,
+    please touch the following file to continue with the ELevate process:
+
+    > touch $touch_file
+
+    The following user(s) have PostgreSQL databases associated with their cPanel accounts:
+    EOS
+
+        $message .= join "\n", sort(@users_with_dbs);
+
+        return $self->has_blocker($message);
+    }
+
+    sub _has_mapped_postgresql_dbs ($self) {
+
+        local $ENV{HTTP_COOKIE} = 'session_locale=en';
+
+        my $users_json   = Cpanel::SafeRun::Simple::saferunnoerror(qw(/usr/local/cpanel/bin/whmapi1 --output=json list_users));
+        my $users_parsed = Cpanel::JSON::Load($users_json);
+        die "WHM API1 list_users call returned result 0: $users_parsed->{metadata}->{reason}" unless $users_parsed->{metadata}->{result};
+        my @users = grep { $_ ne "root" } $users_parsed->{data}->{users}->@*;
+
+        my @users_with_dbs;
+        foreach my $user (@users) {
+            my $dbs_json   = Cpanel::SafeRun::Simple::saferunnoerror( "/usr/local/cpanel/bin/uapi", "--user=$user", qw(--output=json Postgresql list_databases) );
+            my $dbs_parsed = Cpanel::JSON::Load($dbs_json);
+            if ( !$dbs_parsed->{result}->{status} ) {
+                return if ( $dbs_parsed->{result}->{errors}->[0] // '' ) eq 'This server does not support this functionality.';    # sanity check
+                die( "UAPI Postgresql::list_databases call returned status 0 for user $user:\n" . join( "\n", $dbs_parsed->{result}->{errors}->@* ) );
+            }
+            push @users_with_dbs, $user if scalar $dbs_parsed->{result}->{data}->@*;
+        }
+
+        return @users_with_dbs;
+    }
+
     sub _blocker_old_mysql ( $self, $mysql_version = undef ) {
 
         $mysql_version //= $self->cpconf->{'mysql-version'} // '';

lib/Elevate/Blockers/Databases.pm

@@ -12,16 +12,22 @@ Blockers for datbase: MySQL, PostgreSQL...
 
 use cPstrict;
 
-use Cpanel::OS            ();
-use Cpanel::Version::Tiny ();
+use Cpanel::OS              ();
+use Cpanel::Version::Tiny   ();
+use Cpanel::JSON            ();
+use Cpanel::SafeRun::Simple ();
 
 use parent qw{Elevate::Blockers::Base};
 
 use Log::Log4perl qw(:easy);
 
+use constant POSTGRESQL_ACK_TOUCH_FILE => q[/var/cpanel/acknowledge_postgresql_for_elevate];
+
 sub check ($self) {
+    my $ok = 1;
     $self->_warning_if_postgresql_installed;
-    my $ok = $self->_blocker_old_mysql;
+    $ok = 0 unless $self->_blocker_acknowledge_postgresql_datadir;
+    $ok = 0 unless $self->_blocker_old_mysql;
     $ok = 0 unless $self->_blocker_mysql_upgrade_in_progress;
     $self->_warning_mysql_not_enabled();
     return $ok;
@@ -40,6 +46,71 @@ sub _warning_if_postgresql_installed ($self) {
     return 2;
 }
 
+sub _blocker_acknowledge_postgresql_datadir ($self) {
+
+    return 0 unless Cpanel::Pkgr::is_installed('postgresql-server');
+
+    my $touch_file = POSTGRESQL_ACK_TOUCH_FILE;
+    return 0 if -e $touch_file;
+
+    my @users_with_dbs = $self->_has_mapped_postgresql_dbs();
+    return 0 unless scalar @users_with_dbs;
+
+    my $message = <<~"EOS";
+    One or more users on your system have associated PostgreSQL databases.
+    ELevate may upgrade the software packages associated with PostgreSQL
+    automatically, but if it does, it will *NOT* automatically update the
+    PostgreSQL data directory to work with the new version. Without an update
+    to the data directory, the upgraded PostgreSQL software will not start, in
+    order to ensure that your data does not become corrupted.
+
+    For more information about PostgreSQL upgrades, please consider the
+    following resources:
+
+    https://cpanel.github.io/elevate/blockers/#postgresql
+    https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deploying_different_types_of_servers/using-databases#migrating-to-a-rhel-8-version-of-postgresql_using-postgresql
+    https://www.postgresql.org/docs/10/pgupgrade.html
+
+    When you are ready to acknowledge that you have prepared to update the
+    PostgreSQL data directory, or that this warning does not apply to you,
+    please touch the following file to continue with the ELevate process:
+
+    > touch $touch_file
+
+    The following user(s) have PostgreSQL databases associated with their cPanel accounts:
+    EOS
+
+    $message .= join "\n", sort(@users_with_dbs);
+
+    return $self->has_blocker($message);
+}
+
+sub _has_mapped_postgresql_dbs ($self) {
+
+    # Normalize locale:
+    local $ENV{HTTP_COOKIE} = 'session_locale=en';
+
+    # Get a list of cPanel users:
+    my $users_json   = Cpanel::SafeRun::Simple::saferunnoerror(qw(/usr/local/cpanel/bin/whmapi1 --output=json list_users));
+    my $users_parsed = Cpanel::JSON::Load($users_json);
+    die "WHM API1 list_users call returned result 0: $users_parsed->{metadata}->{reason}" unless $users_parsed->{metadata}->{result};
+    my @users = grep { $_ ne "root" } $users_parsed->{data}->{users}->@*;
+
+    # Compile a list of users with PgSQL DBs:
+    my @users_with_dbs;
+    foreach my $user (@users) {
+        my $dbs_json   = Cpanel::SafeRun::Simple::saferunnoerror( "/usr/local/cpanel/bin/uapi", "--user=$user", qw(--output=json Postgresql list_databases) );
+        my $dbs_parsed = Cpanel::JSON::Load($dbs_json);
+        if ( !$dbs_parsed->{result}->{status} ) {
+            return if ( $dbs_parsed->{result}->{errors}->[0] // '' ) eq 'This server does not support this functionality.';    # sanity check
+            die( "UAPI Postgresql::list_databases call returned status 0 for user $user:\n" . join( "\n", $dbs_parsed->{result}->{errors}->@* ) );
+        }
+        push @users_with_dbs, $user if scalar $dbs_parsed->{result}->{data}->@*;
+    }
+
+    return @users_with_dbs;
+}
+
 sub _blocker_old_mysql ( $self, $mysql_version = undef ) {
 
     $mysql_version //= $self->cpconf->{'mysql-version'} // '';

t/blocker-Databases.t

@@ -157,4 +157,89 @@ my $mock_elevate = Test::MockFile->file('/var/cpanel/elevate');
 
 }
 
+{
+    note "PostgreSQL 9.2->10 acknowledgement";
+
+    my $pkgr_mock = Test::MockModule->new('Cpanel::Pkgr');
+    my %installed = ( 'postgresql-server' => 9.2 );
+    $pkgr_mock->redefine( 'is_installed'        => sub ($rpm) { return defined $installed{$rpm} ? 1 : 0 } );
+    $pkgr_mock->redefine( 'get_package_version' => sub ($rpm) { return $installed{$rpm} } );
+
+    my $mock_touchfile = Test::MockFile->file('/var/cpanel/acknowledge_postgresql_for_elevate');
+
+    my @mock_users = qw(cpuser1 cpuser2);
+    $db_mock->redefine( _has_mapped_postgresql_dbs => sub { return @mock_users } );
+
+    my $expected = {
+        id  => q[Elevate::Blockers::Databases::_blocker_acknowledge_postgresql_datadir],
+        msg => <<~'EOS'
+        One or more users on your system have associated PostgreSQL databases.
+        ELevate may upgrade the software packages associated with PostgreSQL
+        automatically, but if it does, it will *NOT* automatically update the
+        PostgreSQL data directory to work with the new version. Without an update
+        to the data directory, the upgraded PostgreSQL software will not start, in
+        order to ensure that your data does not become corrupted.
+
+        For more information about PostgreSQL upgrades, please consider the
+        following resources:
+
+        https://cpanel.github.io/elevate/blockers/#postgresql
+        https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/deploying_different_types_of_servers/using-databases#migrating-to-a-rhel-8-version-of-postgresql_using-postgresql
+        https://www.postgresql.org/docs/10/pgupgrade.html
+
+        When you are ready to acknowledge that you have prepared to update the
+        PostgreSQL data directory, or that this warning does not apply to you,
+        please touch the following file to continue with the ELevate process:
+
+        > touch /var/cpanel/acknowledge_postgresql_for_elevate
+
+        The following user(s) have PostgreSQL databases associated with their cPanel accounts:
+        cpuser1
+        cpuser2
+        EOS
+    };
+    chomp $expected->{msg};
+    is(
+        $db->_blocker_acknowledge_postgresql_datadir(),
+        $expected,
+        "PostgreSQL with cPanel users having databases and without ACK touch file is a blocker"
+    );
+
+    %installed = ();
+    is( $db->_blocker_acknowledge_postgresql_datadir(), 0, "No blocker if no postgresql-server package" );
+    %installed = ( 'postgresql-server' => 9.2 );
+
+    $mock_touchfile->touch();
+    is( $db->_blocker_acknowledge_postgresql_datadir(), 0, "No blocker if touch file present" );
+    $mock_touchfile->unlink();
+
+    @mock_users = ();
+    is( $db->_blocker_acknowledge_postgresql_datadir(), 0, "No blocker if no users have PgSQL DBs" );
+    @mock_users = qw(cpuser1 cpuser2);
+}
+
+{
+    note "check for PostgreSQL databases";
+
+    my $mock_saferun = Test::MockModule->new('Cpanel::SafeRun::Simple');
+
+    $mock_saferun->redefine(
+        saferunnoerror => sub {
+            return $_[0] eq '/usr/local/cpanel/bin/whmapi1'
+              ? '{"metadata":{"command":"list_users","version":1,"reason":"OK","result":1},"data":{"users":["root","cpuser2","cpuser1"]}}'
+              : '{"apiversion":3,"module":"Postgresql","func":"list_databases","result":{"warnings":null,"data":[{"disk_usage":9001,"database":"dontcare","users":["dontcare"]}],"errors":null,"metadata":{"transformed":1},"status":1,"messages":null}}';
+        }
+    );
+
+    is(
+        [ $db->_has_mapped_postgresql_dbs() ],
+        bag {
+            item 'cpuser1';
+            item 'cpuser2';
+            end();
+        },
+        "_has_mapped_postgresql_dbs returns expected list of users"
+    );
+}
+
 done_testing();