Backup and restore container gateway postgres DB #905

ianballou · 2024-07-23T20:27:08Z

Adds backup and restore support for the container gateway DB. Only adds support for the postgres version of the database for now.

One notable change was that the DB commands need to be run as foreman-proxy since no password is saved for the database. The container gateway uses unix socket auth for the connection, so only the owning user can access the DB (as far as I know).

As such, I've added support for running generic commands as a different user.

To test, try backup and restore on both a normal Foreman/Katello install and on a smart proxy with the container gateway installed.

ToDos:

Figure out why stderr is ending up in the container gateway database dump
Write tests

ianballou · 2024-07-24T14:11:18Z

I looked into the issue where stderr makes it into the dump file. It seems that this was always the case since the final command looks like sudo -u foreman-proxy -- pg_dump -Fc database > dump_file 2>&1. That would redirect stderr into stdout, which is being redirected into dump_file.

I'm guessing this wasn't an issue before because pg_dump was always getting run as root. In my case, foreman-proxy had no permissions to change directories into the place where I started the command, which caused an error which then made it literally into the dump file.

ianballou · 2024-07-24T21:19:12Z

I also ended up adding lines to change the backup ownership to foreman-proxy:foreman-proxy in order for the foreman-proxy user to have access. I noticed we deal with this issue slightly differently for restore -- there's a simple error telling the user to change the backup permissions. This is a bit different for backup though, because the user isn't the one creating the inner backup directory. I'm curious if anyone can see any issues coming out of this.

wbclark

Hey @ianballou !

I've shared some thoughts from my initial reading of this.

Regarding the proposal for some more abstraction here now that we've added a fourth(!) DB, it looks like it could also make sense for the definitions for online DB backup procedures and DB restore procedures?

I am curious to know what you think about it, and perhaps @evgeni would have some insight as well about whether this has been discussed before, and whether it's worth doing now

wbclark · 2024-08-06T15:52:55Z

definitions/procedures/restore/extract_files.rb

+      feature(:foreman_database) || feature(:candlepin_database) ||
+        feature(:pulpcore_database) || feature(:container_gateway_database)


Suggested change

feature(:foreman_database) || feature(:candlepin_database) ||

feature(:pulpcore_database) || feature(:container_gateway_database)

%i[

foreman_database

candlepin_database

pulpcore_database

container_gateway_database

].any? { |db| feature(db) }

[nit] this one sparks joy

wbclark · 2024-08-06T16:20:52Z

definitions/checks/container_gateway/db_up.rb

@@ -0,0 +1,29 @@
+module Checks
+  module ContainerGateway
+    class DBUp < ForemanMaintain::Check


[idea] It looks like we'll have nearly identical code at definitions/checks/{foreman,candlepin,pulpcore,container_gateway}/db_up.rb... what do you think about abstracting these into a common subclass of ForemanMaintain::Check?

It sounds like a good idea to me, the files appear to be very similar.

ianballou · 2024-08-08T16:39:22Z

I originally thought this would be required for Katello 4.14 since I assume the change to postgres would introduce a regression in backup & restore. It looks like it does not, in fact it appears that offline backup is backing up the container gateway DB for free somehow. Once the installer is run, the container_gateway DB is correctly created. So, users restoring at least can still just do a smart proxy sync to restore their content.

As such, I don't think we need to rush this for the upcoming Foreman release in case there are other concerns with the implementation that pop up. I'm personally a bit busy with getting other things in shape for the release, but please let me know if anyone thinks there's a new regression that I'm missing.

ianballou · 2024-08-09T12:43:47Z

@wbclark I've implemented your ideas

evgeni · 2024-08-13T18:56:39Z

It looks like it does not, in fact it appears that offline backup is backing up the container gateway DB for free somehow.

It will not anymore (since we merged #893), so caution! :)

definitions/features/container_gateway_database.rb

ianballou · 2024-10-23T13:50:37Z

Since this has slipped a little on our team priorities, I'm going to call this a PoC and get things updated once the strategy is determined to be sound.

evgeni · 2024-11-06T07:43:08Z

This needs a rebase, as we re-did quite some parts of base_database.rb, sorry.
Is my assumption correct, we do not support external DB for proxies at all right now?

ianballou · 2024-11-06T19:00:58Z

* Is my assumption correct, we do not support external DB for proxies at all right now?

I'll preface this by saying I'm not sure if we support external DBs for proxies, but, if we do ...

We did not test external container gateway external DBs. If I remember correctly, we have all of the connection information ready to be modified to an external host, so it should work to be remote.

If my PR here does not respect external databases but should, I'll fix it.

evgeni

I think this can be simplified, see inline comments :)

evgeni · 2024-11-07T12:57:00Z

lib/foreman_maintain/concerns/base_database.rb

+          base_cmd = if config['connection_string']
+                       'pg_restore'
+                     else
+                       base_command(config, 'pg_restore')
+                     end
+          dump_cmd = base_cmd +
                     ' --no-privileges --clean --disable-triggers -n public ' \
                     "-d #{config['database']} #{file}"
+          if config['user']
+            execute!("chown -R #{config['user']}:#{config['user']} #{File.dirname(file)}")
+          end
          execute!(dump_cmd, :hidden_patterns => [config['password']],
-            :valid_exit_statuses => [0, 1])
+            :valid_exit_statuses => [0, 1], :user => config['user'])


If the DB is remote, we don't need to change to config[user] (=foreman-proxy), as the DB communication happens over the network and doesn't require a specific user to initiate it.

evgeni · 2024-11-07T13:08:36Z

definitions/features/container_gateway_database.rb

+    connection_string = config[:db_connection_string]
+    if connection_string
+      uri = URI.parse(connection_string)
+      @configuration['connection_string'] = connection_string
+      @configuration['user'] = 'foreman-proxy'
+      @configuration['database'] = uri.path[1..]
+    end


So looking at theforeman/puppet-foreman_proxy#835, the connection_string is something like postgres:///container_gateway (for a local setup) or postgres://configurableuser:[email protected]:5432/container_gateway (for a remote DB, supported or not).

Now, the other code in this PR depends on the user definition whether it switches to that (shell) user or not.
That switching is unnecessary for the remote-DB case!

Suggested change

connection_string = config[:db_connection_string]

if connection_string

uri = URI.parse(connection_string)

@configuration['connection_string'] = connection_string

@configuration['user'] = 'foreman-proxy'

@configuration['database'] = uri.path[1..]

end

connection_string = config[:db_connection_string]

if connection_string

uri = URI.parse(connection_string)

@configuration['connection_string'] = connection_string

@configuration['user'] = 'foreman-proxy' unless uri.host

@configuration['database'] = uri.path[1..]

end

definitions/procedures/repositories/index_katello_repositories_container_metadata.rb

evgeni · 2024-11-07T13:12:40Z

lib/foreman_maintain/concerns/base_database.rb

      def dump_db(file, config = configuration)
-        execute!(dump_command(config) + " > #{file}", :hidden_patterns => [config['password']])
+        if config['user']
+          execute!("chown -R #{config['user']}:#{config['user']} #{File.dirname(file)}")
+        end
+        execute!(dump_command(config) + " -f #{file}",
+          :hidden_patterns => [config['password']], :user => config['user'])
      end


Right now, this relies on the DB credentials we parse from the config and in the super special case of gateway, the shell user of the proxy.

For restores, we have this "if local, use postgres user, else the parsed config as root", if we use the same logic for dumping, we reduce the special-casing for gateway (as we either use postgres, which has access to all databases anyway, or a remote connection that doesn't require foreman-proxy user).

When I wrote this I didn't think I could use the postgres user for some reason. Using postgres would make the code here a whole lot simpler 👍

ianballou force-pushed the backup-container-gateway branch from 384b86f to 5814c0e Compare July 23, 2024 21:30

ianballou force-pushed the backup-container-gateway branch 4 times, most recently from 214ff13 to 4818b98 Compare July 24, 2024 21:15

ianballou force-pushed the backup-container-gateway branch 2 times, most recently from 3ae9e8c to 9f3a763 Compare July 24, 2024 21:47

ianballou marked this pull request as ready for review July 24, 2024 21:48

ianballou force-pushed the backup-container-gateway branch 2 times, most recently from 0762bac to 1a0161a Compare July 25, 2024 13:53

wbclark reviewed Aug 6, 2024

View reviewed changes

ianballou added 4 commits August 9, 2024 12:13

Backup container gateway DB

c8a7ad2

Allow executing commands as a different user

068d4ac

Use --file and --format with pg_dump

c8fe3c2

Refactor DBUps into subclasses

fb56093

ianballou force-pushed the backup-container-gateway branch from 1a0161a to fb56093 Compare August 9, 2024 12:41

ianballou requested a review from wbclark August 9, 2024 12:43

evgeni reviewed Aug 13, 2024

View reviewed changes

definitions/features/container_gateway_database.rb Show resolved Hide resolved

evgeni requested changes Nov 7, 2024

View reviewed changes

pr-processor bot added the Waiting on contributor label Nov 7, 2024

Fix rubocop errors

cc3b128

ianballou force-pushed the backup-container-gateway branch from 0c1dfc0 to cc3b128 Compare November 7, 2024 18:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backup and restore container gateway postgres DB #905

Backup and restore container gateway postgres DB #905

ianballou commented Jul 23, 2024 •

edited

Loading

ianballou commented Jul 24, 2024

ianballou commented Jul 24, 2024

wbclark left a comment

wbclark Aug 6, 2024

wbclark Aug 6, 2024 •

edited

Loading

ianballou Aug 8, 2024

ianballou commented Aug 8, 2024

ianballou commented Aug 9, 2024

evgeni commented Aug 13, 2024

ianballou commented Oct 23, 2024

evgeni commented Nov 6, 2024

ianballou commented Nov 6, 2024

evgeni left a comment

evgeni Nov 7, 2024

evgeni Nov 7, 2024

evgeni Nov 7, 2024

ianballou Nov 7, 2024

		feature(:foreman_database) \|\| feature(:candlepin_database) \|\|
		feature(:pulpcore_database) \|\| feature(:container_gateway_database)

Backup and restore container gateway postgres DB #905

Are you sure you want to change the base?

Backup and restore container gateway postgres DB #905

Conversation

ianballou commented Jul 23, 2024 • edited Loading

ianballou commented Jul 24, 2024

ianballou commented Jul 24, 2024

wbclark left a comment

Choose a reason for hiding this comment

wbclark Aug 6, 2024

Choose a reason for hiding this comment

wbclark Aug 6, 2024 • edited Loading

Choose a reason for hiding this comment

ianballou Aug 8, 2024

Choose a reason for hiding this comment

ianballou commented Aug 8, 2024

ianballou commented Aug 9, 2024

evgeni commented Aug 13, 2024

ianballou commented Oct 23, 2024

evgeni commented Nov 6, 2024

ianballou commented Nov 6, 2024

evgeni left a comment

Choose a reason for hiding this comment

evgeni Nov 7, 2024

Choose a reason for hiding this comment

evgeni Nov 7, 2024

Choose a reason for hiding this comment

evgeni Nov 7, 2024

Choose a reason for hiding this comment

ianballou Nov 7, 2024

Choose a reason for hiding this comment

ianballou commented Jul 23, 2024 •

edited

Loading

wbclark Aug 6, 2024 •

edited

Loading