Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bareos backup server #94 #395

Draft
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

phillxnet
Copy link
Member

@phillxnet phillxnet commented Dec 5, 2024

Multi-container Bareos server set Rock-on containing:

  • Director Service (overall controller).
  • Catalog Service (Postgres DB for Director).
  • Storage Service (Back-end Storage for Director and File (client) services).
  • Director-local File (client) Service (for Catalog backup capability).
  • Web-UI Service (Director Web-UI interface).

Each of the above 5 services are hosted in their own dedicated containers.

Fixes #94

Information on docker image

  • Docker image: custom-made docker images from Rockstor developers. As yet in-flux and may be differently published. Note Draft PR status.
  • Is an official docker image available for this project?: Not yet.

Checklist

  • Passes JSONlint validation
  • Entry added to root.json in alphabetical order (for new rock-on only)
  • "description" object lists and links to the docker image used
  • "description" object provides information on the image's particularities (advantage over another existing rock-on for the same project, for instance)
  • "website" object links to project's main website

Multi-container Bareos server set Rock-on containing:
- Director Service (overall controller).
- Catalog Service (Postgres DB for Director).
- Storage Service (Back-end Storage for Director and File (client) services).
- Director-local File (client) Service (for Catalog backup capability).
- Web-UI Service (Director Web-UI interface).
Each of the above 5 services are hosted in their own dedicated containers.
Uses custom-made docker images from Rockstor developers as no official
docker images where available.
@phillxnet
Copy link
Member Author

@Hooverdan96 & @FroggyFlox Draft and untested. Pushing to draft PR to aid collaboration in pending development. Plus I can more easily test end-to-end, using back-end mechanisms, when it's a PR, draft or otherwise.

@phillxnet
Copy link
Member Author

phillxnet commented Dec 6, 2024

Initial install test only

[06/Dec/2024 17:18:47] INFO [storageadmin.views.rockon:504] Rock-on definitions retrieved in: 0.16 seconds.
[06/Dec/2024 17:39:48] INFO [storageadmin.tasks:55] Now executing Huey task [install], id: be7029c6-0726-4f55-8cc3-db9298c43ccd.
[06/Dec/2024 17:39:48] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'bareos-db']. output: [''] error: ['Error response from daemon: No such container: bareos-db', '']
[06/Dec/2024 17:39:48] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'bareos-db']. output: [''] error: ['Error response from daemon: No such container: bareos-db', '']
[06/Dec/2024 17:39:59] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:40:15] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:40:20] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'bareos-dir']. output: [''] error: ['Error response from daemon: No such container: bareos-dir', '']
[06/Dec/2024 17:40:20] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'bareos-dir']. output: [''] error: ['Error response from daemon: No such container: bareos-dir', '']
[06/Dec/2024 17:40:30] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:40:46] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:40:55] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'bareos-storage']. output: [''] error: ['Error response from daemon: No such container: bareos-storage', '']
[06/Dec/2024 17:40:55] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'bareos-storage']. output: [''] error: ['Error response from daemon: No such container: bareos-storage', '']
[06/Dec/2024 17:41:01] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:41:16] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:41:24] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'bareos-fd']. output: [''] error: ['Error response from daemon: No such container: bareos-fd', '']
[06/Dec/2024 17:41:24] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'bareos-fd']. output: [''] error: ['Error response from daemon: No such container: bareos-fd', '']
[06/Dec/2024 17:41:31] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:41:46] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:41:56] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'stop', 'bareos-webui']. output: [''] error: ['Error response from daemon: No such container: bareos-webui', '']
[06/Dec/2024 17:41:56] ERROR [system.osi:287] non-zero code(1) returned by command: ['/usr/bin/docker', 'rm', 'bareos-webui']. output: [''] error: ['Error response from daemon: No such container: bareos-webui', '']
[06/Dec/2024 17:42:01] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:42:17] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:42:32] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:42:47] INFO [storageadmin.views.rockon:106] Rockon (Bareos-backup-server) state pending and no pending task: assuming task is mid execution.
[06/Dec/2024 17:43:00] INFO [storageadmin.tasks:63] Task [install], id: be7029c6-0726-4f55-8cc3-db9298c43ccd completed OK
rleap15-6:~ # docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED          STATUS          PORTS                                       NAMES
faf890ab80a7   ghcr.io/phillxnet/bareos-webui:main      "docker-entrypoint.s…"   30 minutes ago   Up 30 minutes   0.0.0.0:9100->80/tcp, :::9100->80/tcp       bareos-webui
659a98937fb2   ghcr.io/phillxnet/bareos-file:main       "docker-entrypoint.s…"   31 minutes ago   Up 31 minutes   9102/tcp                                    bareos-fd
cf268a47c45c   ghcr.io/phillxnet/bareos-storage:main    "docker-entrypoint.s…"   31 minutes ago   Up 31 minutes   0.0.0.0:9103->9103/tcp, :::9103->9103/tcp   bareos-storage
56f9c5881e63   ghcr.io/phillxnet/bareos-director:main   "docker-entrypoint.s…"   32 minutes ago   Up 32 minutes   0.0.0.0:9101->9101/tcp, :::9101->9101/tcp   bareos-dir
a1beec1e528e   postgres:14                              "docker-entrypoint.s…"   32 minutes ago   Up 32 minutes   5432/tcp                                    bareos-db

We have remaining way too many 'fake' double entry passwords here as a hack to supply each container with its version of the passwords subsets require to communicate. Hoping to remove most of these via their mutual shares and each containers self-config via their docker-entrypoint.sh scripts.

@Hooverdan96
Copy link
Member

like on the paperless Rockon, how important/secure is it to have the user set passwords across all containers? Would it make sense for example to also hard-code a Postgres password and hide it in the options section of the Rockon? What about the others?

@phillxnet
Copy link
Member Author

@Hooverdan96 Re passwords. I'm still thinking on this one. We may, as you say, have some scope to set default passwords: and the rpm install do just this, but in this case the user needs to know the password as bareos client installs on other machines to be backed-up will have to enter said password. Initially I'm working towards having the minimum external/user-set passwords to actually enter. Folks always get exited about password stuff so we have to tread carefully here. But the initial plan is to try and stick to upstream as closely as possibly. But the non-shared OS between these abstracted containers is working against us. As per many technologies it kicks problems into another area. But there is hope I think re the share resource via my approach here of a shared volume. Experimentation ongoing and some work as yet as a proof of concept.

Essential we need a secrets manager!! But again that is for another.

My overall aim here is to get something that works - and is non-complex with same defaults and close to upstream. We can then all chip-in with refinements: such as supporting Bareos's supported binaries repo via credentials. But I'm still fathoming getting minimum rights right first. I.e. running everything with only as much access as is required. I'll welcome more input on that font once we have an actual working server set here. Note that this is only the server side. Folks will then install on their laptops directly from upstream the Client/File service, and connect to the server set to host the actual backup and Web-UI.

@phillxnet
Copy link
Member Author

A major concern with passwords here is that as this is a network back-up: it will host potentially all machines data. Ergo we have zero pre-known passwords. Even if the particular services are inaccessible. Tricky thought. As last resort I may use our custom config and set a random password in some places (lots of services here): but we then have to have folks do yet more cli to find that password!! Bit by bit.

The File (client) Service, bareos-fd, must be able to name
resolve the bareos-storage container, as the Director
instructs the File service to seek out a specific Storage
service. File and Storage then need to communicate directly
on a name basis.
@phillxnet
Copy link
Member Author

The same limitation encountered in a prior draft re #383 (comment) has come-up again in this PR with the commit proceeding this comment. The Bareos Web-UI reports the following when executing a build-in/pre-configured job:

  2024-12-07 16:53:45 bareos-fd JobId 2: Error: lib/bsock_tcp.cc:183 BnetHost2IpAddrs() for host "bareos-storage" failed: ERR=No address associated with hostname

Looking to @Hooverdan96 exposition in the comments following the above referenced comment, and their reproducer detailed in the following issue:

As this issue now represents a blocker bug re further development on this Rock-on.

See also @FroggyFlox, author of our docker networks wrapper, #383 (comment) indicating that this limitation is not expected.

@Hooverdan96
Copy link
Member

Hooverdan96 commented Dec 7, 2024

So, a workaround (in absence of a quick solution) could be the manual creation of networks and attaching of all relevant containers with corresponding notes in the instructions of the Rockon? Or, do you "feel" that the root cause can and should be addressed before the next stable?

@phillxnet
Copy link
Member Author

@Hooverdan96 Re:

So, a workaround (in absence of a quick solution) could be the manual creation of networks and attaching of all relevant containers with corresponding notes in the instructions of the Rockon?

Yes that is a good point. It did occur to me: but only belatedly as I super keen on keeping the pre and post user intervention to a minimum on this one. But all-in, with the more recent notes on your linked issue re our current limitations. What you suggest may well be the only way though here. Another way could be to hard-wire a --network element in here. But I'm concerned about race conditions. I'll have an experimental session to what the options hare here:

Re:

Or, do you "feel" that the root cause can and should be addressed before the next stable?

I've just brought up a discussion regarding this on the linked issue. With a call-out to you on your opinion. We should continue that discussion there I think. In short this is a way too deep change this late in testing. So I'll likely follow your work-around here for the Bareos back server approach. At least until we have our next generation of docker network support :).

@FroggyFlox
Copy link
Member

But I'm concerned about race conditions.

I'm afraid it could be a problem, yes. We indeed create networks and connect containers to it as the last step of the rockon install process:
https://github.com/rockstor/rockstor-core/blob/4d60a2c683140aaa10a8e5b201157163cc9a898b/src/rockstor/storageadmin/views/rockon_helpers.py#L367
Hard to predict when the docker run command would fail if it does, in this case, it seems.

@phillxnet
Copy link
Member Author

Work-around re container_links limitations

We need a newtork between Director and all 'other' services (Catalog (DB), Storage, File, Web-UI). As such when we try to define another required network between two of the 'other' services; File, & Storage; we fail via:

  • Rockon Container Links cannot handle duplicate sources #2900

Based on @FroggyFlox's #395 (comment) earlier, re Rock-on/docker-network startup order, we are left with additional/by-hand rock-net config post install.

Rock-net for File/Storage communication

  1. Create a DHCP docker network named BareosFileToStorage in Web-UI networking
  2. Add bareos-fd & bareos-storage containers too this network via Rock-on spanner -Networking.

Pre & post applied summary:

Resource type Internal Reference External Reference / Configured value
Network bareos-storage BareosFileToStorage
Network bareos-fd BareosFileToStorage

Confirm inter container name resolution

Do we fix Files container's ability to resolve the Storage container by name, and visa-versa:
(Required as File service is instructed to use Storage service directly under instruction from overall controller service of Director.
See prior failure detailed in: #395 (comment)

Storage from File

docker exec -it bareos-fd sh

sh-4.4# ping bareos-storage
PING bareos-storage (172.26.0.2) 56(84) bytes of data.
64 bytes from bareos-storage.BareosFileToStorage (172.26.0.2): icmp_seq=1 ttl=64 time=0.087 ms
64 bytes from bareos-storage.BareosFileToStorage (172.26.0.2): icmp_seq=2 ttl=64 time=0.070 ms
^C

File from Storage

docker exec -it bareos-storage sh
sh-4.4$ ping bareos-fd
PING bareos-fd (172.26.0.3) 56(84) bytes of data.
64 bytes from bareos-fd.BareosFileToStorage (172.26.0.3): icmp_seq=1 ttl=64 time=0.059 ms
64 bytes from bareos-fd.BareosFileToStorage (172.26.0.3): icmp_seq=2 ttl=64 time=0.080 ms
^C

@phillxnet
Copy link
Member Author

Using the last comments work-around we have our first successful Job run, initiated via Bareos's Web-UI , of the build-in Job:

  • backup-bareos-fd

Which defaults to using: FileSet: "SelfTest" retrieved by our Director-local File (Client) service - which then tasks the Storage service to backup. Which in-turn uses, by default, a File back-end (File Storage device), using the Full Pool (confusing for us but Bareos also uses the term Pool!!. And we have a new *Full-0001 volume, which for a File Storage device equates to:

rleap15-6:~ # ls -lah /mnt2/bareos-backups/
total 4.8M
drwxr-xr-x 1 bareos bareos   18 Dec  9 19:41 .
drwxr-xr-x 1 root   root    222 Dec  6 17:16 ..
-rw-r----- 1 bareos bareos 4.8M Dec  9 19:42 Full-0001

From small beginnings :).

We intend to run as uid 105, make this explicity via the
uid Rock-on directive.
@phillxnet
Copy link
Member Author

Status update

After a tweak to the bareos-director docker image re adapting an upstream pg_dump command in 'RunBeforeJob' script (make_catalog_backup) as part of the BackupCatalog job, which has the following FileSet:

rleap15-6:~ # docker exec -it bareos-dir sh
sh-4.4$ cat /etc/bareos/bareos-dir.d/fileset/Catalog.conf 
FileSet {
  Name = "Catalog"
  Description = "Backup the catalog dump and Bareos configuration files."
  Include {
    Options {
      Signature = XXH128
    }
    File = "/var/lib/bareos/bareos.sql" # database dump
    File = "/etc/bareos"                  # configuration
  }
}

we now have our first success for this 'self backup' job, which also includes the config for all but the Web-UI containers via the docker image set's use of a single shared Rockstor Share for all mutual config.

There-after the files contained within related backups for this Director-local client can be viewed via the restore Web-UI capability:

Restore-options-re-defalut-BackupCatalog-job

N.B. upon uninstall and re-install it is required to re-establish the containers link to the by-hand created rock-net.

As a server-set, with ordered install, we can inherit
(via the shared config arrangement) all intra Bareos
service passwords. File and Storage containers are now
able to self-config when uses in this intended AIO setup.
This is akin to the rpms default behaviour of reading
each-others config in order to auto-setup the initial server
set.

Includes minor Description rewording.
Intended to enable access to Rockstor as an SMTP relay;
enabling the default Bareos email notification with minimum
setup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bareos backup server
3 participants