Daily backup hangs waiting for Apache to become available

[holocronweaver]

Steps to reproduce

1. Set a daily backup schedule in AIO.
2. Wait for backup to trigger.

Expected behavior

Backup completes.

Actual behavior

Backup begins then retries indefinitely waiting for Apache, even though the site is available without any issues:

2024-09-03T11:30:07.943420364Z Daily backup script has started
2024-09-03T11:30:12.990436098Z nc: getaddrinfo for host "nextcloud-aio-apache" port 11000: Try again
2024-09-03T11:30:12.990552154Z Waiting for apache to become available
2024-09-03T11:30:48.016165775Z nc: getaddrinfo for host "nextcloud-aio-apache" port 11000: Try again
2024-09-03T11:30:48.016605317Z Waiting for apache to become available
2024-09-03T11:31:23.044133187Z nc: getaddrinfo for host "nextcloud-aio-apache" port 11000: Try again
2024-09-03T11:31:23.044219703Z Waiting for apache to become available
...

Other information

Host OS

Ubuntu 22.04

Output of sudo docker info

Client: Docker Engine - Community
 Version:    27.2.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 17
  Running: 14
  Paused: 0
  Stopped: 3
 Images: 36
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 472731909fa34bd7bc9c087e4c27943f9835f111
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.0-119-generic
 Operating System: Ubuntu 22.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.58GiB
 Name: <redacted>
 ID: 779d1acb-3577-451a-a2f6-c12cf0ec10b3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Docker run command or docker-compose file that you used

volumes:
  nextcloud_aio_mastercontainer:
    name: nextcloud_aio_mastercontainer # This line is not allowed to be changed

services:
  nextcloud:
    image: nextcloud/all-in-one:latest
    restart: always
    container_name: nextcloud-aio-mastercontainer # This line is not allowed to be changed
    volumes:
      - nextcloud_aio_mastercontainer:/mnt/docker-aio-config # This line is not allowed to be changed
      - /var/run/docker.sock:/var/run/docker.sock:ro # May be changed on macOS, Windows or docker rootless. See the applicable documentation. If adjusting, don't forget to also set 'DOCKER_SOCKET_PATH'!
    ports:
      - 8080:8080
    environment: # Is needed when using any of the options below
      - APACHE_PORT=11000 # Is needed when running behind a web server or reverse proxy (like Apache, Nginx and else). See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
      - APACHE_IP_BINDING=127.0.0.1 # Should be set when running behind a web server or reverse proxy (like Apache, Nginx and else) that is running on the same host. See https://github.com/nextcloud/all-in-one/blob/main/reverse-proxy.md
      - COLLABORA_SECCOMP_DISABLED=false # Setting this to true allows to disable Collabora's Seccomp feature. See https://github.com/nextcloud/all-in-one#how-to-disable-collaboras-seccomp-feature
      # - DOCKER_SOCKET_PATH=/var/run/docker.sock # Needs to be specified if the docker socket on the host is not located in the default '/var/run/docker.sock'. Otherwise mastercontainer updates will fail. For macos it needs to be '/var/run/docker.sock'
      - DISABLE_BACKUP_SECTION=false # Setting this to true allows to hide the backup section in the AIO interface.
      - NEXTCLOUD_DATADIR=/path/to/my/data # Allows to set the host directory for Nextcloud's datadir. See https://github.com/nextcloud/all-in-one#how-to-change-the-default-location-of-nextclouds-datadir
      # - NEXTCLOUD_MOUNT=/mnt/ # Allows the Nextcloud container to access the chosen directory on the host. See https://github.com/nextcloud/all-in-one#how-to-allow-the-nextcloud-container-to-access-directories-on-the-host
      - NEXTCLOUD_UPLOAD_LIMIT=10G # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-upload-limit-for-nextcloud
      - NEXTCLOUD_MAX_TIME=3600 # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-max-execution-time-for-nextcloud
      - NEXTCLOUD_MEMORY_LIMIT=512M # Can be adjusted if you need more. See https://github.com/nextcloud/all-in-one#how-to-adjust-the-php-memory-limit-for-nextcloud
      # - NEXTCLOUD_TRUSTED_CACERTS_DIR=/path/to/my/cacerts # CA certificates in this directory will be trusted by the OS of the nexcloud container (Useful e.g. for LDAPS) See See https://github.com/nextcloud/all-in-one#how-to-trust-user-defiend-certification-authorities-ca
      - NEXTCLOUD_STARTUP_APPS=deck twofactor_totp tasks calendar contacts notes # Allows to modify the Nextcloud apps that are installed on starting AIO the first time. See https://github.com/nextcloud/all-in-one#how-to-change-the-nextcloud-apps-that-are-installed-on-the-first-startup
      # - NEXTCLOUD_ADDITIONAL_APKS=imagemagick # This allows to add additional packages to the Nextcloud container permanently. Default is imagemagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-os-packages-permanently-to-the-nextcloud-container
      # - NEXTCLOUD_ADDITIONAL_PHP_EXTENSIONS=imagick # This allows to add additional php extensions to the Nextcloud container permanently. Default is imagick but can be overwritten by modifying this value. See https://github.com/nextcloud/all-in-one#how-to-add-php-extensions-permanently-to-the-nextcloud-container
      # - NEXTCLOUD_ENABLE_DRI_DEVICE=true # This allows to enable the /dev/dri device in the Nextcloud container which is needed for hardware-transcoding. See https://github.com/nextcloud/all-in-one#how-to-enable-hardware-transcoding-for-nextcloud
      - TALK_PORT=3478 # This allows to adjust the port that the talk container is using.

Other valuable info

Manual backups complete successfully, and auto backups can work a single time if I set it to be a few minutes after I set the start time. However auto backups never work if set more than an hour out, always hanging with above error.

I stop/start the AIO container daily while doing whole-server backups, not sure if restarting the container causes an issue with daily backups. (There is no overlap between my whole-server and daily backup times, they are ~5 hours apart.)

Here is the generated auto backup script from the logs:

2024-09-03T09:48:33.888020546Z Trying to fix docker.sock permissions internally...
2024-09-03T09:48:33.904371775Z Adding internal www-data to group ping
2024-09-03T09:48:35.597820877Z �[0;92mInitial startup of Nextcloud All-in-One complete!
2024-09-03T09:48:35.597845189Z You should be able to open the Nextcloud AIO Interface now on port 8080 of this server!
2024-09-03T09:48:35.597850420Z E.g. https://internal.ip.of.this.server:8080
2024-09-03T09:48:35.597854515Z ⚠️ Important: do always use an ip-address if you access this port and not a domain as HSTS might block access to it later!
2024-09-03T09:48:35.597859228Z 
2024-09-03T09:48:35.597862955Z If your server has port 80 and 8443 open and you point a domain to your server, you can get a valid certificate automatically by opening the Nextcloud AIO Interface via:
2024-09-03T09:48:35.597867202Z https://your-domain-that-points-to-this-server.tld:8443�[0m
2024-09-03T09:48:37.504799417Z ++ head -1 /mnt/docker-aio-config/data/daily_backup_time
2024-09-03T09:48:37.505708501Z + BACKUP_TIME=11:30
2024-09-03T09:48:37.505722994Z + export BACKUP_TIME
2024-09-03T09:48:37.505728062Z + export DAILY_BACKUP=1
2024-09-03T09:48:37.505732277Z + DAILY_BACKUP=1
2024-09-03T09:48:37.512290967Z ++ sed -n 2p /mnt/docker-aio-config/data/daily_backup_time
2024-09-03T09:48:37.512803151Z + '[' '' '!=' automaticUpdatesAreNotEnabled ']'
2024-09-03T09:48:37.512813104Z + export AUTOMATIC_UPDATES=1
2024-09-03T09:48:37.512817767Z + AUTOMATIC_UPDATES=1
2024-09-03T09:48:37.521644486Z ++ sed -n 3p /mnt/docker-aio-config/data/daily_backup_time
2024-09-03T09:48:37.522593163Z + '[' '' '!=' successNotificationsAreNotEnabled ']'
2024-09-03T09:48:37.522608292Z + export SEND_SUCCESS_NOTIFICATIONS=1
2024-09-03T09:48:37.522613084Z + SEND_SUCCESS_NOTIFICATIONS=1
2024-09-03T09:48:37.522617139Z + set +x
2024-09-03T09:48:37.551106765Z [Tue Sep 03 09:48:37.550862 2024] [mpm_event:notice] [pid 142:tid 142] AH00489: Apache/2.4.62 (Unix) OpenSSL/3.3.1 configured -- resuming normal operations
2024-09-03T09:48:37.551300517Z [Tue Sep 03 09:48:37.551204 2024] [core:notice] [pid 142:tid 142] AH00094: Command line: 'httpd -D FOREGROUND'
2024-09-03T09:48:37.564471512Z {"level":"info","ts":1725356917.564089,"msg":"using config from file","file":"/Caddyfile"}
2024-09-03T09:48:37.565977205Z {"level":"info","ts":1725356917.565827,"msg":"adapted config to JSON","adapter":"caddyfile"}
2024-09-03T09:48:37.569845694Z [03-Sep-2024 09:48:37] NOTICE: fpm is running, pid 148
2024-09-03T09:48:37.569869026Z [03-Sep-2024 09:48:37] NOTICE: ready to handle connections

Here is my reverse proxy config in Caddy:

https://mysite.com:443 {
        reverse_proxy localhost:11000
}

Please let me know if any additional info would help!

[szaimen]

Hi, can you please run sudo docker exec -it nextcloud-aio-mastercontainer docker ps --format "{{.Names}}" | grep "^nextcloud-aio-apache$"; sudo docker exec -it nextcloud-aio-mastercontainer nc -z nextcloud-aio-apache "11000" and post the output here?

Also, can you follow #4989 and check if that helps?

[holocronweaver]

$ sudo docker exec -it nextcloud-aio-mastercontainer docker ps --format "{{.Names}}" | grep "^nextcloud-aio-apache$"; sudo docker exec -it nextcloud-aio-mastercontainer nc -z nextcloud-aio-apache "11000"
nc: getaddrinfo for host "nextcloud-aio-apache" port 11000: Try again

After running the commands from #4989 , the output is:

$ sudo docker exec -it nextcloud-aio-mastercontainer docker ps --format "{{.Names}}" | grep "^nextcloud-aio-apache$"; sudo docker exec -it nextcloud-aio-mastercontainer nc -z nextcloud-aio-apache "11000" 
Connection to nextcloud-aio-apache (172.25.0.7) 11000 port [tcp/*] succeeded!

So seems promising! Will post the results once the daily backup is triggered.

[holocronweaver]

Unfortunately the issue persists, daily backups stall with the last two lines repeating roughly every 30 seconds for hours.

2024-09-06T11:30:21.666753826Z Daily backup script has started
2024-09-06T11:30:26.731293476Z nc: getaddrinfo for host "nextcloud-aio-apache" port 11000: Try again
2024-09-06T11:30:26.731542675Z Waiting for apache to become available

However once I log into the AIO config page, a backup begins immediately. It seems to be that the daily backup cannot complete until I am logged in for some reason:

2024-09-06T18:22:30.153381608Z Connection to nextcloud-aio-apache (172.25.0.7) 11000 port [tcp/*] succeeded!
2024-09-06T18:22:30.189007531Z Starting mastercontainer update...
2024-09-06T18:22:30.189042129Z (The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
2024-09-06T18:22:32.036033030Z Waiting for watchtower to stop
2024-09-06T18:23:02.059391903Z Creating daily backup...
2024-09-06T18:23:18.017828219Z Waiting for backup container to stop
2024-09-06T18:23:48.035303338Z Waiting for backup container to stop
2024-09-06T18:24:18.053113523Z Waiting for backup container to stop
2024-09-06T18:24:48.070931798Z Waiting for backup container to stop
2024-09-06T18:25:18.088124231Z Waiting for backup container to stop
2024-09-06T18:25:48.105222646Z Starting and updating containers...
2024-09-06T18:25:56.973783784Z Waiting for the Nextcloud container to start
2024-09-06T18:26:27.011195517Z Connection to nextcloud-aio-nextcloud (172.25.0.5) 9000 port [tcp/*] succeeded!
2024-09-06T18:26:27.011253392Z Sending backup notification...
2024-09-06T18:26:29.919904162Z Daily backup script has finished

The positive output of the command also persists, so perhaps there is progress towards a solution?

$ sudo docker exec -it nextcloud-aio-mastercontainer docker ps --format "{{.Names}}" | grep "^nextcloud-aio-apache$"; sudo docker exec -it nextcloud-aio-mastercontainer nc -z nextcloud-aio-apache "11000" 
Connection to nextcloud-aio-apache (172.25.0.7) 11000 port [tcp/*] succeeded!

[szaimen]

I stop/start the AIO container daily while doing whole-server backups, not sure if restarting the container causes an issue with daily backups.

How exactly do you do this? (Which commands are involved)?

[holocronweaver]

I typically do docker compose down followed by docker compose up -d. I do this vs. stop/start so all containers are started with a fresh state, and maybe updated if applicable.

Your question gave me a hint, so I tried switching to stop/start and it worked!

2024-09-08T11:30:11.792315307Z Daily backup script has started
2024-09-08T11:30:11.834582476Z Connection to nextcloud-aio-apache (172.25.0.7) 11000 port [tcp/*] succeeded!
2024-09-08T11:30:11.871058789Z Starting mastercontainer update...
2024-09-08T11:30:11.871088059Z (The script might get exited due to that. In order to update all the other containers correctly, you need to run this script with the same settings a second time.)
2024-09-08T11:30:13.944188021Z Waiting for watchtower to stop
2024-09-08T11:30:43.965921692Z Creating daily backup...
2024-09-08T11:31:00.634397081Z Waiting for backup container to stop
2024-09-08T11:31:30.653617011Z Waiting for backup container to stop
2024-09-08T11:32:00.673962941Z Starting and updating containers...
2024-09-08T11:32:08.972480250Z Waiting for the Nextcloud container to start
2024-09-08T11:32:39.007697777Z Connection to nextcloud-aio-nextcloud (172.25.0.5) 9000 port [tcp/*] succeeded!
2024-09-08T11:32:39.007974161Z Sending backup notification...
2024-09-08T11:32:41.873537404Z Daily backup script has finished

No other container service I run is affected by down/up. Is AIO the exception because it relies on a specific network config that is only fully set up after logging into the AIO interface? Is that why following a down/up the backup only succeeds after I log into the interface?

If this is expected behavior, might be worth documenting (unless it already is and I missed it somewhere?).

[szaimen]

Is AIO the exception because it relies on a specific network config that is only fully set up after logging into the AIO interface? Is that why following a down/up the backup only succeeds after I log into the interface?

Yes, this is the reason.

If you want to keep doing it like that, you need to add the mastercontainer to the nextcloud-aio network via your docker-compose file.

However better would probably be to either use https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-stopstartupdate-containers-or-trigger-the-daily-backup-from-a-script-externally or https://github.com/nextcloud/all-in-one?tab=readme-ov-file#how-to-enable-automatic-updates-without-creating-a-backup-beforehand

[Crocmagnon]

If you want to keep doing it like that, you need to add the mastercontainer to the nextcloud-aio network via your docker-compose file.

This solved the issue for me, thanks 👍🏻

[holocronweaver]

Thanks Simon! I will think more about which of these options works best in my situation.

Github issues are sort of de facto documentation, so hopefully this issue will help anyone googling this problem.

Closing since my auto-backups are working after switching from docker compose down to docker compose stop.