Logrotate #1387

PromoFaux · 2023-07-19T16:42:47Z

We need a mechanism to ensure logrotate is working

edgd1er · 2023-07-19T17:47:31Z

ideas:

check if log is listed in var/lib/logrotate/status
according to rotation (daily, weekly, monthly), log creation date is ok.

https://serverfault.com/questions/189320/how-can-i-monitor-what-logrotate-is-doing

buxm · 2024-02-20T22:01:00Z

So I suppose mine is a known issue...
I've just realised my pihole.log is 8GB and my FTL.log is 70MB.
Both stopped rotating back in August 2022.
I'm running release 2024.02.0 on Synology DSM 7.1.1-42962 Update 6.

Here is what I've tried:

#docker exec -it pihole1 bash
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data    999  290 Aug 25  2022 .
drwxr-xr-x 1 root     root    174 Feb 20 22:33 ..
-rw-r--r-- 1 www-data pihole  70M Feb 20 22:42 FTL.log
-rwx------ 1 root     root    14K Aug 25  2022 FTL.log.1
-rwx------ 1 root     root   2.5K Aug 24  2022 FTL.log.2.gz
-rwx------ 1 root     root    20K Aug 23  2022 FTL.log.3.gz
-rwx------ 1 root     root    19M Jul  9  2022 pihole-FTL.log
-rw-r--r-- 1 www-data pihole 8.0G Feb 20 22:49 pihole.log
-rwx------ 1 root     root   9.3M Aug 25  2022 pihole.log.1
-rwx------ 1 root     root   849K Aug 24  2022 pihole.log.2.gz
-rwx------ 1 root     root   703K Aug 23  2022 pihole.log.3.gz
-rwx------ 1 root     root   880K Aug 22  2022 pihole.log.4.gz
-rwx------ 1 www-data root   8.4K Feb 20 22:05 pihole_updateGravity.log
root@pihole1:/# pihole flush once
  [i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because the file owner is wrong (should be root or user with uid 0).
  [✓] Flushed /var/log/pihole/pihole.log
  [✓] Deleted  queries from database

I've tried to force root ownership of the log files:

root@pihole1:/# chown root:root /var/log/pihole/pihole.log /var/log/pihole/pihole-FTL.log
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data    999  290 Aug 25  2022 .
drwxr-xr-x 1 root     root    174 Feb 20 22:33 ..
-rw-r--r-- 1 www-data pihole  70M Feb 20 22:50 FTL.log
-rwx------ 1 root     root    14K Aug 25  2022 FTL.log.1
-rwx------ 1 root     root   2.5K Aug 24  2022 FTL.log.2.gz
-rwx------ 1 root     root    20K Aug 23  2022 FTL.log.3.gz
-rwx------ 1 root     root    19M Jul  9  2022 pihole-FTL.log
-rw-r--r-- 1 root     root   8.0G Feb 20 22:54 pihole.log
-rwx------ 1 root     root   9.3M Aug 25  2022 pihole.log.1
-rwx------ 1 root     root   849K Aug 24  2022 pihole.log.2.gz
-rwx------ 1 root     root   703K Aug 23  2022 pihole.log.3.gz
-rwx------ 1 root     root   880K Aug 22  2022 pihole.log.4.gz
-rwx------ 1 www-data root   8.4K Feb 20 22:05 pihole_updateGravity.log
root@pihole1:/# pihole flush once
  [i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because the file owner is wrong (should be root or user with uid 0).
  [✓] Flushed /var/log/pihole/pihole.log
  [✓] Deleted  queries from database

If I restart the container, the ownership of the pihole.log is restored to the previous values:

#docker restart pihole1
pihole1
# docker exec -it pihole1 bash
root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data    999  290 Aug 25  2022 .
drwxr-xr-x 1 root     root    174 Feb 20 22:33 ..
-rw-r--r-- 1 www-data pihole  70M Feb 20 22:55 FTL.log
-rwx------ 1 root     root    14K Aug 25  2022 FTL.log.1
-rwx------ 1 root     root   2.5K Aug 24  2022 FTL.log.2.gz
-rwx------ 1 root     root    20K Aug 23  2022 FTL.log.3.gz
-rwx------ 1 root     root    19M Jul  9  2022 pihole-FTL.log
-rw-r--r-- 1 www-data pihole 8.0G Feb 20 22:55 pihole.log
-rwx------ 1 root     root   9.3M Aug 25  2022 pihole.log.1
-rwx------ 1 root     root   849K Aug 24  2022 pihole.log.2.gz
-rwx------ 1 root     root   703K Aug 23  2022 pihole.log.3.gz
-rwx------ 1 root     root   880K Aug 22  2022 pihole.log.4.gz
-rwx------ 1 www-data root   8.4K Feb 20 22:05 pihole_updateGravity.log

I'm running pihole using the following docker-compose.yaml

version: '3.7'

networks:
  
  # references externally defined network host_lan, see docker-create-host_lan.sh
  host_lan:
    external: true
  
services:
  pihole:
    container_name: ${PIHOLE_CONTAINER_NAME1}
    image: ${PIHOLE_IMG_NAME}:${TAG:-latest}
    hostname: ${PIHOLE_HOSTNAME1}
    domainname: ${DOMAIN_NAME}

    networks:
      # connect to host_lan with static IP address
      host_lan:
        ipv4_address: ${IPV4_PIHOLE1}
    
    # set DNS servers required by pihole
    dns:
      - 127.0.0.1
      - 8.8.8.8

    # set environment as required by pihole
    env_file: pihole.env
    environment:
      
      FTLCONF_LOCAL_IPV4: ${IPV4_PIHOLE1}
      
    volumes:
    # set volume binding as required by pihole
      - ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/pihole:/etc/pihole
      - ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/dnsmasq.d:/etc/dnsmasq.d
      - ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/log/:/var/log/pihole
    
    restart: unless-stopped

    healthcheck:
      disable: true

and this is my pihole.env

WEBPASSWORD=xxxxxxx
TZ=Europe/Rome
VIRTUAL_HOST=pihole.mydomain.com
REV_SERVER=true
REV_SERVER_TARGET=192.168.1.80
REV_SERVER_DOMAIN=mydomain.com
REV_SERVER_CIDR=192.168.1.1/24
PIHOLE_DNS_=192.168.1.80
WEBTHEME=default-auto
PIHOLE_UID=1041
PIHOLE_GID=100
WEB_UID=1041
WEB_GID=100

rdwebdesign · 2024-02-20T23:43:15Z

Do you see the same behavior if you remove WEB_UID and WEB_GID variables?

buxm · 2024-02-21T19:11:16Z

@rdwebdesign yes, even if I remove those variables and recreate the container, I see almost the same behaviour.
The ownership of the log files differs, as follows:

root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 pihole   999  290 Aug 25  2022 .
drwxr-xr-x 1 root   root   174 Feb 21 20:08 ..
-rw-r--r-- 1 pihole users  70M Feb 21 20:08 FTL.log
-rwx------ 1 root   root   14K Aug 25  2022 FTL.log.1
-rwx------ 1 root   root  2.5K Aug 24  2022 FTL.log.2.gz
-rwx------ 1 root   root   20K Aug 23  2022 FTL.log.3.gz
-rwx------ 1 root   root   19M Jul  9  2022 pihole-FTL.log
-rw-r--r-- 1 pihole users 8.0G Feb 21 20:08 pihole.log
-rwx------ 1 root   root  9.3M Aug 25  2022 pihole.log.1
-rwx------ 1 root   root  849K Aug 24  2022 pihole.log.2.gz
-rwx------ 1 root   root  703K Aug 23  2022 pihole.log.3.gz
-rwx------ 1 root   root  880K Aug 22  2022 pihole.log.4.gz
-rwx------ 1 pihole root  8.4K Feb 20 22:05 pihole_updateGravity.log

instead, here is the ownership with the two variables set:

root@pihole1:/# ls -lah /var/log/pihole
total 8.1G
drwx------ 1 www-data    999  290 Aug 25  2022 .
drwxr-xr-x 1 root     root    174 Feb 21 20:10 ..
-rw-r--r-- 1 www-data pihole  70M Feb 21 20:12 FTL.log
-rwx------ 1 root     root    14K Aug 25  2022 FTL.log.1
-rwx------ 1 root     root   2.5K Aug 24  2022 FTL.log.2.gz
-rwx------ 1 root     root    20K Aug 23  2022 FTL.log.3.gz
-rwx------ 1 root     root    19M Jul  9  2022 pihole-FTL.log
-rw-r--r-- 1 www-data pihole 8.0G Feb 21 20:16 pihole.log
-rwx------ 1 root     root   9.3M Aug 25  2022 pihole.log.1
-rwx------ 1 root     root   849K Aug 24  2022 pihole.log.2.gz
-rwx------ 1 root     root   703K Aug 23  2022 pihole.log.3.gz
-rwx------ 1 root     root   880K Aug 22  2022 pihole.log.4.gz
-rwx------ 1 www-data root   8.4K Feb 20 22:05 pihole_updateGravity.log

In both cases pihole flush once command fails as follows:

root@pihole1:/# pihole flush once
  [i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because it is writable by group or others.
  [✓] Flushed /var/log/pihole/pihole.log
  [✓] Deleted  queries from database

iamdoubz · 2024-03-29T21:35:29Z

I went to "mirror" my PiHole to another server and I saw that my pihole.log was over 11GB!

There is a logrotate file in my "pihole" storage directory and it contains the right data to logrotate, but doesn't appear to run. There are 6 files in the /etc/logrotate.d directory: alternatives, apt, btmp, dpkg, lighttpd, wtmp. But no pihole. I copied the contents of these files, created a new directory to store these files, copied the files there, then also copied the logrotate file.

Initially, I got an error, but it wanted the files to be "root" created so I did that.

Then I tested logrotate with command logrotate -d /etc/logrotate.d/logrotate. It appeared to work!

Hope this helps someone...

Update

Another error I got was that there is no /var/lib/logrotate/status file. The other thing I noticed is that there is not a crontab setup for the root user. But, idk how logrotate can be called in other ways.

PromoFaux · 2024-03-29T22:04:50Z

Have just checked my running v6 container and logrotate certainly appears to be working correctly here:

dev-v6:/# ls -lah /var/log/pihole
total 9.0M
drwxr-xr-x 1 pihole pihole  348 Mar 29 00:00 .
drwxr-xr-x 1 root   root     60 Mar 15 21:30 ..
-rw-r----- 1 pihole pihole  14K Mar 29 21:57 FTL.log
-rw-r----- 1 pihole pihole  15K Mar 29 00:00 FTL.log.1
-rw-r----- 1 pihole pihole 1.4K Mar 28 00:00 FTL.log.2.gz
-rw-r----- 1 pihole pihole 1.3K Mar 27 00:00 FTL.log.3.gz
-rw-r----- 1 pihole pihole 4.8M Mar 29 21:57 pihole.log
-rw-r----- 1 pihole pihole 3.1M Mar 29 00:00 pihole.log.1
-rw-r----- 1 pihole pihole 413K Mar 28 00:00 pihole.log.2.gz
-rw-r----- 1 pihole pihole 286K Mar 27 00:00 pihole.log.3.gz
-rw-r----- 1 pihole pihole 236K Mar 26 00:00 pihole.log.4.gz
-rw-r----- 1 pihole pihole 205K Mar 25 00:00 pihole.log.5.gz
-rw-r--r-- 1 root   root   1.2K Mar 23 03:25 pihole_updateGravity.log
-rw-r----- 1 pihole pihole    0 Mar 16 00:00 webserver.log
-rw-r----- 1 pihole pihole  424 Mar 16 00:00 webserver.log.1

Which is fired nightly by cron

dev-v6:/# cat /etc/crontabs/root 
25 3 * * 6 PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updateGravity >/var/log/pihole/pihole_updateGravity.log || cat /var/log/pihole/pihole_updateGravity.log
00 00 * * * PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole flush once quiet
3 19 * * * PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker

Looking at the command that is run (pihole flush once quiet) we see this code:

And the same in my v5 container:

root@docker-pihole-pi4:/# ls -lah /var/log/pihole
total 158M
drwxr-xr-x 1 root   root   4.0K Mar 29 21:55 .
drwxr-xr-x 1 root   root   4.0K Mar  1 06:44 ..
-rw-r--r-- 1 pihole pihole 216K Mar 29 21:58 FTL.log
-rw-r--r-- 1 pihole pihole  99M Mar 29 21:55 FTL.log.1
-rw-r--r-- 1 pihole pihole 7.5M Mar 29 00:00 FTL.log.2.gz
-rw-r--r-- 1 pihole pihole 8.8M Mar 28 00:00 FTL.log.3.gz
-rw-r--r-- 1 pihole pihole  75K Mar 29 21:58 pihole.log
-rw-r--r-- 1 pihole pihole  33M Mar 29 21:55 pihole.log.1
-rw-r--r-- 1 pihole pihole 2.2M Mar 29 00:00 pihole.log.2.gz
-rw-r--r-- 1 pihole pihole 2.5M Mar 28 00:00 pihole.log.3.gz
-rw-r--r-- 1 pihole pihole 2.9M Mar 27 00:00 pihole.log.4.gz
-rw-r--r-- 1 pihole pihole 2.9M Mar 26 00:00 pihole.log.5.gz
-rw-r----- 1 root   pihole  34K Jan  3 17:46 pihole_debug.log
-rw-r--r-- 1 root   root   1.2K Mar 24 04:33 pihole_updateGravity.log

PromoFaux · 2024-05-08T19:34:58Z

Does anyone in this thread still have problems with the logs not being rotated?

buxm · 2024-05-08T20:38:28Z

@PromoFaux yes, the problems I reported earlier are still there:

$ sudo docker exec -it pihole1 bash
root@pihole1:/# pihole flush once
  [i] Flushing /var/log/pihole/pihole.log ...error: Ignoring /etc/pihole/logrotate because the file owner is wrong (should be root or user with uid 0).
  [✓] Flushed /var/log/pihole/pihole.log
  [✓] Deleted  queries from database
root@pihole1:/#
exit
$ ls -lah /volume1/docker/pihole1/log/
total 9.6G
drwxrwxrwx+ 1 pihole synopkgs  290 Aug 25  2022 .
drwxrwxrwx+ 1 pihole users      12 Jan 26  2020 ..
-rw-r--r--  1 pihole     1000  96M May  8 22:34 FTL.log
-rwxrwxrwx+ 1 root   root      14K Aug 25  2022 FTL.log.1
-rwxrwxrwx+ 1 root   root     2.5K Aug 24  2022 FTL.log.2.gz
-rwxrwxrwx+ 1 root   root      20K Aug 23  2022 FTL.log.3.gz
-rwxrwxrwx+ 1 root   root      19M Jul  9  2022 pihole-FTL.log
-rw-r--r--  1 pihole     1000 9.5G May  8 22:34 pihole.log
-rwxrwxrwx+ 1 root   root     9.3M Aug 25  2022 pihole.log.1
-rwxrwxrwx+ 1 root   root     849K Aug 24  2022 pihole.log.2.gz
-rwxrwxrwx+ 1 root   root     703K Aug 23  2022 pihole.log.3.gz
-rwxrwxrwx+ 1 root   root     880K Aug 22  2022 pihole.log.4.gz
-rwxrwxrwx+ 1 pihole root     7.5K May  5 03:42 pihole_updateGravity.log

PromoFaux · 2024-05-08T20:41:31Z

What does your compose file look like? (Also, have you tried the beta v6 image?)

buxm · 2024-05-08T20:48:23Z

my compose file:

version: '3.7'

networks:
  
  # references externally defined network host_lan, see docker-create-host_lan.sh
  host_lan:
    external: true
  
services:
  pihole:
    container_name: ${PIHOLE_CONTAINER_NAME1}
    image: ${PIHOLE_IMG_NAME}:${TAG:-latest}
    hostname: ${PIHOLE_HOSTNAME1}
    domainname: ${DOMAIN_NAME}

    networks:
      # connect to host_lan with static IP address
      host_lan:
        ipv4_address: ${IPV4_PIHOLE1}
    
    # set DNS servers required by pihole
    dns:
      - 127.0.0.1
      - 8.8.8.8

    # set environment as required by pihole
    env_file: pihole.env
    environment:
      
      FTLCONF_LOCAL_IPV4: ${IPV4_PIHOLE1}
      
    volumes:
    # set volume binding as required by pihole
      - ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/pihole:/etc/pihole
      - ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/etc/dnsmasq.d:/etc/dnsmasq.d
      - ${HOST_DOCKER_DIR}/${PIHOLE_CONTAINER_NAME1}/log/:/var/log/pihole
    
    restart: unless-stopped

    healthcheck:
      disable: true

my pihole.env:

WEBPASSWORD=xxxxxxx
TZ=Europe/Rome
VIRTUAL_HOST=pihole.mydomain.com
REV_SERVER=true
REV_SERVER_TARGET=192.168.1.80
REV_SERVER_DOMAIN=mydomain.com
REV_SERVER_CIDR=192.168.1.1/24
PIHOLE_DNS_=192.168.1.80
WEBTHEME=default-auto
PIHOLE_UID=1041
PIHOLE_GID=100
WEB_UID=1041
WEB_GID=100

No, I haven't tried the beta v6 image.

PromoFaux · 2024-05-08T20:57:56Z

PIHOLE_UID=1041
PIHOLE_GID=100
WEB_UID=1041
WEB_GID=100

Here in lies the root cause of the issue.

See the Experimental Variables section of the README file. PIHOLE_UID and WEB_UIID must not be the same

buxm · 2024-05-12T17:00:50Z

Thank you, I hadn't seen that...
But now I'm going through permission hell...
If I provide no UID and no GID variable, FTL won't start.
If I provide only PIHOLE_UID and PIHOLE_UID (matching a specific user and group created on my Synology NAS host) but no WEB_?ID variable, then the web interface doesn't show anything meaningful apart from permission errors in the settings page.
But then should I create also a user on my Synology NAS for the web interface? Does the setup need to be so complex or is anything wrong with mine? Confused....

PromoFaux · 2024-05-12T17:18:43Z

Ah, Synology NAS:

#963

Actually, at this stage I'd recommend using the v6 beta image instead as it's fairly "close" to release, which removes the WEB_?ID variables as the web page is now served by pihole-FTL itself.

I'm currently running the v6 images on my Synology with DNSMASQ_USER set to pihole (which is the default anyway) and as I'm using a named volume rather than a host mount - I haven't needed to set the pihole_?id variables

Take a read over the new Readme first, though, as you'll need to make a few changes to your environment variables

buxm · 2024-05-13T09:57:26Z

Thank you for pointing that out.
For the moment I reverted back to my previous situation, i.e. keeping my UID's and GID's the same (although it's wrong) and also stick to image 2024.03.2, since also upgrading to 2024.05.0 by itself seemed to disrupt my pihole (maybe because of my wrong variables).
Now at least I've got my pihole working. I've lived with my huge log file for quite a while, so I'll wait a bit more for v6 to be released, since it's fairly close. Then I'll try to upgrade and hopefully fix the logrotate issue.
Things seem pretty messy with my current set up, I'd rather avoid adding potential new issues due to beta releases at this point.

magikid · 2024-07-03T18:26:12Z

Going back to the original issue raised, what should happen if the system detects that logrotate isn’t running properly?

PromoFaux · 2024-07-03T18:54:38Z

It's a good question - I'm not 100% sure there is an answer to it. I suppose we could set a cron script to periodically check the size of the logs - but this does not seem like an ideal solution.

I created this issue before I'd really looked into whether or not logrotate was working in the image... as it happens, it appears to be. Do you have any suggestions?

buxm · 2024-07-03T20:50:09Z

How do you generate messages like this?

Maybe at the same time as you check the CPU load you can check the log size and/or age of last archived log. Then generate a message accordingly.

PromoFaux · 2024-07-05T17:01:09Z

The more I think of it, with a properly configured container - there shouldn't be an issue with logrotate.

That diagnosis table is populated by FTL - not sure it would be a good idea for FTL to monitor the size of logs / log rotation status. Certainly on size, what is large for one user is tiny for another.

@buxm's issue appears to be caused by some permission issues on the filesystem - and I've a feeling it would be sorted out with the v6 image in any case

magikid · 2024-07-09T01:14:14Z

It's a good question - I'm not 100% sure there is an answer to it. I suppose we could set a cron script to periodically check the size of the logs - but this does not seem like an ideal solution.

I created this issue before I'd really looked into whether or not logrotate was working in the image... as it happens, it appears to be. Do you have any suggestions?

My initial thought was a script to check that the logrotate location was writable by the proper user, write something to the log, force a rotate, and check it's non-zero. If any of those fail, error and alert the user somehow.

The more I think about it though, the less I think anything should be done. This is a giant container doing a lot of things. If logrotate works in the base image, no need to overcomplicate its setup and end up scaring off a newcomer who can't diagnose what logrotate is and why it might be throwing an error.

PromoFaux added the v6 label Jul 19, 2023

PromoFaux added this to Pi-hole v6 Jul 19, 2023

PromoFaux moved this to 📋 Backlog in Pi-hole v6 Jul 19, 2023

quinte17 mentioned this issue Oct 27, 2024

logrotate: fix permissions on configfile #1655

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Logrotate #1387

Logrotate #1387

PromoFaux commented Jul 19, 2023

edgd1er commented Jul 19, 2023

buxm commented Feb 20, 2024

rdwebdesign commented Feb 20, 2024

buxm commented Feb 21, 2024 •

edited

Loading

iamdoubz commented Mar 29, 2024 •

edited

Loading

PromoFaux commented Mar 29, 2024

PromoFaux commented May 8, 2024

buxm commented May 8, 2024

PromoFaux commented May 8, 2024

buxm commented May 8, 2024

PromoFaux commented May 8, 2024

buxm commented May 12, 2024

PromoFaux commented May 12, 2024

buxm commented May 13, 2024

magikid commented Jul 3, 2024

PromoFaux commented Jul 3, 2024

buxm commented Jul 3, 2024

PromoFaux commented Jul 5, 2024

magikid commented Jul 9, 2024

Logrotate #1387

Logrotate #1387

Comments

PromoFaux commented Jul 19, 2023

edgd1er commented Jul 19, 2023

buxm commented Feb 20, 2024

rdwebdesign commented Feb 20, 2024

buxm commented Feb 21, 2024 • edited Loading

iamdoubz commented Mar 29, 2024 • edited Loading

Update

PromoFaux commented Mar 29, 2024

PromoFaux commented May 8, 2024

buxm commented May 8, 2024

PromoFaux commented May 8, 2024

buxm commented May 8, 2024

PromoFaux commented May 8, 2024

buxm commented May 12, 2024

PromoFaux commented May 12, 2024

buxm commented May 13, 2024

magikid commented Jul 3, 2024

PromoFaux commented Jul 3, 2024

buxm commented Jul 3, 2024

PromoFaux commented Jul 5, 2024

magikid commented Jul 9, 2024

buxm commented Feb 21, 2024 •

edited

Loading

iamdoubz commented Mar 29, 2024 •

edited

Loading