Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Time-based Automated Conversion #342

Open
wants to merge 9 commits into
base: master
Choose a base branch
from

Conversation

DrWarpMan
Copy link

Adds the ability to allow automated conversion to work only in certain time ranges via AUTOMATED_CONVERSION_ALLOWED_TIME_RANGES variable.

Examples of how can you set the AUTOMATED_CONVERSION_ALLOWED_TIME_RANGES variable:

  • Tue-14-18: Convert videos only on Tuesday, between 14:00 and 17:59.
  • Wed-9-17,Thu-9-17,Fri-9-17: Convert videos only on Wednesday, Thursday and Friday, between 9:00 and 16:59.
  • Sun-1-2: Convert videos only on Sunday, between 1:00 and 1:59. Equivalent to Sun-1.
  • Mon-0-24: Convert videos only on Monday, at any hour. Equivalent to Mon.
  • Fri-18-24,Sat,Sun,Mon-0-6: Convert videos from Friday 18:00 to Monday 5:59.

Detailed usage is documented in the README.

Closes #340

@aronmgv
Copy link

aronmgv commented May 3, 2024

@DrWarpMan thanks for having time to look at it and make this PR.. however it doesn't address the problem when the conversion starts e.g. 1 minute before the given slot for no conversion time slot..

Having this approached to be changed in a way we could pause the thread when we happen to enter no conversion time slot might be difficult so let me politely ask @jlesage to have a look and let us know if this PR can be rewritten in such a way..

  • dont start new conversion when current time is within the no conversion time slot
  • pause existing conversion when we enter no conversion time slot

Appreciate!

@jlesage
Copy link
Owner

jlesage commented May 3, 2024

I don't think it's worth pausing a conversion when we exit the specified range. I think we have to see that configured ranges define when conversions are allowed to start (and not when they are allowed to run).

do
eval "AC_$VAR=\"\${AUTOMATED_CONVERSION_${VAR}_${DIR#*/watch}:-\$AC_$VAR}\""
done
fi

if ! is_now_within_time_range "$AC_ALLOWED_TIME_RANGES"; then
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not the correct place to do the check. Doing it here means that the whole watch folder will be processed. The watch folder might contains a lot of videos and converting all of them might take multiple hours, going well over the defined range.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check should be done in process_watch_folder(), in the "Process video files" loop.

@aronmgv
Copy link

aronmgv commented May 3, 2024

@jlesage well that's the point.. some videos might take hours to convert and if they happen to start minutes before no conversion time slot they will affect the whole machine since converting videos is quite resourceful process..

I can have a look myself but I would like to get some suggestions from you if it is even possible to pause the conversion thread and where should it be done.. Thanks!

@jlesage
Copy link
Owner

jlesage commented May 3, 2024

Some comments about the range format:

  • I would not allow the end time to be omitted. It is not intuitive for users what happen when not specified. At first, I thought that not specifying means "until the end of the day". Thus, to reduce confusion, the end hour should be mandatory (when the start hour is specified).
  • I think it should be possible to define a range of days. For example, Sat-Sun.
  • Separator between the days and hours should be different. Maybe : instead.

Example for "everyday during night":

  • Sun-Sat:0-6,Sun-Sat:22-24

This would even be better (if day is not defined, we use the previous one):

  • Sun-Sat:0-6,22-24

Example for "Weekdays during night, weekend all the time":

  • Mon-Fri:0-6,22-24,Sat-Sun

@jlesage
Copy link
Owner

jlesage commented May 3, 2024

some videos might take hours to convert and if they happen to start minutes before no conversion time slot they will affect the whole machine since converting videos is quite resourceful process..

If one has strict restrictions about their CPU usage, should it configure the ranges accordingly ? For example, the end hour could be reduced to account for cases where a conversion would be started just before.

Also, note that setting APP_NICENESS variable properly should minimize impact of the conversion on the system.

I can have a look myself but I would like to get some suggestions from you if it is even possible to pause the conversion thread and where should it be done..

I think this could be achieved by sending SIGTSTP or SIGSTOP signal to the HandBrake cli. But I don't know how it would react to these signals.

@aronmgv
Copy link

aronmgv commented May 6, 2024

Also, note that setting APP_NICENESS variable properly should minimize impact of the conversion on the system.

I tried setting APP_NICENESS=15 but cant see much difference.. still the system become a bit laggy Is there a way to verify the NICENESS of the process itself?

What I tried:

root@caradhras:[~]: getpcaps 1567822
1567822: =
root@caradhras:[~]: cat /proc/1567822/status
Name:   HandBrakeCLI
Umask:  0022
State:  S (sleeping)
Tgid:   1567822
Ngid:   0
Pid:    1567822
PPid:   1567018
TracerPid:      0
Uid:    1000    1000    1000    1000
Gid:    1000    1000    1000    1000
FDSize: 256
Groups: 44 105
NStgid: 1567822 1782
NSpid:  1567822 1782
NSpgid: 1567018 1246
NSsid:  1567018 1246
VmPeak:  2744500 kB
VmSize:  2738256 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   2527828 kB
VmRSS:   2523608 kB
RssAnon:         2502308 kB
RssFile:           21300 kB
RssShmem:              0 kB
VmData:  2644740 kB
VmStk:       136 kB
VmExe:     44700 kB
VmLib:     17876 kB
VmPTE:      5276 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
THP_enabled:    1
Threads:        180
SigQ:   1/514633
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: 0000000000000002
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 000001ffffffffff
CapAmb: 0000000000000000
NoNewPrivs:     0
Seccomp:        0
Seccomp_filters:        0
Speculation_Store_Bypass:       thread vulnerable
SpeculationIndirectBranch:      conditional enabled
Cpus_allowed:   ffffffff
Cpus_allowed_list:      0-31
Mems_allowed:   00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        4474
nonvoluntary_ctxt_switches:     104
root@caradhras:[~]: capsh --decode=000001ffffffffff
0x000001ffffffffff=cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read,cap_perfmon,cap_bpf,cap_checkpoint_restore
root@caradhras:[~]:

EDIT:

I just tried to compare APP_NICENESS of 15 vs -15 and didnt see any difference.. Same sever is running a VM where I am right now and it is responsive on the same level no matter the APP_NICENESS value..

version: "3.7"

services:
  handbrake:
    container_name: general.handbrake.50129
    image: jlesage/handbrake
    restart: always
    privileged: true
    cap_add: [ 'SYS_NICE' ]
    ports:
      - 50129:5800
    volumes:
      # CONFIG
      - $PWD/config:/config:rw
      # LOGS
      - /lvm/raid/private/media/.convert/.stats/conversion.log:/config/log/hb/conversion.log:rw
      - /lvm/raid/private/media/.convert/.stats/failed_conversions:/config/failed_conversions:rw
      - /lvm/raid/private/media/.convert/.stats/successful_conversions:/config/successful_conversions:rw
      # STORAGE
      - /lvm/raid/private/media:/storage/media:ro
      - /lvm/raid/private/data:/storage/data:ro
      # OUTPUT FOLDER
      - /lvm/raid/private/media/.convert/@DONE:/output:rw
      # WATCH FOLDERS
      - /lvm/raid/private/media/.convert/${MKV_2160P_ULTRA}:/watch:rw
...
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
      - DARK_MODE=1
      - APP_NICENESS=15
      - DISPLAY_WIDTH=1600
      - DISPLAY_HEIGHT=900
      - AUTOMATED_CONVERSION_MAX_WATCH_FOLDERS=30
      # WATCH 1: MKV_2160P_ULTRA
      - AUTOMATED_CONVERSION_PRESET=Matroska/${MKV_2160P_ULTRA}
      - AUTOMATED_CONVERSION_FORMAT=mkv
      - AUTOMATED_CONVERSION_KEEP_SOURCE=1
      - AUTOMATED_CONVERSION_OUTPUT_DIR=/output
...

networks:
  default:
    name: general.handbrake

@aronmgv
Copy link

aronmgv commented May 7, 2024

I think this could be achieved by sending SIGTSTP or SIGSTOP signal to the HandBrake cli. But I don't know how it would react to these signals.

Tested via CLI and works!! process will pause itself, and then will resume. For each conversion it spawns new HandBrake CLI process so it needs to track the process ID in order to pause it..

root@caradhras:[/docker/handbrake]: ps -e | grep HandBrake
2895183 ?        00:41:40 HandBrakeCLI



root@caradhras:[/docker/handbrake]: top
top - 15:21:14 up 9 days, 11:06,  8 users,  load average: 32.90, 27.76, 25.69
Tasks: 1517 total,   3 running, 1513 sleeping,   0 stopped,   1 zombie
%Cpu(s):  61.2/9.1    70[||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||                              ]
MiB Mem : 128729.8 total,   8029.3 free,  97235.6 used,  44132.9 buff/cache
MiB Swap:  32764.0 total,  19398.5 free,  13365.5 used.  31494.2 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2895183 macgyver  39  19 1393872   1.2g  19136 S  1594   0.9  42:12.00 HandBrakeCLI
   2765 libvirt+  20   0   38.7g 270972   9516 S 516.7   0.2     2w+1d qemu-system-x86



root@caradhras:[/docker/handbrake]: kill -s STOP 2895183



root@caradhras:[/docker/handbrake]: top
top - 15:23:15 up 9 days, 11:08,  8 users,  load average: 10.27, 21.17, 23.55
Tasks: 1512 total,   2 running, 1508 sleeping,   1 stopped,   1 zombie
%Cpu(s):   8.1/6.5    15[|||||||||||||||                                                                                     ]
MiB Mem : 128729.8 total,   5140.4 free,  99713.5 used,  47036.5 buff/cache
MiB Swap:  32764.0 total,  19399.5 free,  13364.5 used.  29016.3 avail Mem



root@caradhras:[/docker/handbrake]: kill -s CONT 2895183



root@caradhras:[/docker/handbrake]: top
top - 15:23:29 up 9 days, 11:08,  8 users,  load average: 9.70, 20.49, 23.29
Tasks: 1512 total,   3 running, 1508 sleeping,   0 stopped,   1 zombie
%Cpu(s):  64.2/7.4    72[|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||                             ]
MiB Mem : 128729.8 total,   5103.2 free,  99709.8 used,  47077.4 buff/cache
MiB Swap:  32764.0 total,  19399.5 free,  13364.5 used.  29020.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2895183 macgyver  39  19 1393564   1.2g  19136 S  1711   0.9  47:41.78 HandBrakeCLI
   2765 libvirt+  20   0   38.5g 268848   9516 R 350.0   0.2     2w+1d qemu-system-x86
root@caradhras:[/docker/handbrake]:

@aronmgv
Copy link

aronmgv commented May 7, 2024

Trying to think of different approach here.. Would it be feasible to have a single function which will do a check every new minute if we happened to enter no conversion time slot and if so, just pause the thread.. and once out of the window resume it?

@aronmgv
Copy link

aronmgv commented May 13, 2024

Just found out the APP_NICENESS has no effect at all.. Tried many different values with the same performance impact.. However quite feasible solution would be a pause button on the popup here. Also withe the feature of automatic resume (user will input time in minutes/hours or some predefined drop-down values):

) | /opt/base/bin/yad \
--fixed \
--width=384 \
--posx=$(expr $DISPLAY_WIDTH / 2 - 384 / 2) \
--posy=5 \
--title "Automatic Video Converter" \
--window-icon /opt/novnc/images/icons/master_icon.png \
--borders 10 \
--text "Encoding $video..." \
--no-buttons \
--progress \
2>/dev/null

@jlesage what do you think? Can you comment on different approach for the auto-convert schedule as well? Thanks!

@jlesage
Copy link
Owner

jlesage commented May 25, 2024

Just found out the APP_NICENESS has no effect at all.

Can you try to run ps ax -o pid,comm,nice inside the container. This shows the nice value of each running processes.

However quite feasible solution would be a pause button on the popup here. Also withe the feature of automatic resume (user will input time in minutes/hours or some predefined drop-down values):

This popup window is there only to report status of the conversion. In its current form, there is no way to add user interactions.

Trying to think of different approach here.. Would it be feasible to have a single function which will do a check every new minute if we happened to enter no conversion time slot and if so, just pause the thread.. and once out of the window resume it?

I guess this could be a possibility. We have to be careful with the provided feedback, otherwise the conversion would just appear to hang.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature request] Automated Conversion triggered only within specified time range
3 participants