Auto replace plots older than a given date

[zorner]

This is a duplicate discussion from Machinaris repo.

All my plots were OG(solo) plots. I have created some custom archive scripts to replace the OG plots with NFT(portable) plots.

This is still a work in progress.

# See: https://github.com/guydavis/machinaris/wiki/Plotman#archiving
archiving:
        #target: rsyncd
        #target: local_rsync
        target: local_rsync_plot_replacement
        #target: rsyncd_plot_replacement
        env:
                site_root: /plots
                #user: cryptnoid
                #host: 192.168.1.95
                #rsync_port: 12000
                #site: .
                newest_OG_date: "2021-07-08"    # 2021-07-08 - 1st release (1.2.0) date with support for NFT plots.
                                                # Your newest OG date will likely be later. Adjust to your newest OG plot + 1 day.
                                                # This script will replace any plots before newest_OG_date by modification date.
                                                # A plot's modification date will likely be the date the plot was copied to its current location.
                                                # OG and NFT plots look the same to the script.
                                                # Comment out to prevent the relpacing any plots.
                max_plot_size: 106370000        # Appoximately the largest 32k plot size in KB. Used to guess if there room for the plot.
                                                # When rsync runs, it will use the actual plot size.
                                                # This script only deletes plot if the drive looks to be full.
        target_definitions:
                local_rsync_plot_replacement:
                        env:
                                # A value of null indicates a mandatory option
                                command: rsync
                                #options: --preallocate --remove-source-files --skip-compress plot --whole-file # --preallocate does not work inside docker containers.
                                options: --remove-source-files --skip-compress plot --whole-file
                                site_root: null
                                site_path_filter: ""
                                path_suffix: ""
                                newest_OG_date: ""
                                max_plot_size: 106370000
                        
                        # The disk space script must return a line for each directory
                        # to consider archiving to with the following form.
                        #
                        #   /some/path:1000000000000
                        #
                        # That line tells plotman that it should consider archiving
                        # plots to files at paths such as /some/path/theplotid.plot and
                        # that there is 1TB of space available for use in that
                        # directory.
                        disk_space_script: |
                                #!/bin/bash
                                #set -evx
                                site_root_stripped=$(echo "${site_root}" | sed 's;/\+$;;')
                                if [ ! -z "${newest_OG_date}" ]; then
                                    unset -v has_space
                                    has_space=$( df -aBK ${site_root_stripped}/* | grep "${site_path_filter}" | awk -v plot_size=${max_plot_size} '{ gsub(/K$/,"",$4); if (plot_size < int($4)) print "true" }' )
                                    if [ -z "$has_space" ]; then
                                        old_file=$( find -L ${site_root_stripped}/${site_path_filter} -type f -name "*.plot"| sort | head -1 )
                                        if (( $(date -r "$old_file" +%s) < $(date -d "${newest_OG_date}" +%s) )); then
                                            rm "$old_file"
                                        fi
                                    fi
                                fi
                                # printf with %.0f used to handle mawk such as in Ubuntu Docker images
                                # otherwise it saturates and you get saturated sizes like 2147483647
                                df -aBK ${site_root_stripped}/* | grep "${site_path_filter}" | awk '{ gsub(/K$/,"",$4); printf "%s:%.0f\n", $6, $4*1024 }'
                        transfer_script: |
                                #!/bin/bash
                                #set -evx
                                start=$(date +%s.%N)
                                
                                function exit_with_error {
                                duration=$(echo "$(date +%s.%N) - $start" | bc)
                                echo "Script Execution Time: $( date -u --date=@${duration} +%X )"
                                exit 1
                                }
                                
                                echo $(date -Iseconds): Launching transfer activit
                                full_destination=$(realpath --canonicalize-missing "${destination}/${path_suffix}")
                                echo $(date -Iseconds): Transfering ${source}
                                echo $(date -Iseconds): Transfering to ${full_destination}
                                "${command}" ${options} "${source}" "${full_destination}/"
                                duration=$(echo "$(date +%s.%N) - $start" | bc)
                                echo $(date -Iseconds): "Transfer Time: $( date -u --date=@${duration} +%X )"
                        transfer_process_name: "{command}"
                        transfer_process_argument_prefix: "{site_root}"
                rsyncd_plot_replacement:
                        env:
                                # A value of null indicates a mandatory option
                                command: rsync
                                #options: --bwlimit=80000 --preallocate --remove-source-files --skip-compress plot --whole-file # --preallocate does not work inside docker containers.
                                # Options above is missing ssh tunnel to protect the rsync.
                                #options: --rsh=ssh --bwlimit=80000 --remove-source-files --skip-compress plot --whole-file
                                options: --bwlimit=120000 --remove-source-files --skip-compress plot --whole-file
                                rsync_port: 873
                                ssh_port: 22
                                user: null
                                host: null
                                site_root: null
                                site_path_filter: ""
                                site: "."
                                path_suffix: ""
                                newest_OG_date: ""
                                max_plot_size: 106370000
                        disk_space_script: |
                                #!/bin/bash
                                #set -evx
                                site_root_stripped=$(echo "${site_root}" | sed 's;/\+$;;')
                                if [ ! -z "${newest_OG_date}" ]; then
                                    unset -v has_space
                                    has_space=$( ssh -p "${ssh_port}" "${user}@${host}" "df -aBK ${site_root_stripped}/* | grep \"${site_path_filter}\" | awk -v plot_size=${max_plot_size} '{ gsub(/K\$/,\"\",\$4); if (plot_size < int(\$4)) print \"true\" }'" )
                                    if [ -z "$has_space" ]; then
                                        old_file=$( ssh -p "${ssh_port}" "${user}@${host}" "find -L ${site_root_stripped}/${site_path_filter} -type f -name \"*.plot\"| sort | head -1" )
                                        old_size=$(ssh -p "${ssh_port}" "${user}@${host}" "date -r "$old_file" +%s" )
                                        if (( $old_size < $(date -d "${newest_OG_date}" +%s) )); then
                                            ssh -p "${ssh_port}" "${user}@${host}" "rm $old_file"
                                        fi
                                    fi
                                fi
                                # printf with %.0f used to handle mawk such as in Ubuntu Docker images
                                # otherwise it saturates and you get saturated sizes like 2147483647
                                ssh -p "${ssh_port}" "${user}@${host}" "df -aBK ${site_root_stripped}/* | grep \"${site_path_filter}\" | awk '{ gsub(/K\$/,\"\",\$4); printf \"%s:%.0f\n\", \$6, \$4*1024 }'"
                        transfer_script: |
                                #!/bin/bash
                                #set -evx
                                start=$(date +%s.%N)
                                
                                function exit_with_error {
                                duration=$(echo "$(date +%s.%N) - $start" | bc)
                                echo "Script Execution Time: $( date -u --date=@${duration} +%X )"
                                exit 1
                                }
                                
                                echo $(date -Iseconds): Launching transfer activity
                                full_destination=$(realpath --canonicalize-missing "${destination}/${path_suffix}")
                                relative_path=$(realpath --canonicalize-missing --relative-to="${site_root}" "${full_destination}")
                                url_root="${user}@${host}:"
                                echo $(date -Iseconds): Transfering ${source}
                                echo $(date -Iseconds): Transfering to ${url_root}${full_destination}
                                "${command}" ${options} "${source}" "${url_root}${full_destination}/"
                                duration=$(echo "$(date +%s.%N) - $start" | bc)
                                echo $(date -Iseconds): "Transfer Time: $( date -u --date=@${duration} +%X )"
                        transfer_process_name: "{command}"
                        transfer_process_argument_prefix: "{user}@{host}:"
        #  external_script:
        #    env:
        #      some_common_value_with_a_default: /a/path
        #      some_mandatory option: null
        #    disk_space_path: /home/me/my_disk_space_script.sh
        #    transfer_path: /home/me/my_transfer_script.sh
        #    transfer_process_name: rsync
        #    transfer_process_argument_prefix: /the/destination/directory/root

The disk_space_script tries to guess if there is enough free space for another plot. If not, it looks for the oldest plot by modification date. This is often when the plot was moved, not when it was created. If the date is before newest_OG_date, it deletes that plot and then gets the list of dir:free_space as normal.

Some logging has been added to the transfer_script.

local_rsync_plot_replacement -> disk_space_script:

#!/bin/bash
#set -evx
site_root_stripped=$(echo "${site_root}" | sed 's;/\+$;;')
if [ ! -z "${newest_OG_date}" ]; then
    unset -v has_space
    has_space=$( df -aBK ${site_root_stripped}/* | grep "${site_path_filter}" | awk -v plot_size=${max_plot_size} '{ gsub(/K$/,"",$4); if (plot_size < int($4)) print "true" }' )
    if [ -z "$has_space" ]; then
        old_file=$( find -L ${site_root_stripped}/${site_path_filter} -type f -name "*.plot"| sort | head -1 )
        if (( $(date -r "$old_file" +%s) < $(date -d "${newest_OG_date}" +%s) )); then
            rm "$old_file"
        fi
    fi
fi
# printf with %.0f used to handle mawk such as in Ubuntu Docker images
# otherwise it saturates and you get saturated sizes like 2147483647
df -aBK ${site_root_stripped}/* | grep "${site_path_filter}" | awk '{ gsub(/K$/,"",$4); printf "%s:%.0f\n", $6, $4*1024 }'

local_rsync_plot_replacement -> transfer_script:

#!/bin/bash
#set -evx
start=$(date +%s.%N)

function exit_with_error {
    duration=$(echo "$(date +%s.%N) - $start" | bc)
    echo "Script Execution Time: $( date -u --date=@${duration} +%X )"
    exit 1
}

echo $(date -Iseconds): Launching transfer activit
full_destination=$(realpath --canonicalize-missing "${destination}/${path_suffix}")
echo $(date -Iseconds): Transfering ${source}
echo $(date -Iseconds): Transfering to ${full_destination}
"${command}" ${options} "${source}" "${full_destination}/"
duration=$(echo "$(date +%s.%N) - $start" | bc)
echo $(date -Iseconds): "Transfer Time: $( date -u --date=@${duration} +%X )"

rsyncd_plot_replacement is similar, only using ssh to reach the harvester.