Prioritize archiving to the most full disk

[Proulx-S]

I am using the local archiving functionality of plotman. I would like to the plotman archive to pick the disk that is closest to be full.

That would not be disk_space_script: right? I feel it would be transfer_script:, but I don't understand how "${destination}" is selected.

Any idea anyone?

And in case anyone wonders why:
the idea is to not evenly fill my disks, but instead fill one as fast as possible so I can move it away. This would be useful for replotting hpool plots to portable ones. One could begin with two drives in the site_root: archiving location for plotman to fill one then fallback to the other when full. Then one could add disks one by one and maximize farming the old hpool plots and new portable plots while keeping hpool hardware separated from the rest (better safe then sorry).

Edit-------------------------
Mmm, I just noticed the plotman-transfer-script that plotman interactive produces when it starts archiving. It says:

#!/bin/bash
set -evx
full_destination=$(realpath --canonicalize-missing "${destination}/${path_suffix}")
"${command}" ${options} "${source}" "${full_destination}/"

Where does path_suffix come from?
Out of disk_space_script: somehow?

[altendky]

If I remember correctly the disk space script returns the list of drives with available space and plotman filters for those with free space and then uses the archiving: index: configuration value (defaults to 0) to select from that list. So it doesn't pick "the fullest" but it does fill one first and then move on. Have you observed other behavior? I think you could tweak the disk space script to order however you want if you want something else.

Note that you can create your own target definitions in your config file so that you don't have to "edit the source" and deal with upgrades overwriting your custom script. If you need one...

The path_suffix comes from archiving: env: path_suffix:. I use it to specify a subdirectory to archive into to distinguish pool plots for example. I have a site_root of /farm/sites with disks like 001 and 002 and a path_suffix of pool.xch1234abcd for final targets of /farm/sites/002/pool.xch1234abcd/whatever.plot.

https://github.com/altendky/farm/blob/6cfb12d5a3530b0ff844db93dcab9871834c44bc/config.server.yaml#L60-L66

archiving:
   target: local_rsync
   index: 0
   env:
     # me
     site_root: /farm/sites
     path_suffix: pool.xch1ph3flzr053eswzjc72gjvxjes6fcdamvl9kq5yqw6dnmwtpmlzkq8lzjmn

In summary, I think that plotman comes close to what you want in that it focuses on filling "the first drive" before moving on. Though yes, that isn't exactly "fill the fullest drive". Let me know what questions remain after this explanation.

[Proulx-S]

I actually got confused between the behavior of plotman plot and plotman archive, where the first directs plots evenly across destination drives (though not in an adaptive way).

plotman archive have so far been selecting the fullest drive, but given your answer, it was probably pure fluke.

Thanks for your tip! I'll have to post-pone trying that out for later.

[vladaepro]

I solved it by adding sort to target_definitions, like this:
df -aBK | grep " ${site_root_stripped}/" | awk '{ gsub(/K$/,"",$4); printf "%s:%.0f\n", $6, $4*1024 }'| sort -t: -n -k2

However, you also need to remove sorted function from archive.py so it left available[index] only

[Proulx-S]

That seems like a nice fix! Would it be too much to ask to point out more precisely the modifications you made to archive.py, e.g. the specific line to comment out... ?

Thanks!

[altendky]

If the built in target definitions were updated to do the sorting as the Python presently does, it could be removed from the Python. Then plotman wouldn't need to be modified to sort by available space instead and it could be achieved just with a custom target definition in the user's own configuration file.

[Proulx-S]

That sounds like a nice way to use yaml power :-)

[vladaepro]

I'm not 100% sure sure, but i think target definitions change wasn't sufficient as there were sorting performed in archive.py - so instead of sorted(available)[index] i just left available[index]