Speed up sequential plotting by separating the plotting process and transfer of completed plot

[atlasstrategic]

Plotting with network-attached storage as destination

Consider the case where one creates sequential plots with the destination folder -d set to a NAS (Network Attached Storage) directory. For example, suppose we have a plotting process for two consecutive plots initiated with: chia plots create -k 32 -n 2 -t /media/user/SSD -d /home/user/mnt/my_shared_nas_volume

1. Process starts plotting, phase 1 to phase 4.
2. On completion of phase 4, the completed plot is transferred from /media/user/SSD (local drive) to /home/user/mnt/my_shared_volume (network attached drive via NFS or CIFS).
3. Once the transfer is complete, only then can the second process start with the second plot.

The file transfer will "block" the creation of the second plot (step 2 above) — with WiFi connection this can take well over 3.5 hours to complete!

Potential time saving

1. Manual method: Select a local folder for -d. When the plot is completed, it can then be transferred / copied to the NAS while the second plot has started.
2. Potential new feature? Allow for an Async process to do the transfer in parallel, so that the second plot may be started directly after the first plot has been completed.

👋 I might have missed something obvious, so please correct me if there is a better way to handle such a situation.

[loppefaaret]

you are not wrong... but you could potentially cut it a bit down, if you add the -2 switch into the command, so that it is chia plots create -k 32 -n 2 -t /media/user/SSD -2 /home/user/mnt/my_shared_nas_volume -d /home/user/mnt/my_shared_nas_volume
that way, after the first table in phase 3, rather than building the resulting file in the -t directory, it will start sending it over WIFI to the NAS, at around 15G~ a piece... it might cause a small delay in phase 4, but you wont have the 3 hours waiting for a 100G transfer, while its doing nothing else

[xorinox]

There is still a significant amount of IO to -2, I don't recommend using a NAS for -2, one should do some testing but
faster
-t /media/user/SSD -2 /media/user/SSD -d /home/user/mnt/my_shared_nas_volume
slower
-t /media/user/SSD -2 /home/user/mnt/my_shared_nas_volume -d /home/user/mnt/my_shared_nas_volume

I expect you loose more time using as NAS target for -2 versus local storage but then wait at the end for the transfer to happen. If you were to run several concurrent plotting processes and use the NAS for -2 target, you likely are going to overload the NAS. I assume the NAS is some kind of home use NAS.. If you NAS is supported by dozens of spindles or all flash based, I may change my statement.

[xorinox]

I recently proposed to #dev to wrap the final step (rename and where it matters most copy) into a thread, so plotting of a new file is not delayed.

[4thFourthHorseman]

I have a general question regarding the plotting. Is the data being transfered sequential or random?

[loppefaaret]

you really should come by the official keybase chat
it is mostly sequential, but both read and write requests are coming fast