Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow Randomization of Archive Drive Destinations #856

Open
MikeKroell opened this issue Jul 12, 2021 · 3 comments
Open

Allow Randomization of Archive Drive Destinations #856

MikeKroell opened this issue Jul 12, 2021 · 3 comments
Labels
enhancement New feature or request

Comments

@MikeKroell
Copy link

When using multiple Plotters, Plotman will constantly use the same archive drive across all plotters resulting in a dramatic slowdown of transfers and writes due to non-sequential writes to the drive over sequential. This happens even with just two plotters.

Now I am not a coder, but here is how I modified the code to make this work for me. Note I did remove some of the checks of the config for this and not sure exactly what other effects it has but it is working for me:

archive.py

    if len(available) > 0:
        index = random.randrange(len(available) - 1)
        #Original Code:
        #index = min(arch_cfg.index, len(available) - 1)
        (archdir, freespace) = sorted(available)[index]
        print(f'Selected {archdir} for archive destination')

This ensures that the majority of the time plotters will be using separate drives. The more drives with available space, the more diverse. They will occasionally hit the same drive, but unless we write some type of temp files for Plotman to read as a marker, I'm not sure how to keep multiple plotters aware of other's writes vs a write that might have not completed.

@MikeKroell MikeKroell added the enhancement New feature or request label Jul 12, 2021
@altendky
Copy link
Collaborator

Why not just use the index feature as is and specify a different number for each plotter? Though there is a change related to that in #855 based on discussion in #847. Also, in cases of "extreme" archiving needs you may be interested in https://github.com/rjsears/chia_plot_manager.

@MikeKroell
Copy link
Author

I must have missed the index feature, where was that?

@altendky
Copy link
Collaborator

Hidden somewhere in the code probably... yeah, our documentation needs work, sorry.

archiving:
  index: 0

Just set it to a different number on each plotter, counting up from 0. This doesn't cover all cases and might become fancier as you suggested by checking for rsync tmp files or other, but it does do a useful thing. When you get down to fewer drives than you have plotters then yeah, there's hazard of collisions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants