Plots not moving files to temp dir

[thecodeassassin]

Hey Guys,

For some reason my plots aren't being moved to the destination dir prior to being archived. Did something change in the configuration that prevents this from happening?

I am using the madmax plotter.

This is my config:
https://gist.github.com/thecodeassassin/11012ed49fe4143f0e149160eca001fa

[altendky]

Hum... Are the plots being archived? Where are the plot files being left? Can you run ps aux | grep chia_plot to see what the plotting command line is?

I don't see anything obviously wrong with the config. I personally leave the plots on the tmp disks and archive straight from there.

[thecodeassassin]

The plots are still on the NVMe disks. The only way I can get it to work is by running plotman interactive.

(venv) stephen@test-vm4:~/chia-blockchain$ ls -lha /mnt/nvme2/*.plot
-rw-r--r-- 1 xxx xxx102G Jul  9 20:49 /mnt/nvme2/plot-k32-2021-07-09-14-02-c6c2c265a7c822b11a3faeed1d39f0f4e9d3290bf6693a12cd1f5b90b0f991d2.plot
-rw-r--r-- 1 xxx xxx102G Jul  9 21:12 /mnt/nvme2/plot-k32-2021-07-09-14-57-2fde7550686677942a192d74fe9edb7987b701a03add8bc7940b7866d933831a.plot
-rw-r--r-- 1 xxx xxx102G Jul  9 21:50 /mnt/nvme2/plot-k32-2021-07-09-19-54-0086a45b433d5d94ecb957cec3a33ca5f35d0a2f37f01bcc24d1d247898334a1.plot

the output:

chia_plot -n 1 -r 64 -u 256 -t /mnt/nvme1/ -d /mnt/temp/ -f xxx-c xxxx

I was trying to run it in the background as such:

nohup plotman plot >> ~/chia/plotman.log 2>&1 &
nohup plotman archive >> ~/chia/plotman_archive.log 2>&1 &

Is this the correct way ?

So that I can turn my PC off because I don't need to keep an SSH window open.

Is disabling the "dst" section enough to just move them off the nvme directly to the miner?

[altendky]

So, interactive works but plot doesn't? That's a lead at least. The only comments about your commands would be that you can &> in bash (I would guess you can &>> as well but I can't say I've done it), but that's certainly not relevant to your issue.

The placing of the file on the dst directory is done by the plotter, not plotman. plotman's involvement in the process is to pass the -d parameter to the plotter. Could you check ps aux | grep chia_plot in both the working and not working cases and share that for comparison? And I'll go look and see if I can find any code difference between the two subcommands.

This is now making me think about a few funny things that happened when I was starting to play with madMAx. But... nothing I can remember well enough to deduce anything useful from.

And yes, you can just comment out the dst entirely and it'll use tmp2 if specified otherwise just tmp. Generally people have enough space on their tmp that they can afford to have 100GB sitting around until it's archived.

[thecodeassassin]

At some point the plotting just stops, i.e. it just creates whatever plots it needed to create and just halts. It's supposed to run in a loop right? But i've only gotten that to work correctly with plotman interactive somehow.

Also, it seems to spawn the maxmax process wrongly:

stephen 30266 3094 2.5 21997308 10088692 ? SNLsl 14:02 60:51 chia_plot -n 1 -r 64 -u 256 -t /mnt/nvme1/ -d /mnt/temp/ -f xxx -c xxx

it launches with -r 64 but I configured it to use 32 threads:

        madmax:
                # madMAx plotter: https://github.com/madMAx43v3r/chia-plotter
                n_threads: 32         # Default is 4, crank up if you have many cores
                n_buckets: 256        # Default is 256

[altendky]

Yes, plotman just keeps on creating plots limited by the configured restrictions. A quick look at the threads doesn't show any obvious place we would change it. Did you save the config and then restart plotman and then make sure to only look at newly launched processes?

[thecodeassassin]

stephen  16468  0.9  0.0  93632 22572 ?        Ss   Jul09  54:35 /home/xxx/chia-blockchain/venv/bin/python /home/xxx/chia-blockchain/venv/bin/plotman plot
stephen  24990  0.4  0.0  93228 28036 ?        S    Jul12   3:35 /home/xxx/chia-blockchain/venv/bin/python /home/xxx/chia-blockchain/venv/bin/plotman archive

yes I restarted plotman many times

Problems I am facing:

1. Plotman archive doesn't move the files for some reason even though there are plots ready to move
2. Plotman plot somehow has active jobs but they aren't doing anything
3. The plotman config is not respected it seems

 plot id   plotter    k           tmp          dst    wall   phase    tmp     pid   stat      mem    user    sys   io
e0f487b4    madmax   32   /mnt/nvme6/   /mnt/temp/   21:17     4:1   109G   11256    SLP   14.0Gi   16:08   2:05   0s
938b8436    madmax   32   /mnt/nvme5/   /mnt/temp/   21:35     4:1   109G    7201    SLP   17.5Gi   15:26   1:52   0s
e7b69488    madmax   32   /mnt/nvme4/   /mnt/temp/   22:00     4:1   109G    3112    SLP   16.7Gi   15:05   1:47   0s
17891c5c    madmax   32   /mnt/nvme3/   /mnt/temp/   22:06     4:1   109G    1496    SLP   20.8Gi   14:33   1:46   0s
f2efb115    madmax   32   /mnt/nvme2/   /mnt/temp/   22:42     4:1   109G   44625    SLP   16.9Gi   15:19   1:46   0s
c89b7ea4    madmax   32   /mnt/nvme1/   /mnt/temp/   23:11     4:1   109G   40472    SLP   20.5Gi   15:05   1:59   0s
ab2a70be    madmax   32   /mnt/nvme6/   /mnt/temp/   23:16     4:1   109G   39588    SLP   16.0Gi   14:43   1:46   0s
9d0504a5    madmax   32   /mnt/nvme5/   /mnt/temp/   23:21     4:1   109G   38521    SLP   15.9Gi   15:00   1:59   1s
77c964e8    madmax   32   /mnt/nvme4/   /mnt/temp/   23:26     4:1   109G   37848    SLP   19.7Gi   14:24   1:48   0s
6cbb0934    madmax   32   /mnt/nvme3/   /mnt/temp/   23:31     4:1   109G   36788    SLP   15.1Gi   15:01   1:46   0s
6ffc26ca    madmax   32   /mnt/nvme2/   /mnt/temp/   23:36     4:1   109G   36022    SLP   17.7Gi   14:32   1:46   0s
4454863b    madmax   32   /mnt/nvme1/   /mnt/temp/   23:41     4:1   109G   34841    SLP   16.8Gi   14:17   1:40   1s
b9c2f658    madmax   32   /mnt/nvme6/   /mnt/temp/   23:46     4:1   109G   34287    SLP   14.0Gi   14:44   1:45   0s
59f8d77d    madmax   32   /mnt/nvme5/   /mnt/temp/   23:52     4:1   109G   33308    SLP   19.8Gi   14:29   1:49   0s
f3df8acf    madmax   32   /mnt/nvme4/   /mnt/temp/   23:57     4:1   109G   32544    SLP   16.2Gi   14:10   1:40   0s
3ba060c7    madmax   32   /mnt/nvme3/   /mnt/temp/   24:02     4:1   109G   31988    SLP   17.8Gi   15:04   1:44   1s
2b5170b7    madmax   32   /mnt/nvme2/   /mnt/temp/   24:07     4:1   109G   30998    SLP   17.3Gi   15:00   1:56   1s
b9225962    madmax   32   /mnt/nvme1/   /mnt/temp/   24:12     4:1   109G   30266    SLP   14.6Gi   14:09   1:52   0s

Total jobs: 18
Jobs in /mnt/nvme1/: 3
Jobs in /mnt/nvme2/: 3
Jobs in /mnt/nvme3/: 3
Jobs in /mnt/nvme4/: 3
Jobs in /mnt/nvme5/: 3
Jobs in /mnt/nvme6/: 3

BTW shouldn't plotman plot also trigger archiving ?

[altendky]

Hmm, you have configured archival to look in /mnt for subdirectories that are mounted drives and to archive to them. The things it will find include all your /mnt/nvme* (tmp) and your /mnt/temp (dst). All of those will be considered places to archive to.

edit: err, maybe read too quickly. above is probably wrong since presumably your rsyncd is host is another system. let me try again, sorry...

[altendky]

Ok, I guess the first error in terms of the flow is the number of threads getting passed wrong. Let's reset and look at just that. Share the prompt, command, and output of each step.

• Make sure you have all plotman processes terminated then ps aux | grep plotman.
• Make sure all madMAx processes are stopped then ps aux | grep chia_plot.
• cat ~/.config/plotman/plotman.yaml and copy it here (you can use ```yaml to start the block and it will highlight).
• plotman plot and let it start a new plotting process then ctrl+c.
• ps aux | grep -E '(chia_plot|plotman)'.
• Share the log file for that plot from /home/xxx/chia/plots.log/<timestamp>.plot.log (no need to wait for it to finish, just long enough that we can see the number of threads reported in the log as well).