0 reward points (SOLVED) [rp-issue form]

[gtmed19]

Describe the problem

I have 8 machines running and they all have the following GPU specs (the systems have 64 GB ram and some 128GB)

They are all running docker through windows ( I used to have them all on Ubunu 22.04) but after 3 months of working perfectly getting the right points , after I did the last lilypad update it stopped getting rewards , moreover I was slashed more than100k points per total which was a big disappointment for all of my effort of keeping them running 24/7 for 3 months straight.
So I was forced to switch to windows, and gave windows docker a try. Now I am using windows docker and at least some are getting some points but some are still getting 0 points (for almost 2 weeks now). At least the slashing is stopped but I'd like to make them all run properly again. Below are the devices with their GPUs and their daily reward points :

1. 1x 4090 GPU - getting around 3k daily points , sometimes 2k randomly with no changes from my side
2. 1x 4090 GPU - getting around 3k daily points , sometimes 2k - same
3. 1x 4090 GPU - getting around 1k daily to 300 points , sometimes 2k , it's oscillating
4. 1x 4090 GPU - 0 points
5. 1x 4090 GPU - around 400 points randomly one day , one day getting 0.. and so on
6. 2x 3090 GPUS - around 1k points
7. 2x 3090 GPUS - 0 points
8. 2x 3090 GPUS - 400 points

So out of the 2x 3090 devices only 1 works properly. I am running same commands to run docker on all except some give me this error below.
For now I don't have them at a stable daily rewards rate like they all used to.

I am using Alchemy rpc for all of them.
Weird that same pc specs , same os and method and some are behaving properly and some not.
Any insights and potential fixes would be greatly appreciated.
In the meantime I just keep erasing docker images (on the ones with the problem) and re-running script daily, changing the RPC with another one and so forth.

The biggest issue that seem to appear on all is "nonce too low"
For the sake of this example I am only giving you one 4090pc with the problem (all the other ones have the same problem)
0xc9AC67468D06a85e033C6170614f7521B646e790
Weirdly enough , it worked then it didn't work then it worked again as you can see the points in the grafana dashboard.

Ensure all steps below are taken and check this box yes when completed

• Yes

RP hardware and software info

Hardware specs:
Lilypad version:

Eth address (Wallet address)

0xc9AC67468D06a85e033C6170614f7521B646e790

RP logs

Paste your logs he70.275102561708762024/11/16 09:00:08 GPU Card Info: NVIDIA GeForce RTX 4090, GPU-e2c13ed1-896c-d913-1f14-a933190acbf8, 00000000:05:00.0, 3135 MHz, 10501 MHz
map[challange:0xabcb38412e94a0cda517d4899663a47d3a555db7951984be2edcfbc58885242f clockmaxmem:10501 clockmaxsm:3135 cudacount:16384 difficulty:505469245430026822523187709738088416614776124004897893818060544598343 executable:3430c473592cd18b481009057f8e4d6d4592015f4f55822edb26952b42f32c65 gpuName:NVIDIA GeForce RTX 4090 gpuUuid: GPU-e2c13ed1-896c-d913-1f14-a933190acbf8 mhash:70.27510070800781 nvidia-smi:0e71c28e991fcae7f491ea654f1cb93f668d22e3f1dcbbd24e41f8c28d85edf7 pow-id:bbac7795-5585-4f1c-b6fd-dd36e0a54e8c signature:0xffaf119c30af8018ace8cf59b70609508bc6a7e859e47c921096fd40c39d06a73ce4800401ef56ed544218de1ef2cdf0749422bf7a914c4fd132023dd2f9686701 tensorcount:512 timestamp:1731747608 version:v4 vram:24563 walletAddress:0xc9AC67468D06a85e033C6170614f7521B646e790]
2024/11/16 09:00:08 Received Created response: 201 Created
API request sent successfully

/tmp/card-16905375442024-11-16T09:00:08Z INF ../../local/go/src/runtime/asm_amd64.s:1695 > Success! Elapsed Time=393.190029ms HashNumber=233551569462162402410208989634223550649858487530840662659783172182388 Nonce=5758541081908174801684062901249458956507107282020083726165403711369181921280 WorkerID=0 challenge=77704555316986025553423046899161785567576073971888870883559839580806124479535
2024-11-16T09:00:11Z INF ../../local/go/src/runtime/asm_amd64.s:1695 > Hash speed: 72 kilohashes/s
2024-11-16T09:00:23Z INF pkg/resourceprovider/minerctl.go:213 > Mine and submit successfully Nonce=5758541081908174801684062901249458956507107282020083726165403711369181921280 address=0xc9AC67468D06a85e033C6170614f7521B646e790 nodeid=QmPMNmXfRbbqA7gb7S3VPRUqWB51aS8LGkijBto1UcwXVe txid=0x3686d5accfd91782424b127085c92d75ef3612d2660b171a23994d1a9015c873
2024/11/16 09:10:31 [DEBUG] GET http://localhost:1234/api/v1/orchestrator/nodes
09:10:31.2 | WRN pkg/publicapi/middleware/version.go:41 > received request from client without version [ClientID:] [ClientVersion:] [NodeID:n-c9403291] [RequestID:] [ServerVersion:1.3.2]
09:10:31.201 | INF pkg/publicapi/middleware/logger.go:50 > [ClientID:] [Duration:3.480462] [JobID:] [Method:GET] [NodeID:n-c9403291] [Referer:] [RemoteAddr:::1] [RequestID:] [Size:2316] [StatusCode:200] [URI:/api/v1/orchestrator/nodes] [UserAgent:Go-http-client/1.1]
2024-11-16T09:15:03Z ERR pkg/resourceprovider/resourceprovider.go:126 > 🔵 RP error submitting pow.GenerateChallenge="nonce too low: address 0xc9AC67468D06a85e033C6170614f7521B646e790, tx: 3063 state: 3064"
2024-11-16T09:15:03Z ERR ../../local/go/src/runtime/asm_amd64.s:1695 > Unable to fetch challenge error="nonce too low: address 0xc9AC67468D06a85e033C6170614f7521B646e790, tx: 3063 state: 3064"
2024/11/16 09:21:00 [DEBUG] GET http://localhost:1234/api/v1/orchestrator/nodes
09:21:00.5 | WRN pkg/publicapi/middleware/version.go:41 > received request from client without version [ClientID:] [ClientVersion:] [NodeID:n-c9403291] [RequestID:] [ServerVersion:1.3.2]
09:21:00.5 | INF pkg/publicapi/middleware/logger.go:50 > [ClientID:] [Duration:1.987757] [JobID:] [Method:GET] [NodeID:n-c9403291] [Referer:] [RemoteAddr:::1] [RequestID:] [Size:2316] [StatusCode:200] [URI:/api/v1/orchestrator/nodes] [UserAgent:Go-http-client/1.1]
2024/11/16 09:31:30 [DEBUG] GET http://localhost:1234/api/v1/orchestrator/nodes
09:31:30.152 | WRN pkg/publicapi/middleware/version.go:41 > received request from client without version [ClientID:] [ClientVersion:] [NodeID:n-c9403291] [RequestID:] [ServerVersion:1.3.2]
09:31:30.152 | INF pkg/publicapi/middleware/logger.go:50 > [ClientID:] [Duration:2.587388] [JobID:] [Method:GET] [NodeID:n-c9403291] [Referer:] [RemoteAddr:::1] [RequestID:] [Size:2316] [StatusCode:200] [URI:/api/v1/orchestrator/nodes] [UserAgent:Go-http-client/1.1]
2024/11/16 09:42:01 [DEBUG] GET http://localhost:1234/api/v1/orchestrator/nodes
09:42:01.468 | WRN pkg/publicapi/middleware/version.go:41 > received request from client without version [ClientID:] [ClientVersion:] [NodeID:n-c9403291] [RequestID:] [ServerVersion:1.3.2]
09:42:01.468 | INF pkg/publicapi/middleware/logger.go:50 > [ClientID:] [Duration:2.418469] [JobID:] [Method:GET] [NodeID:n-c9403291] [Referer:] [RemoteAddr:::1] [RequestID:] [Size:2316] [StatusCode:200] [URI:/api/v1/orchestrator/nodes] [UserAgent:Go-http-client/1.1]
2024/11/16 09:52:31 [DEBUG] GET http://localhost:1234/api/v1/orchestrator/nodes
09:52:31.967 | WRN pkg/publicapi/middleware/version.go:41 > received request from client without version [ClientID:] [ClientVersion:] [NodeID:n-c9403291] [RequestID:] [ServerVersion:1.3.2]
09:52:31.967 | INF pkg/publicapi/middleware/logger.go:50 > [ClientID:] [Duration:2.042661] [JobID:] [Method:GET] [NodeID:n-c9403291] [Referer:] [RemoteAddr:::1] [RequestID:] [Size:2316] [StatusCode:200] [URI:/api/v1/orchestrator/nodes] [UserAgent:Go-http-client/1.1]re...

[gtmed19]

I might've caught the problem. Will keep an eye on them for a few more days and get back here with an update.

[galaxyxtwo]

thanks, let us know any additional context that may help others.

[gtmed19]

I finally solved my issue. All of my 8 nodes are working properly now.

So the problem was that 3 of my machines started to run the nodes in parallel on Ubuntu alongside with docker and they would interfere most likely. (I must've just stopped the lilypad and bacalhau services instead of removing them completely).

So , on windows with Ubuntu 22.04 LTS I had to completely erase the scripts lilypad and bacalhau (not just stop them) before using docker with windows because they would start running again for some unknown reason and I presume it would interfere with docker; however, I still don't have a clear answer why the 3.0 lilypad version stopped working with Ubuntu.

Therefore, for some unknown reason after running for 3 months with Ubuntu 22.04 it stopped working for me after the 3.0 lilypad update so switching to windows with docker solved it for me.

It's a little sad that I lost around 100k points due to slashing after I did the 3.0 update. I checked them daily , but 1 day was enough on all of my 8 nodes, but hey this is the testnet. Hopefully this won't happen again.

Now after switching to docker I would recommend everyone to use docker because it's only 2 commands. It's a gazillion times faster and easier to run the nodes. :D

[galaxyxtwo]

really appreciate the time spent to investigate and detailed notes above. Glad the nodes are running smoothly again.

So the problem was that 3 of my machines started to run the nodes in parallel on Ubuntu alongside with docker and they would interfere most likely.

Very useful to know this case! We have seen issues before where PoW cant run as the RP doesnt have full access to the server resources. This would also cause Lilypad jobs sent to these RPs to fail.

[gtmed19]

I am glad I could finally get the hang of it. Glad I could be of help! Should I encounter any other errors or find other fixes I will post them back here.