Feedback on running a Lilypad RP

[galaxyxtwo]

Hello! 👋

The Lilypad team is thrilled to welcome so many new Resource Providers (RPs) to the network!

This discussion board can be used to provide feedback on the experience of running a Lilypad node (RP). We are also interested to hear feedback on the Lilybit_ rewards system, documentation, and anything else to keep in mind as a RP!

[likhita-8091]

At present, it has been running normally for many days and there are also pow records, but the rewards are getting less and less. At the beginning, it was 10000, then 9000, then 8000, and now it is over 7000. Why is this?

[DeveloperAlly]

Hello please use the #i-need-help channel.

This is specifically a discussion for feedback

[galaxyxtwo]

thanks for your feedback @likhita-8091 . If this is still occuring, please open a ticket in https://discord.com/channels/1212897693450641498/1230231823674642513 with your wallet ID and details about the issue.

[galaxyxtwo]

thanks @wolfcubecho for providing additonal feedback on the Lilypad slashing mechanism-> https://github.com/orgs/Lilypad-Tech/discussions/20

[rhochmayr]

Hey LP Team

Giving some feedback based on the conversation in Discord

• Running a node
  Running a node is generally straightforward, with various options available. Some setups require more troubleshooting, but issues are usually addressed quickly. However, it’s sometimes unclear how widespread certain problems are, which can be frustrating and time-consuming

  For instance, in mid-July, I faced issues due to a network-wide problem that wasn’t communicated, leading to unnecessary trial and error. A quick update in Discord could have saved time. This has gotten better though. It still would be nice to have some indicators of the working state of the various components involved in keeping the network functioning.

  Another thing that took me quite a while initially was making sure the node is actually able to process modules that require a gpu. Although I'm not sure this is even required to collect rewards at the moment?

  The need for regular service restarts also remains a concern, especially when nodes go offline and stop processing PoW signals. While automatic reconnections have improved, some nodes still face challenges. Regular restarts are manageable when you’re available, but they can lead to downtime if unattended for a few days.

  I do wonder how this looks like for the other providers. Maybe some of the issues just stem from the fact that my node runs on Windows + WSL2 and things would be slightly less bumpy if it where to run just on a native ubuntu machine somewhere.

  Changing system though seems hard to quantify without knowing which setups and gpus operate at which level in regards to stability and the pow mining.

• Lilybit_ rewards system
  Transperancy
  Currently we don't have the information to figure out how nodes get to their points exactly. It would be nice if we actually understood the mechanism and can check if things seem off, and why. Having information on the type of gpu for each node would also be awesome.
  Slashing
  I never fully understood the reasoning behind the 10% daily slashing. I absolutely get that you want to attract long term participants, but this seems a bit much. Especially when the slashing occurs due to bugs in the network or provided components. One could have operated their node for 3 months and would loose almost all points if the node goes offline and/or stops processing signal while someone is away for a week.

  Just two days of slashing can easily mean an effective setback of 10-14 days when considering that points also couldn't be earned in that time.

  The later this happens the more disgruntled I could imagine node operators get. They essentially loose all multiplier points from their earlier phases and have to catchup in a phase with a considerably lower multiplier. They are basically punished for having been an early member if anything where to go wrong at a later stage.

  I think there would be multiple ways around those, however it is hard to judge how much effort can go into adapting this without knowing how many of the 800+ nodes actually have issues. Maybe it's just a handful of providers on certain platforms.

  Either way, here are some ideas in regards to the slashing mechanism, although I assume you will have discussed all of those in great lengths at some point already:

  • Grace Period Before Slashing Begins
    A grace period before slashing starts. For example, a node might be allowed a couple of days per month to address any issues without incurring penalties.
  • Tiered/Progressive Slashing Mechanism
    A tiered system where penalties increase progressively the longer a node remains offline. For example:
    1-5 days offline: 2.5% reduction
    5-10 days offline: 5% reduction
    More than 10 days: 10% reduction
  • Slashing Pause for Network-Induced Issues
    A mechanism to pause slashing if a network-wide issue is identified. It’s unfair to penalize the operators for those.
  • Dynamic Slashing Based on Historical Uptime
    Adjust the severity of slashing based on the historical uptime of the node. Nodes with a track record of high availability could face lower penalties for downtime compared to less reliable nodes.

• documentation
  It's a work in progress, and always will be :-) It's great to see that it is constantly updated with new information and feedback you guys are gathering from the community.

• anything else
  Keep up the great work, it is an amazing project!

[galaxyxtwo]

Appreciate this @rhochmayr ! I will communicate feedback with the engineering team and will also discuss internally at DevRel.

[DeveloperAlly]

• Grace Period Before Slashing Begins
  A grace period before slashing starts. For example, a node might be allowed a couple of days per month to address any issues without incurring penalties.
• Tiered/Progressive Slashing Mechanism
  A tiered system where penalties increase progressively the longer a node remains offline. For example:
  1-5 days offline: 2.5% reduction
  5-10 days offline: 5% reduction
  More than 10 days: 10% reduction
• Slashing Pause for Network-Induced Issues
  A mechanism to pause slashing if a network-wide issue is identified. It’s unfair to penalize the operators for those.
• Dynamic Slashing Based on Historical Uptime
  Adjust the severity of slashing based on the historical uptime of the node. Nodes with a track record of high availability could face lower penalties for downtime compared to less reliable nodes.

Hello!

Thank you for the thought and productive feedback on the mechanism.

It's incredibly useful to receive implementable ideas to improve the system and we really appreciate you taking the time to think about them.

We are working on a plan to upgrade the Rewards mechanism this week and will provide additional details on this early next week

Thanks again!

[wolfcubecho]

Ive already spoken in here and discord about slashing but feel the need to re address it as the pain of the last few days has caused no end of frustration. Slashing hurts operators and add un needed stress whilst problem solving. We not only don't make rewards but loose them also.

I had been monitoring my node and arbscan the last few days and seeing transactions, plus rolling logs showing all the right things so i never deep dived into transaction times etc. After all my rewards were increasing and my logs and arbscan showed activity. Imagine my shock when I am down 15k tokens yesterday and realize its been missing some calls...

Now I spent my evening feverishly trying to solve at the detriment of my marriage due to the bloody pressure of the slashing (my own choice but included as its relevant to my case of pointless stress on loyal supporters). I then saw it working and went to sleep. Only to wake up and see it missed a check through the night and now I am down another 5k tokens.

This is unacceptable to punish community like this. I don't expect to earn points but to loose ones I have earnt when I still had 25 calls go through during the day shows clearly, I am active albeit there's a problem and should at very least mean I don't loose my reward points... Surely the simple logic is if a node has checked in at some point of the day the rewards arent slashed then unless the node checks in over 4 consecutive hours successfully it doesnt earn?

I am down 30k tokens now. The first 10 I own, I let my wallet run dry. Since then my wallet is loaded every few days. I restart every day etc. This last 20k tokens is disappointing. It wasn't obviously not working unless I closely checked logs of which I can take fault for, but clearly as an op who doesnt even own the tokens ive earnt I cant spend all day every day watching rolling logs and calculating check in times on arb scan.

Today I have made restart scripts and monitors and TG alerts and perhaps thats what should have been done so as not to be crippled by your systems. I take ownership of my faults in here and openly say ok I should not have earnt tokens over these last two days as there was something causing the drop outs. However I present once again that the slashing needs to be thought about and changed asap. Simple things like this crush small honest operators - the ones you claim you want to support your project.

I would prepose that if there is daily activity from a node however menial then it saves it from slashing. Then have the other metric for actual earn thats fine. I also think the previous points need to be considered and either thoughtfully responded to or swiftly implemented because case and point of my current situation you have hurt an operator who has been active over the whole period of slashing.

Restoration of wrongfully slashed points in light of changes should also be seriously considered. We want to support you and long term but it costs us and currently you show us no trust in return for the trust we show you via the dedication and supply of our service and infrastructure.

Thankyou for considering and hearing these thoughts, frustrations and ideas. I look forward to moving forward and fixing these problems so we can continue as a community long into the future.

Wolf.

[Dingyingshang]

It has been running normally for many days and there are POW records, but the points awarded are getting less and less. Why is this?

[DeveloperAlly]

Hello please use the #i-need-help channel.

This is specifically a discussion for feedback