fix: claimed in staking info endpoint

[Imod7]

Description

Closes #1433

This PR restores the field claimed (under staking) in the staking-info endpoint's response which was "broken". The root cause of this is explained here.

Suggested Fix

The suggested solution includes :

• Adding an IStakingLedger interface to have more control on the response fields under staking.
• Retrieving the claimed eras from :
  • 1. stakingLedger.legacyClaimedRewards OR stakingLedger.claimedRewards (depending on what call is available every time)
  • 2. AND staking.claimedRewards (if available)

For the Reviewer

Please refer to this hackMD doc for more information and a step-by-step on how to check, test and review this PR.

Todos

• Implement new logic based on comment
• Update the description of this PR
• Add a test that includes the era when the migration to the new logic happened
• Add a test that includes also partially claimed eras & erasStakersOverview = null
• Update docs again based on the new implementation
• Should return a total amount of eras (and their status) that are equal or less than depth.
• Merge master branch & update all historical tests (with new response type)
• Handle the case in early blocks/eras where there is lastReward instead of claimedRewards under stakingLedger.
• 20240805 : Update logic again of claimed field for validator account
• 20240805 : Update logic again of claimed field for nominator account
• Add the case of era < 518 and chain = isKusama -> the era is considered as claimed "automatically".
• Add case of status = undefined -> which is when we do not have enough info to determine the status of the era.
• Add the nominator logic mentioned in this comment
• Update docs
• Add IMPLEMENTATION_DETAILS guide
• Add reviewer's HackMD

Nice to have

• A test for the case staking.erasStakersOverview = null & erasStakers = 0 which can be implemented by mocking this request http://127.0.0.1:8080/accounts/CmjHFdR59QAZMuyjDF5Sn4mwTgGbKMH2cErUFuf6UT51UwS/staking-info?at=22869643

Changelog

This PR introduces breaking changes to the staking-info endpoint response by :

1. Removing the returned field lastReward for early eras and adding claimedRewards updated accordingly.
2. Restoring the claimedRewards field name after the breaking change of renaming it to legacyClaimedRewards
3. Changing the structure/type of the returned claimedRewards field (under staking) : before we had an array of eras that were claimed, now we have an object with the list of eras and their corresponding status
4. Refactoring the whole logic that returns the claimedRewards
5. Adding the possibility to query nominator's claimed Rewards apart the validator's.

These changes should be explicitly mentioned in the corresponding Changelog and Release Notes of the release in which this PR is included.

[IkerAlus]

The change is reasonable. Sidecar user does not care of internal renaming changes since the retrieved info has the same meaning.

Is this PR ready for review or are we waiting for the Todos list to be checked?

[Imod7]

The change is reasonable. Sidecar user does not care of internal renaming changes since the retrieved info has the same meaning.

Is this PR ready for review or are we waiting for the Todos list to be checked?

Yes, ideally the todos needs to be completed before a review. However, if any review will speed things up then please feel free to add them.

[Imod7]

Update
After getting feedback from @Ank4n, for the 2nd check mentioned in the logic section above, I concluded that the logic and thus the implementation in this PR should change to the following:

• If staking.erasStakersOverview.pageCount == query.staking.claimedRewards -> era claimed
• If staking.erasStakersOverview.pageCount != query.staking.claimedRewards. length -> era partially claimed
• If overview == null && erasStakers > 0 -> pageCount = 1 : so then it depends on query.staking.claimedRewards value to see if era claimed or unclaimed

The response will also needs to be changed from a Vec of eras to a Vec of objects of type {era: eraNumber, status : claimed | unclaimed | partially claimed}

Related Resource
polkadot-js/api#5859 (comment)

[marshacb]

LGTM! Just had a question about unwraps

[IkerAlus]

Where/how can the user find the relevant linked info?

[Imod7]

Added specific links of the PR and linked issue. Let me know if this covers your question. I also updated the schema description since I no longer return lastReward. Changes included in this commit.

[IkerAlus]

The response time of the endpoint is significantly slower now

Do we have specific data for this statement? How long does it take to return the staking info in the any worst case scenario (for example, 84 eras in different pages and/or with different logic)?

[Imod7]

The response time of the endpoint is significantly slower now

Do we have specific data for this statement? How long does it take to return the staking info in the any worst case scenario (for example, 84 eras in different pages and/or with different logic)?

You are right, I didn't have any formal data when I wrote that. It was based on my observations from the tests I did and also on my knowledge of the calculations of each implementation.
After adding some simple code to measure the execution time of fetchAccountStakingInfo, it looks like there are cases with similar runtimes (old code vs new code) and others have significant differences. However, these results can be influenced by factors such as the connection and the machine used. I am including below some results but to back this statement I need to run proper benchmarks.

• http://127.0.0.1:8080/accounts/GpyTMuLmG3ADWRxhZpHQh5rqMgNpFoNUyxA1DJAXfvsQ2Ly/staking-info?at=16869643 - kusama

  • old code : 608 ms
  • new code : 1.612 ms

• http://127.0.0.1:8080/accounts/CmjHFdR59QAZMuyjDF5Sn4mwTgGbKMH2cErUFuf6UT51UwS/staking-info?at=22869643 - kusama

  • old code : 372 ms
  • new code : 5.312 ms [this is an edge case- goes super slow]

• http://127.0.0.1:8080/accounts/11VR4pF6c7kfBhfmuwwjWY3FodeYBKWx7ix2rsRCU2q6hqJ/staking-info?at=18157800 - polkadot

  • old code : 433 ms
  • new code : 408 ms

[Imod7]

@IkerAlus After our last convo on whether the status partially claimed is valid or not and some more feedback from the staking master @Ank4n :

For a validator to get all commission, they should claim reward for all pages of an era.

Here is the next round of conclusions about claimed:

• When a user queries the staking-info endpoint with a validator address and at a specific era&block
  • The possible status is still : claimed, partially claimed or unclaimed
• When a user queries a nominator address and at a specific era&block
  • @IkerAlus you were right, the possible status can only be : claimed or unclaimed

Currently, I am also considering that in both scenarios, it is useful to add the undefined status which is the case when we only have lastReward available (early eras). In such cases, we cannot determine if the queried era was claimed unless it matches the era noted in lastReward. If the queried era does not correspond to the lastReward era then claimed should be set as undefined.