Add REST endpoint to retrieve historical_summaries #6675

kdeme · 2024-10-23T21:10:27Z

In draft currently as it is not in spec. Needs to be accompanied still with a proposal for beacon REST API specifications.

The idea is that historical_summaries is very useful for verifying if beacon blocks (with their accompanied proofs) are part of the canonical chain, however there is no way to actually retrieve the historical_summaries except by using the /eth/v2/debug/beacon/states/{state_id} endpoint, which is quite the download in terms of size (We are currently using this endpoint in Portal/fluffy for our portal_bridge, but it takes long).

This PR adds an endpoint that provides the historical_summaries with its proof against the state root. This is done so that the historical_summaries can then be verified for example with the finalized state root that a (light) node might hold from its finality update.
Again, this is specifically useful in Portal (as a portal_bridge can inject it in the p2p network as is, and receiving nodes can verify it if they are light client synced) . But you could make an argument that it could also be useful on a consensus light client (assuming blocks with proofs would be somehow available).

edit: moved to /nimbus/v1/debug/ namespace and removed draft

kdeme · 2024-10-23T21:12:59Z

beacon_chain/rpc/rest_debug_api.nim

+            RestApiResponse.jsonResponseFinalized(
+              response, node.getStateOptimistic(state), node.dag.isFinalized(bslot.bid)
+            )
+          elif contentType == sszMediaType:


I opted for JSON and SSZ, as that is what the debug state endpoint does and also because I was using the SSZ version already in our portal_bridge.

I'm not sure what the convention is, or if there is a convention, of when to opt also for SSZ. I've seen quite some big responses that do not seem to offer SSZ as an option in the beacon API.

preferredContentType function provides interface to choose what client want to get. Client could use different versions of this construction:

Accept: application/json

Accept: application/octet-stream (SSZ)

Accept: application/json;q=0.5, application/octet-stream;q=0.9

Accept: application/json;q=0.5, application/octet-stream

and so on...

when using preferredContentStream server side portion could which order is preferred for the server. And preferredContentStream() establish consensus and return you agreed content type.

kdeme · 2024-10-23T21:14:18Z

beacon_chain/rpc/rest_debug_api.nim

+            )
+          elif contentType == sszMediaType:
+            let
+              headers = [("eth-consensus-version", consensusFork.toString())]


I added this as it seems to be added always (?), even though I can't think of any use for it in the specific case of historical_summaries. (Perhaps if the BeaconState amount of fields grows > 32 it could be useful for the proof)

kdeme · 2024-10-23T21:15:49Z

beacon_chain/rpc/rest_debug_api.nim

+
+  router.metricsApi2(
+    MethodGet,
+    "/eth/v1/debug/beacon/states/{state_id}/historical_summaries",


Added it under the debug namespace for now.

Although I realize that debug is not actually purely a list of debug endpoints as the state one is used by users for trusted node sync.

debug section is part of official specification, you should not add endpoints to this section if its not yet part of official specification, we have nimbus section for such calls.

Yes, this is why the PR is in draft for now. I want to hopefully get it added in official specification.

I was not aware about our specific nimbus prefixed API. I'd be happy to move it there already if that would mean earlier merging in nimbus :)

github-actions · 2024-10-23T22:50:17Z

Unit Test Results

      12 files ±0   1 814 suites ±0 54m 41s ⏱️ - 1m 1s
  5 232 tests ±0   4 884 ✔️ ±0 348 💤 ±0 0 ❌ ±0
29 077 runs ±0 28 625 ✔️ ±0 452 💤 ±0 0 ❌ ±0

Results for commit 566e69b. ± Comparison against base commit e06af18.

♻️ This comment has been updated with latest results.

cheatfate · 2024-10-24T01:26:20Z

beacon_chain/spec/eth2_apis/rest_debug_calls.nim

+
+proc getHistoricalSummariesV1*(
+    client: RestClientRef, state_id: StateIdent, cfg: RuntimeConfig, restAccept = ""
+): Future[Option[GetHistoricalSummariesV1Response]] {.async.} =


We are not using simple {.async.} anymore, it should use new syntax with asyncraises.
By default PRESTO annotates client functions which returns RestPlainResponse with
[CancelledError, RestEncodingError, RestDnsResolveError, RestCommunicationError] as you can see here
https://github.com/status-im/nim-presto/blob/master/presto/client.nim#L489-L509

Of course you can use [RestError, CancelledError] because RestError is common ancestor for all exceptions generated by PRESTO, but its not recommended.

In your case it should also include RestResponseError which you generating manually.

Right, I noticed the {.async.} still being used in the debug calls and wasn't sure if everything was ready to use asyncraises. But looking at the other APIs now I see it has been using asyncraises everywhere.
I willl adjust to asyncraises.

All the calls which are using rest pragma annotation already include asyncraises (but its hidden). All other wrappers will be converted soon.

beacon_chain/spec/eth2_apis/rest_debug_calls.nim

tersec · 2024-10-24T10:46:44Z

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

kdeme · 2024-10-24T12:24:35Z

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

cheatfate · 2024-10-25T06:42:28Z

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

tersec · 2024-10-25T06:46:25Z

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

cheatfate · 2024-10-25T08:06:03Z

One consideration for me is that if this does get standardized, we don't end up with both a pre-standardized and standardized version which needs to be supported.

One approach could be that I move it under the /nimbus/v1/debug/ prefix for now, assuming that /nimbus/v1/debug/ would mean custom and subject to removal without notice (like for the debug cli flags).

Why you need to remove it? You can establish redirect() call from one endpoint to another when it will be part of specification, we was doing this many times.

Well, part of the point is not to end up with extra, redundant endpoints. The point of this subject to removal without notice would be to exactly allow experimentation without obligation such directions later.

There could be exactly 2 situations:

Change will be proposed, but declined.
Change will be proposed and accepted.

In both this cases it could be better to start with /nimbus/ endpoint. Because /nimbus/ endpoint could not be regulated by EF, so we could add/remove any functions. Otherwise in case 1 you will be forced to remove it from /debug/.

cheatfate · 2024-10-30T12:36:18Z

beacon_chain/rpc/rest_nimbus_api.nim

+      return withState(state):
+        when consensusFork >= ConsensusFork.Capella:
+          # Build the proof for historical_summaries field (28th field in BeaconState)
+          let gIndex = GeneralizedIndex(59) # 31 + 28 = 59


I think this index is heavily depends on current version of BeaconState object, which means that if BeaconState object will change in next fork this index should be modified or at least visible for new fork maintainers.
So we should add something like

static: doAssert high(ConsensusFork) == ConsensusFork.Electra

here.

Yep, and also, >= Capella is wrong because Electra has different gindex than Deneb.

Right, I see in Electra the amount of fields grows > 32. Basically what I write here (#6675 (comment)) is already happening.

I will adjust index for different forks and assert for unsupported/unknown forks.

kdeme · 2024-10-30T16:18:45Z

beacon_chain/spec/eth2_apis/rest_types.nim

@@ -337,6 +337,11 @@ type
  RestEpochRandao* = object
    randao*: Eth2Digest

+  RestHistoricalSummaries* = object
+    historical_summaries*: seq[HistoricalSummary]
+    proof*: array[5, Eth2Digest]


Comment #6675 (comment) means also that this will become array[6, Eth2Digest] for Electra. For the JSON part that is not really an issue. In Nim this means I will need to either create different versions of RestHistoricalSummaries depending on fork or perhap use seq instead.

kdeme commented Oct 23, 2024

View reviewed changes

cheatfate reviewed Oct 24, 2024

View reviewed changes

beacon_chain/spec/eth2_apis/rest_debug_calls.nim Outdated Show resolved Hide resolved

cheatfate reviewed Oct 24, 2024

View reviewed changes

beacon_chain/spec/eth2_apis/rest_debug_calls.nim Outdated Show resolved Hide resolved

cheatfate reviewed Oct 24, 2024

View reviewed changes

beacon_chain/spec/eth2_apis/rest_debug_calls.nim Outdated Show resolved Hide resolved

cheatfate reviewed Oct 24, 2024

View reviewed changes

beacon_chain/spec/eth2_apis/rest_debug_calls.nim Outdated Show resolved Hide resolved

kdeme force-pushed the historical-summaries-endpoint branch from e3d435b to d7047dd Compare October 24, 2024 12:48

kdeme marked this pull request as ready for review October 24, 2024 12:49

kdeme mentioned this pull request Oct 24, 2024

Use new historical_summaries beacon API endpoint in portal_bridge status-im/nimbus-eth1#2775

Draft

cheatfate reviewed Oct 30, 2024

View reviewed changes

kdeme commented Oct 30, 2024

View reviewed changes

kdeme added 3 commits November 8, 2024 10:26

Add REST endpoint to retrieve historical_summaries

816276d

Adress review comments: Correct errors + asyncraises

39d1d82

Move endpoint to /nimbus/v1/debug/ namespace for now

566e69b

kdeme force-pushed the historical-summaries-endpoint branch from d7047dd to 566e69b Compare November 8, 2024 09:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add REST endpoint to retrieve historical_summaries #6675

Add REST endpoint to retrieve historical_summaries #6675

kdeme commented Oct 23, 2024 •

edited

Loading

kdeme Oct 23, 2024

cheatfate Oct 25, 2024 •

edited

Loading

kdeme Oct 23, 2024

kdeme Oct 23, 2024

cheatfate Oct 24, 2024

kdeme Oct 24, 2024

github-actions bot commented Oct 23, 2024 •

edited

Loading

cheatfate Oct 24, 2024

cheatfate Oct 24, 2024

kdeme Oct 24, 2024

cheatfate Oct 25, 2024

tersec commented Oct 24, 2024

kdeme commented Oct 24, 2024 •

edited

Loading

cheatfate commented Oct 25, 2024

tersec commented Oct 25, 2024

cheatfate commented Oct 25, 2024

cheatfate Oct 30, 2024

etan-status Oct 30, 2024

kdeme Oct 30, 2024

kdeme Oct 30, 2024

Add REST endpoint to retrieve historical_summaries #6675

Are you sure you want to change the base?

Add REST endpoint to retrieve historical_summaries #6675

Conversation

kdeme commented Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

cheatfate Oct 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Oct 23, 2024 • edited Loading

Unit Test Results

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tersec commented Oct 24, 2024

kdeme commented Oct 24, 2024 • edited Loading

cheatfate commented Oct 25, 2024

tersec commented Oct 25, 2024

cheatfate commented Oct 25, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kdeme commented Oct 23, 2024 •

edited

Loading

cheatfate Oct 25, 2024 •

edited

Loading

github-actions bot commented Oct 23, 2024 •

edited

Loading

kdeme commented Oct 24, 2024 •

edited

Loading