Local register index and state re-execution

[sideninja]

Problem definition
Right now Flow execution node is responsible for executing all the transactions including EVM transactions. More specifically EVM became part of FVM where transactions are executed. While the EVM Gateway indexes transactions and blocks emitted via events.

The problem with this approach is while the EVM Gateway has transactions and blocks indexed it doesn't keep an EVM state, which is required for some of the JSON RPC APIs. How we currently implemented ability to also inspect the state is that we exposed some of the APIs through the EVM Core contract, methods like getBalance or getCode, which we were then able to call using a Cadence script. There are issues with the current approach (albeit being the simplest):

• all the requests require the execution of Cadence scripts which brings resource penalty as well as time penalty
• new future APIs will require for EVM Core contract to be updated, which is not ideal since it's deployed on the network and potentially introduces breaking changes which we want to avoid
• but most importantly, it prevents us from inspecting historic state defined by the EVM block height, how we currently handled this issue was, we executed Cadence scripts calling EVM Core contract API to inspect the state at previous Cadence heights, but since the Cadence height to EVM height is not always 1:1 mapping we are facing issues where that isn't the case

The following APIs require inspection of the state and allow for providing historic block heights:

• eth_getBalance loads the account balance
• eth_getStorageAt value from a storage position at a given address
• eth_getTransactionCount the number of transactions sent from an address
• eth_getCode code at a given address

Besides the getStorageAt which is not supported, we implemented the rest relying on the EVM Core contract API calls, which like explained fails with certain historic EVM heights as well as introduce other issues explained.

Proposal
The EVM Gateway was built to implement JSON RPC API specification, which is usually implemented by an "archive node" on the EVM networks. Such a node has a local state of the blockchain available to query and inspect.

We initially avoided doing this because it requires quite a bit of work to achieve but we have come at a point where this became unavoidable, specifically because of the EVM height problem, as well as the requirement to support getStorageAt API.

Because this is a specific problem to the EVM we would want to isolate the solution to the EVM Gateway itself. We could take an alternative approach of extending Flow archive node to index registers at EVM height resolution but that would leak complexity from EVM into Flow and is not a good idea.

The proposal is to extend EVM Gateway so it can operate in the archive mode, which besides indexing transactions and blocks as it already does, it also builds and indexes the state. It does so by re-executing all the transactions it indexes locally using an emulator.

Requirements
In order to achieve rebuilding the state on EVM Gateway we need a complete history of changes applied to the state (transactions), which we already have, but there is one caveat. Flow EVM exposes Cadence arch precompiles, which have the ability to read into the Cadence state, and that would not be available on EVM Gateway by simply executing EVM transactions. So we need to somehow have access to the Cadence state on EVM Gateway. The easiest solution for that is to include results (outputs) from the Cadence arch calls in the EVM transaction receipts, this way we can locally build and mock access to the Cadence state for already known calls, then when we re-execute the transaction we can provide the results from Cadence state it read while executing on the execution node.

Implementation
Emulator
We will have to create an emulator that will be able to execute EVM. There are two options here, either use existing Flow emulator, which in turn already integrates EVM, but this has some overhead of also containing complete FVM execution which is not really needed, the alternative solution would be to implement a simpler emulator only hosting EVM.

Event ingestion
We must improve event ingestion to build and index of all the Cadence values read through the Cadence arch, which would then be read with mocked precompiles provided to the above emulator EVM.

Ledger
EVM emulator would use the Flow EVM wrapper, which uses the stateDB wrapper and the stateDB wrapper relies on a ledger that stores key-values. We should investigate whether we write our own simple Pebble-backed ledger implementation.

Traces
Trace production would be done on the EVM Gateway since the execution will be done locally and can be used to produce the traces.

Tasks

Beta Give feedback

1. Ledger implementation #449
   +0
   
2. State reexecution #450
   +0
   
3. State query #451
   +0
   
4. Validating state index #452
   +0
5. Add missing transaction context #458
   +0
   
6. Implement checksum checking #572
   +0
7. Test against live networks #573
   +0
8. Implement circuit-breaker for querying state #587
   +0

[j1010001]

We have started implementation of improved solution in dry-run (onflow/flow-go#6539 , so this issue is no longer needed.