Emulator command for migrating contracts/state to Cadence 1.0

[turbolent]

Issue To Be Solved

The CLI should have a command to take in an Emulator snapshot created using a recent Emulator version with Cadence v0.42, run the Cadence 1.0 contract and state migrations on it, and produce a new Emulator snapshot for the latest version of the Emulator with Cadence v1.0.

Suggest A Solution

• Add a command like flow migrate
• Call the Emulator's Cadence 1.0 migration function, added in Support state migration for emulator state flow-emulator#557

[gregsantos]

@turbolent is this currently blocking this issue

[turbolent]

@gregsantos Though that issue is not closed yet, the work should be complete, it just hasn't been reviewed yet, so work on this issue can already commence by using the linked PR until it gets merged: onflow/flow-emulator#557.

The test case included in the PR shows what the CLI command needs to do, see https://github.com/onflow/flow-emulator/pull/557/files#diff-717939af182d8688c65dd7e4a5b5471360a4e3719f2dba0b34629167f3ce70afR33-R66

[turbolent]

@gregsantos what is the ETA for when this could be picked up and finished by the Dev Ex team? (No pressure, just asking for planning purposes)

[gregsantos]

@turbolent hoping to get kicked off this week just need to scope out with team. Myself, @chasefleming or @ianthpun will update estimate on completion

[gregsantos]

2 main parts need discussion and design

• How to manage 2 CLI installations
• How are we going to trigger the migration (auto start or manual stop/start)

We also need to sync with @SupunS to help expedite understanding of how his linked DRAFT PR works and if we can use it as is to get started.

[turbolent]

How are we going to trigger the migration (auto start or manual stop/start)

What is meant by "manual stop/start"? I imagine when the flow migrate command is run, it immediately performs the migration and exits when it is complete.

expedite understanding of how his linked DRAFT PR works and if we can use it as is to get started.

All that needs to be done is to call the MigrateCadence1 function, like demonstrated in the linked test case. Is there anything that is unclear?

[SupunS]

Just cross-posting from the offline thread:

So yeah, like Bastian has mentioned above, at the moment, the draft PR has this function func MigrateCadence1(...) which takes-in a snapshot file as a *sqlite.Store, and migrate the snapshot file in-place.
So from the CLI point of view, what we would need is a command that will directly call this function
i.e: a simple command that would do something similar to:
https://github.com/onflow/flow-emulator/blob/201c987932fc2a3b4f0939eb5d24ce550e2200c0/storage/migration/cadence1_test.go#L54-L62

I guess some confusion might be coming from this previous thread: https://discord.com/channels/613813861610684416/1123314820763111465/1202314935053594715, where I did some brainstorming to see how we could do this migration as a SPI in emulator. But given that turned out to be overly complex, we thought of doing the migration on the snapshot file directly. So we would ask developers to take a snapshot file from an old emulator, run the migration against it, and then use it in (copy the file over to) the new emulator, and start the emulator.

[bthaile]

Can the snapshot be of a fork testnet to the emulator? Then devs can get all contract dependencies locally to test the migration with their own contracts.

[SupunS]

a fork testnet to the emulator

Not really sure what that is / what you meant there 🤔

This particular command should work as long as the snapshot was previously created by an emulator (SQLite file), with the --persist flag.

Running the migrations against a testnet/mainnet state, would require running the state-migrations tool against a checkpointing file. This part is only for us as the protocol devs, because this will migrate all the checkpoint data + contracts.

[turbolent]

@janezpodhostnik mentioned that we might be able to run TN/MN state in the Emulator, by extending the Emulator's storage layer to support protocol state tries, but that's a separate discussion.

Let's focus on what this issue is about, allowing the migration of Emulator snapshots.

[gregsantos]

The plan is to add this functionality to the emulator as a --migrate flag.
The idea is if the user runs C1.0 emulator with --migrate (e.g. flow emulator --migrate) the emulator will load from a snapshot & migration will be run at the time of the emulator startup.
"manual stop/start" refers to the user having to stop the pre-C1.0 emulator and run the C1.0 emulator with this flag
The migration won't necessarily exist as a standalone command but as a part of the emulator startup.

[jribbink]

Moving this issue to flow-emulator repository since this is where the code changes will exist.

[gregsantos]

Let's focus on what this issue is about, allowing the migration of Emulator snapshots.

Can we elucidate a bit why this is needed or particularly useful for devs needing to update?
I'm concerned that the dependency management required to test staging/migrating locally makes this impractical and unlikely to be utilized (as opposed to the Testnet migration environment).

[turbolent]

What is the current status / latest update here?

If Dev Ex does not currently have bandwidth for this, @SupunS offered to pick this up / continue the effort.

[turbolent]

@gregsantos

Can we elucidate a bit why this is needed or particularly useful for devs needing to update?
I'm concerned that the dependency management required to test staging/migrating locally makes this impractical and unlikely to be utilized (as opposed to the Testnet migration environment).

As I mentioned in the last office hours, where we discussed this issue, this feature allows developers to quickly iterate on upgrading their contracts to Cadence 1.0, by locally performing the state and contract upgrade.

They will also be able to iterate using networks, but iteration will be much slower, as staging a contract update will not provide immediate feedback like this feature does – instead developers have to wait until the next migration is run by the Flow team, which might be hours or even days away.

[jribbink]

Based on discussion in onflow/flow-emulator#572 (review), I guess this should probably be moved back the the CLI repo?

[SupunS]

Added in #1426